Major update for Guru Archives coming soon (spoiler: search)!

Hello everybody - I've got some exciting news to share!

The Guild Wars Guru archives have seen a lot of change over the last few years, going from a huge static HTML file dump to a more modernized and easier browsing experience. Now, we are ready to take the next step and I'm glad to say that a lot of improvements have been made to the archives that will be going live soon.

So, let's get straight to it!

Ripping out the database and replacing it

When I started scraping the database, I ended up utilizing a database solution for it called MariaDB (which is a fork of MySQL, one of the most popular open source database systems in the world). The choice for MariaDB was logic, as Guild Wars Legacy itself uses MariaDB as well (and Guild Wars Guru, back in the day, ran on MySQL as well).

MariaDB has a lot of advantages, but the big disadvantage of it is that you have a database program running on your server that you need to maintain. The Guru archives are a static archive (as you can't post on it anymore) and running on top of MariaDB ment that the big dataset of it was taken automatically by my tooling and replicated, so that made data loss very unlikely. However, it also makes development a bit more of a hassle and it complicates server moves - the dataset behind the Guru Archives is currently about 7 GB large soo that's quite a big and unwieldly database to manage. MariaDB is certainly up for this task, without a doubt, but I had been working more and more with SQLite lately and the Guru Archives looked like a great match with SQLite: the database is a single file, the performance (especially for our data set) is even faster than MariaDB and backing it up is very easy.

So, in a few days work, the entire archive was migrated to a SQLite database - the most work went into migrating the actual data to the new database. There are some tools like mysql2sqlite that help automate this process, but it took 20 hours and it still didn't finish... so I went with another solution here and wrote a custom tool in Go, which completed the entire migration of the data in a mere 6 minutes, which is 200 times faster compared to the tool that didn't even finish!

Fixing broken posts

There were quite a few issues with the lay-out of the archive: while it certainly did not look bad, it ended up having quite a lot of issues here and there and the lay-out often broke due to how we had to scrape the data before. We ended up with quite some tags that weren't closed and that broke the lay-out, sometimes spectacularly. So, that was something that needed to be taken care of.

So once again, I wrote a tool to fix the broken HTML and to properly close all tags. It had to go through all of the posts (nearly 5 million posts in total!), look through each and every single post, fix the HTML and extract the right part of it. So, you are probably wondering how long that took... well, it was actually included in my migration tool so that took about 6 minutes in total, for both the migration and all of the fixing of the HTML...

In the end, this was no luxury because it turned out that 3,544,317 posts required HTML fixes — that's 72% of posts that had broken HTML!

Refreshing the lay-out

Now, the lay-out of the archive wasn't the worst, but it certainly was a bit rough and could use some attention. So, a lot of time and attention went into creating a new lay-out for the archives.

While the previous version of the Archives used Bootstrap for the lay-out, this time around I utilized templ and Tailwind to cobble together the lay-out. And, I have to say, it looks amazing!

The new homepage of the archives no longer just present data, but it also provides you with an overview of the amount of threads and posts in each category. Everything is grouped nicely together and only categories with threads/posts are now shown instead of those that we did not have data for.

Now, that's not the only thing that has changed!

Fixing images and avatars

Let's just spoil this directly: images and avatars now work a lot more than they used to work in the previous version of the archive. The images were always there, but the paths in the database, the broken HTML and more just ended up breaking them a lot. No longer, as I've made a big effort to ensure most images load. Some, however, are still not available but hopefully we can recover that one some day.

So, how does this look now? Let's take a quick look at the current look:

And this is the same page in the new version:

Not only do (almost) all images load as they should, but most avatars on the archive have been fixed as well (though, unfortunately, not all). Users without an avatar get a nice golden touch now.

Threads

Now, previously, when browsing the archive you got quite a static overview of posts with almost no reference: when were these posted, by whom, how many posts were made? So, I made sure to resolve this by adding a lot more information to the overview:

This way, you can find what you are looking for much more readily.

Now, is there more to be told? Absolutely!

Something new

Now, I also wanted to add something completely new to the archive, that ever since Guru went down was no longer possible: sear... no, not search, sorry! But I'm glad to say that the archives now have user pages, where you can easily find posts made by a user:

I'm pretty certain that Cosyfiep has the most posts on the Archive, before some database optimizations this page took seconds to open as there are 1911 pages of posts made by Cosyfiep!

The user pages should help finding posts in the archive easier than ever. Ofcourse, what could make finding information even faster?

Search!

Now, the keen eyed will probably have seen in all the previous screenshots that there is a search bar and a Search link... so yes, the Archive now has a search function! One of the big reasons for this entire update and all changes was to make search possible (as well as easier maintenance for myself!). So, it is now possible to search through all nearly 5 million posts on the archive!

So you can always search for your favourite post about Grawls. Because they rock!

You can search in specific categories or search through the whole forum. There are some restrictions in place (5 searches per minute when we launch), but those will probably be raised once I can measure the real impact of the searches on the server!

Search optimizations

The first searches I did took 14 to 20 seconds to complete - we have a huge dataset. So, improvements had to be made here and at the cost of tripling the size of our database, the searches are now 40 to 50 times faster and complete in less than half a second, and we now have full text search.

That means a huge update to the entire archive and helps improve the speed a ton! So, have fun searching through exactly 4,911,097 posts!

What is next?

There's still a lot of work to do. Some parts of the archive aren't accessible yet because of missing links that still need to be made in the database - some posts are missing a thread, or a link to the category. Sometimes you'll find posts linked to a user that give an error - I'm aware of this and this will require time to fix as it requires re-scraping the files, which is a huge undertaking. But, eventually, it will happen.

But this is without a doubt the biggest update to the archive yet and I can't wait to share it with you all soon. I'm still testing things and working on deploying it, but I hope to share this within days rather than weeks!

So, let me know what you think of this update and I'll make sure to reply!

- Kevin