Apostolof's Blog

Nothing important going on here.

Lessons learned from updating a relic

Table of Contents

01. Premise

02. Current status

03. Old work

04. Plan for the future

05. Updates

 5.1. Meeting with the administration and moderation team 📜

 5.2. Clean repository setup, a small disaster? 😫

 5.3. New repository gets commits. Production database password gets leaked. 🤦

 5.4. Our first team call 🎉

 5.5. Get them OBFuscated dumps 💪

 5.6. Great success 👍

 5.7. Managerial stuff are a pain 😵

 5.8. Strife for mods 👊

 5.9. A hard choice 👾

 5.10. The Easter took us out 🐐


Premise

thmmy.gr is an online forum set up and maintained by students of the Electrical and Computer Engineering school at the Aristotle university of Thessaloniki. It is almost 20 years old and currently has 30.454 topics and 1.394.342 posts! It is quite a big forum for Greek standards. I have been a member, moderator and webslave (sys-admin) there for many years.

The forum was first online on July of 2003 and has been mostly online ever since. It originally lived on a redundant computer the university provided. It might actually have migrated through a few machines the faculty provided. A couple of years back, after yet another down-time during an exams period, we decided to migrate to an external VPN.

Moving to a stable system has been a good first step. Now, it is important to secure the deployed instance, improve the user experience and install a few necessary automated pipelines. In this post I will be publishing updates about the effort. Any work done will be summarised here, problems faced, interesting hacks developed, insights, (hopefully) all posted. I hope when/if the project ends, we will have accumulated some useful knowledge and may extract conclusions about the process.

Current status

First, lets record the current status of the deploy. The forum is deployed on a famous German VPS service. We are running SMF 1.1 with about a dozen mods (plugins). Also, a few modifications have been made over the years directly inside the running instance SMF source code. No git history or any other documentation available for these.

SMF runs on PHP 5.3 and the database used is MariaDB 5.5. You can easily find a Docker image for MariaDB/MySQL 5.5, but PHP 5.3 is a whole story. It is a very old version and it's hard to find binaries for it, let alone find a working Docker image. Nevertheless, we found a Dockerfile that after some slight modifications reached a semi-buildable state. Sometimes it fails due to some obscure linus repositories being down. Thankfully, since we have built it at least once, we are covered as long as we have the image backed up.

Old work

Various people, including myself, have made efforts to update the platform in the past, but failed mainly due to lack of time. These efforts have so far been purely documented. The two main hurdles are the SMF update path (i.e. the database schema update) and the update of the mods. Mods are also interfering with the platform update, because some of them apply migrations to the database. Also, there are custom mods created by people on old management teams whose purpose and function is neither documented nor clear.

SMF's documentation is decent, and it recommends doing the update on two steps, first going to v2.0 and then to the latest. On my old attempts one of the updates usually breaks. I only managed to make it succeed one time, but I did not document the exact process 🤦. The main problem that arises that the database migration fails due to unexpected columns on the tables (see migrations introduced by mods mentioned above).

The latest update efforts have been my personal project. But, a couple other people have stepped up and offered to help. To accommodate this while exposing as little of the data as possible, I wrote an SQL script that deletes and/or obfuscates all entries except the ones the developer already has access to. This way I can extract personalised dumps that they can use to develop the update process. That is the last thing I've done and I have not been occupied with this project in a long while.

Plan for the future

Now that we have a summary of the situation, we can lay out the desired goals. This includes:

Updates

February 20, 2023

Meeting with the administration and moderation team

We met up to discuss various subjects about the forum, including the efforts to update. Some new people have expresses interest to help with the process 🙂

February 22, 2023

Clean repository setup, a small disaster?

Following the new interest found, I added everyone to the discord server I have about the update and set up a fresh git repository.
I also took a new backup of both the database and the application (PHP) file of the server.
Sadly, I realized I have lost almost all previous work done for the project 🤦 It seems I did not migrate the old repository when I formatted my system a while back, thinking that I had pushed the repo to some remote. That is not the case, since I cannot find the repo to any of the git hosts I use.

February 26, 2023

New repository gets commits. Production database password gets leaked.

I replicated all the information I could remember from the old repository. I copied the application files from the production, which leaked the production database password because of course it's saved in plaintext on the root application directory. 🤦

March 09, 2023

Our first team call 🎉

We had our first team call. We discussed various aspects of the update and set the final goals. Happy to say everyone was pretty much on the same page. Also, some ideas I previously contemplated were brought up by other people, which is a good sign that they are good ideas.

March 13, 2023

Get them OBFuscated dumps 💪

I created the script that obfuscates and/or truncates parts of the database according to a user ID. I generated and distributed the personalized data dumps to each member of the team.
I also added a template to use for the documentation of existing modifications to the forum, which is the first step we decided to take.

March 19, 2023

Great success 👍

After a member of the team took a deep dive in the project, we have our first successfully update instance of the forum 🎉
Following the steps of his attempt, I wrote an upgrade documentation that is reproducible.

March 19, 2023

Managerial stuff are a pain 😵

I created a Github project for managing the project. Github's projects really lacks in comparison to Trello, we'll see how it goes.
I started experimenting with migrating the database to PostgreSQL using pgloader. It didn't really work.

March 22, 2023

Strife for mods 👊

The team started experimenting with updating/re-installing the necessary mods, namely TinyPortal. We haven't yet found a way to make this work.

March 22, 2023

A hard choice 👾

After a week of discussions and exploring our options, we decided against migrating to another forum platform. NodeBB was a strong contestant, but based on the gross additional work needed for the migration and some technical difficulties with it, we ruled it out.

April 17, 2023

The Easter took us out 🐐

After a week of vacation during Easter, we came back strong. I got a new VPS where I set up a staging environment with the updated forum.
After passing said staging infos to the forum moderation team, they came back with a huge list of problems they spotted 🙃
The real work begins. I set up a trello board to manage the issue reporting.
We also started investigating the changes needed for mThmmy to work with the updated forum. A reminder: mThmmy is the custom Android App we have created a few years ago for the forum.