Lessons learned from updating a relic
Table of Contents
01. Premise
02. Current status
03. Old work
05. Updates
5.1. Meeting with the administration and moderation team 📜
5.2. Clean repository setup, a small disaster? 😫
5.3. New repository gets commits. Production database password gets leaked. 🤦
5.5. Get them OBFuscated dumps 💪
5.6. Great success 👍
5.7. Managerial stuff are a pain 😵
5.8. Strife for mods 👊
5.9. A hard choice 👾
5.10. The Easter took us out 🐐
Premise
thmmy.gr is an online forum set up and maintained by students of the Electrical and Computer Engineering school at the Aristotle university of Thessaloniki. It is almost 20 years old and currently has 30.454 topics and 1.394.342 posts! It is quite a big forum for Greek standards. I have been a member, moderator and webslave (sys-admin) there for many years.
The forum was first online on July of 2003 and has been mostly online ever since. It originally lived on a redundant computer the university provided. It might actually have migrated through a few machines the faculty provided. A couple of years back, after yet another down-time during an exams period, we decided to migrate to an external VPN.
Moving to a stable system has been a good first step. Now, it is important to secure the deployed instance, improve the user experience and install a few necessary automated pipelines. In this post I will be publishing updates about the effort. Any work done will be summarised here, problems faced, interesting hacks developed, insights, (hopefully) all posted. I hope when/if the project ends, we will have accumulated some useful knowledge and may extract conclusions about the process.
Current status
First, lets record the current status of the deploy. The forum is deployed on a famous German VPS service. We are running SMF 1.1 with about a dozen mods (plugins). Also, a few modifications have been made over the years directly inside the running instance SMF source code. No git history or any other documentation available for these.
SMF runs on PHP 5.3 and the database used is MariaDB 5.5. You can easily find a Docker image for MariaDB/MySQL 5.5, but PHP 5.3 is a whole story. It is a very old version and it's hard to find binaries for it, let alone find a working Docker image. Nevertheless, we found a Dockerfile that after some slight modifications reached a semi-buildable state. Sometimes it fails due to some obscure linus repositories being down. Thankfully, since we have built it at least once, we are covered as long as we have the image backed up.
Old work
Various people, including myself, have made efforts to update the platform in the past, but failed mainly due to lack of time. These efforts have so far been purely documented. The two main hurdles are the SMF update path (i.e. the database schema update) and the update of the mods. Mods are also interfering with the platform update, because some of them apply migrations to the database. Also, there are custom mods created by people on old management teams whose purpose and function is neither documented nor clear.
SMF's documentation is decent, and it recommends doing the update on two steps, first going to v2.0 and then to the latest. On my old attempts one of the updates usually breaks. I only managed to make it succeed one time, but I did not document the exact process 🤦. The main problem that arises that the database migration fails due to unexpected columns on the tables (see migrations introduced by mods mentioned above).
The latest update efforts have been my personal project. But, a couple other people have stepped up and offered to help. To accommodate this while exposing as little of the data as possible, I wrote an SQL script that deletes and/or obfuscates all entries except the ones the developer already has access to. This way I can extract personalised dumps that they can use to develop the update process. That is the last thing I've done and I have not been occupied with this project in a long while.
Plan for the future
Now that we have a summary of the situation, we can lay out the desired goals. This includes:
- upgrade from SMF 1.1 to the latest version
- upgrade the mods to the latest version or find new equivalents
- migrate the database to postgres
- setup automated backups
- add a couple of modern themes
- add a mod for exposing an API
- add a mod for elastic integration
- setup some monitoring, if necessary
Updates
February 20, 2023
Meeting with the administration and moderation team
We met up to discuss various subjects about the forum, including the efforts to update. Some new people have expresses interest to help with the process 🙂
February 22, 2023
Clean repository setup, a small disaster?
Following the new interest found, I added everyone to the discord server I have about the update and set
up a fresh git repository.
I also took a new backup of both the database and the application (PHP) file of the server.
Sadly, I realized I have lost almost all previous work done for the project 🤦 It seems I did not
migrate the old repository when I formatted my system a while back, thinking that I had pushed the repo
to some remote. That is not the case, since I cannot find the repo to any of the git hosts I use.
February 26, 2023
New repository gets commits. Production database password gets leaked.
I replicated all the information I could remember from the old repository. I copied the application files from the production, which leaked the production database password because of course it's saved in plaintext on the root application directory. 🤦
March 09, 2023
Our first team call 🎉
We had our first team call. We discussed various aspects of the update and set the final goals. Happy to say everyone was pretty much on the same page. Also, some ideas I previously contemplated were brought up by other people, which is a good sign that they are good ideas.
March 13, 2023
Get them OBFuscated dumps 💪
I created the script that obfuscates and/or truncates parts of the database according to a user ID. I
generated and distributed the personalized data dumps to each member of the team.
I also added a template to use for the documentation of existing modifications to the forum, which is
the first step we decided to take.
March 19, 2023
Great success 👍
After a member of the team took a deep dive in the project, we have our first successfully update
instance of the forum 🎉
Following the steps of his attempt, I wrote an upgrade documentation that is reproducible.
March 19, 2023
Managerial stuff are a pain 😵
I created a Github project for managing the project. Github's projects really lacks in comparison to
Trello, we'll see how it goes.
I started experimenting with migrating the database to PostgreSQL using pgloader. It didn't really work.
March 22, 2023
Strife for mods 👊
The team started experimenting with updating/re-installing the necessary mods, namely TinyPortal. We haven't yet found a way to make this work.
March 22, 2023
A hard choice 👾
After a week of discussions and exploring our options, we decided against migrating to another forum platform. NodeBB was a strong contestant, but based on the gross additional work needed for the migration and some technical difficulties with it, we ruled it out.
April 17, 2023
The Easter took us out 🐐
After a week of vacation during Easter, we came back strong. I got a new VPS where I set up a staging
environment with the updated forum.
After passing said staging infos to the forum moderation team, they came back with a huge list of
problems they spotted 🙃
The real work begins. I set up a trello board to manage the issue reporting.
We also started investigating the changes needed for mThmmy to work with the updated forum. A reminder:
mThmmy is the custom Android App we have created a few years ago for the forum.