State of the Instance (September 2022)
For a few months, I have done nothing at all around Cawfee Club, so these articles have ceased. Now, however, there's finally a few things!
Outages
This month, we've had a few outages that lasted over a few hours. This was mainly because of free disk space - when there's no space left, the database shuts down. I didn't keep an eye on this and somehow it always happened when I was either away from the computer, or sleeping.
The big outage
The worst outage happened on the evening of 17th of Septeber, when I suddenly felt like updating Pleroma - this has never wrecked things before. This time, it ran a database migration that consumed all remaining free space and left Pleroma unable to start!
I was away from home and preoccupied with other things, so it took me a day to get the instance running again. (Lesson learned: Never update if you're not ready to fix things.)
Performance grinds to a halt
After the instance was running again, the performance dropped so much that it got unusable. (Apparently, other admins had the same problem.) It turned out that there was now a new database migration that was supposed to periodically run in the background and delete some objects that were not necessary. The intention was good, but it completely saturated the database throughput.
The best suggestion I got was to delete these objects manually and then... vacuum the database?
Vacuuming
Vacuuming is an operation that compacts database tables and reclaims space taken up by empty rows. However, a temporary instance of the table is created in the process. That already needs a considerable amount of free disk space to start with - which I didn't have!
The only way to do this without upgrading the server plan was to shut down Pleroma and then shuffle a lot of data around and even move some of it off-server, which resulted in all uploaded media being only on my hard drive for a few hours. Thankfully, the restore after performing the vacuum went well. This entire operation took a little over a day again.
By the end of it, we were left with 25GB of free space, which - judging by current usage patterns - should last us for at least another three months. In the meantime, I'll figure out something else.
Backups (sort of)
So as a side effect of that, all media is now backed up on my hard drive (see update from February). The database itself is not backed up off-site yet, and I'll still have to figure out how to do that without as little downtime as possible. This is hard to safely experiment with, which will be partially solved by the next point...
Testing instances!
These instances will be very ephemeral and used mainly for testing stuff, such as (but not limited to):
- aforementioned backups and their recovery
- various Pleroma/DB settings
- different frontends and backends, and how they federate
- configuration management and automation with Ansible
- Pleroma development (maybe one day)
The first such instance is already live at 000.testing.cawfee.club. Sign-ups are open for the time being, but you have to count with the fact that the instance may disappear any time.
As a side effect, I now have a minimal list of steps that will provide me with a functional instance. I decided to go with Rocky Linux, a replacement for CentOS, which we're using and which doesn't receive updates ever since Red Hat pulled the plug at the end of 2021. (Thanks, assholes!) I'd like to migrate to that in the near future, so it's nice to see that Pleroma runs fine on it.
(I didn't want to post an update until at least this was done.)
Operating costs
This will be a recurring segment. I'll be posting about last month's costs to make things more transparent.
| Operating costs for August 2022 | |
|---|---|
| cawfee.club (server) | $30.00 USD |
| cawfee.club (block storage) | $1.25 USD |
| old.cawfee.club (server) | $3.50 USD |
| Total | $34.75 USD |
old.cawfee.club is currently pointing to the server that originally hosted the instance before I scaled up with a fresh install, but I've kept it around. Some stuff still runs on it. I'll probably move monitoring there to save space on the main server.
New guide
While trying to install Pleroma on the testing instance, I stumbled upon a guide on Pleroma-FE in the Pleroma docs, so I linked that in the welcome post. Originally there was none, so I planned to write some myself (heh), but this one seems pretty good for getting started.
