State of the Instance (July 2021)
Outages
What the hell is going on?
The graph
Some of the graphs I have for monitoring show the (technical) state of the instance quite neatly. Since the HTTP requests graph takes data directly from Pleroma, any time Pleroma is down for any reason, there is no data on HTTP requests and a gap in the graph is formed.
There's been three of such outages in the last two weeks. What exactly happened?
2021-06-25
We ran out of disk space.
2021-06-27
We ran out of disk space again.
Apparently the graph service has a distinction between free and available space, the former of which is deceptively higher. That's how I was misled into thinking there's still more than 4GiB of space left on the main partition, until it crashed to zero while I slept. (It always happens when you sleep.)
2021-07-04
Probably the most tragicomic outage of the three.
The credit card I use for invoices related to this server ran out of funds. I have two of them and this one is currently not my primary one, so I let it deplete, forgetting about the invoices.
(Our service provider even sent me a bunch of emails warning me about this but alas, I was off the grid and I don't check that inbox often. The usual procedure is that if invoices aren't paid on the 1st, the service gets suspended on the 4th and terminated on the 7th. A day longer and this instance wouldn't exist anymore!)
But here's the "fun" part, I couldn't even use the other credit card because SMS 2FA wasn't working! A day and a half later, I finally found a solution and paid for the server... so that's resolved now.
