Oops, no more space
Day 17 of #100DaysToOffload
I love Matrix, and use it quite often. I both participate in matrix rooms and talk to some people individually. I also self-host my Matrix instance (as evidenced by username on my contact page), so when a problem not related to the client occurs, my server could be to blame. A few days ago, my client (just plain old Element), had the “Connectivity to the server has been lost” error, but my wifi worked and I had no issues connecting to the server. Spoiler: the title gives it away.
A little bit of panic
At first, I simply thought my internet was doing something weird. Nope that worked. My server? Nope, I could ssh in just fine. Matrix acting up? Nope, connect and even log in through Element web! Now, some (probably unjust)[^1] panic ensued. All my initial diagnosis didn't turn anything up, and I was slightly worried something had gone horribly wrong and I would have to restore from a backup, which would mean needing to load many, many messages (again, if you look at the title, this would not have helped).
My Matrix (among other things) server runs on NixOS, so I decided to just run a rebuild (perhaps something needed updating or some other weird issue). I got a ton of errors which briefly overwhelmed me. After a deep breath (at least a mental one), all the red text complained about not enough free space.
A little bit of shock
My server runs on a Hetzner CPX11 instance, with about 40GB of storage. It shocked me I had already filled that up. Did Matrix messages really take up that much space?? Nope it was entirely my fault!
Forgetting to resize
Originally, I used a simple CX11 instance, but I realized I a) needed more storage and b) needed another vCPU for preformance for another self-hosted app. When I rescaled, I remembered to change the parition size, but I had completely forgot to extend the file system to the parition size! I thought I did, but after running a
resize2fs command, all was fixed. My Matrix server was up and running!
 Luckily, I had other methods of reaching important contacts, but I was also somewhat active in a room discussion at the time and I did not want to respond later than I already was. Although, it really would not have been a big deal.