Oops, no more space

Day 17 of #100DaysToOffload


I love Matrix, and use it quite often. I both participate in matrix rooms and talk to some people individually. I also self-host my Matrix instance (as evidenced by username on my contact page), so when a problem not related to the client occurs, my server could be to blame. A few days ago, my client (just plain old Element), had the “Connectivity to the server has been lost” error, but my wifi worked and I had no issues connecting to the server. Spoiler: the title gives it away.

A little bit of panic

At first, I simply thought my internet was doing something weird. Nope that worked. My server? Nope, I could ssh in just fine. Matrix acting up? Nope, connect and even log in through Element web! Now, some (probably unjust)[^1] panic ensued. All my initial diagnosis didn't turn anything up, and I was slightly worried something had gone horribly wrong and I would have to restore from a backup, which would mean needing to load many, many messages (again, if you look at the title, this would not have helped).

nixos-rebuild switch

My Matrix (among other things) server runs on NixOS, so I decided to just run a rebuild (perhaps something needed updating or some other weird issue). I got a ton of errors which briefly overwhelmed me. After a deep breath (at least a mental one), all the red text complained about not enough free space.

A little bit of shock

My server runs on a Hetzner CPX11 instance, with about 40GB of storage. It shocked me I had already filled that up. Did Matrix messages really take up that much space?? Nope it was entirely my fault!

Forgetting to resize

Originally, I used a simple CX11 instance, but I realized I a) needed more storage and b) needed another vCPU for preformance for another self-hosted app. When I rescaled, I remembered to change the parition size, but I had completely forgot to extend the file system to the parition size! I thought I did, but after running a resize2fs command, all was fixed. My Matrix server was up and running!

[1] Luckily, I had other methods of reaching important contacts, but I was also somewhat active in a room discussion at the time and I did not want to respond later than I already was. Although, it really would not have been a big deal.