On Thursday afternoon, I updated the server hypervisor and restarted it to apply a new kernel update. When it booted up, I couldn’t get to this blog or a couple other web services I run for some reason.
All but one of the broken services used the same virtual network bridge, so I tried switching to a different bridge with no change. I also tired unplugging and replugging the ethernet cable from corresponding port on the network card. The made no difference either.
I took a look at my reverse proxy settings on my router. It appeared that my problem was HAProxy because changing HAProxy settings made some difference. When I disabled the health checking in HAProxy, the sites would load, but very slowly. I thought maybe the router needed to be restarted, so I did that. Unsurprisingly, that made no difference either.
I figured the slow page loading had something to do with the Apache server having a messed up configuration or an incorrect module activated, even though I didn’t change any settings earlier. All the config files looked correct, and all the right modules were loaded. I looked a resource usage and noticed very low ram usage during page loads compared to working services. I tried updating the virtual machines, but that didn’t do anything.
Then I tried updating the working services to see if they’d break. That made no difference (probably a good thing). Finally I tried updating my database server. It couldn’t reach its update server or ping anything. Finally, a problem that could be solved. The DNS server setting was incorrect. I set up a Pi-Hole for my server subnet and I must have changed the DNS server setting in /etc/resolv.conf. Ubuntu uses /etc/netplan to set DNS servers, and I was unaware of that. I corrected the DNS server setting and everything was back to normal.