After too many weeks of struggling and waiting and struggling some more, I finally have a functional server running almost all the services I was planning on, including this blog. Last we left off, about a month ago, I was rounding up the parts I needed to assemble a white label server. So here’s what I got.
The Hardware
Intel S2400SC motherboard: This is a dual socket LGA1356 motherboard with eight RAM slots total and a decent selection of PCI express slots. I picked it because it has PCIe x16 and x8 slots that would be capable of holding a graphics card. I initially planned to use the built-in SAS mini ports to handle my hard drives, but this didn’t end up working out. I used the same Xeon E5-2450 CPUs I bought for the HP server. I also decided to add 16GB more memory just in case.
Dell PERC H310: I flashed this popular and cheap SAS card to IT mode which allows it to be used as an HBA so I can use ZFS instead of hardware RAID.
Corsair RM850x power supply: I spent a tad more than I wanted to really, but I wanted an 80 Plus Gold power supply giving at least 850 watts with semi-modular cables at the minimum. They were all priced similarly, but in and out of stock, probably due to the pandemic.
HP NC365T: A popular and cheap 4-port gigabit network card.
Antec P101 case: I picked this because it billed itself as a quiet case, it came with four fans and it said it should hold an E-ATX motherboard, meaning my CEB motherboard shouldn’t be a problem.
Turned out, the CEB motherboard was a tiny problem. Despite what the internet said would happen, only three screw holes lined up with the standoffs in the case. The standoffs that didn’t line up were touching the motherboard, so they had to be removed. The motherboard was definitely not mounted securely with only three screws, and it was a bit too flexible with no standoffs behind it. There were some foam standoffs on the motherboard, but only two and they weren’t in the right places.
I moved and super glued the foam standoffs into new positions, and then I 3D printed a few more for the rest of the board.
With that done, it was no trouble the get the board in and screwed down. I had to use a zip tie in the upper right corner, but it gets the job done just fine. Building in this case was alright I guess. I’m not a fan of the immovable power supply shroud, and I don’t really care for the tool-less drive bays. On the other hand, there wasn’t much choice for an E-ATX case at this price.
With the basic components assembled, it was time to install the hard drives and get going. Flashing the H310 to IT mode was a piece of cake and not worth talking about more. I bought two Seagate 1TB SAS drives to use in RAIDZ for my hypervisor. For some reason, these are incompatible with the H310. It knows the drives are connected, but doesn’t pass them though to the OS. I couldn’t get to them in Proxmox or a GParted live USB drive. It might be the fault of the hard drives, and not the H310, but I don’t have any other SAS equipped devices to test that theory out. So I bought another copy of an old 750GB SATA drive I had laying around. That worked just fine.
The Software
With the computer finally up and running, it was time to get some services installed. I got WordPress installed first, and it was running fine when it was being accessed via port forwarding on my router. I realized I was going to need a reverse proxy to handle access to multiple services on a single domain name. pfSense has HAProxy available in its repositories, so I decided to go with that. It’s a little convoluted to set up if you have no experience with reverse proxies, but after watching a couple videos and reading a few articles, I got it going. There was one problem though; no formatting (CSS and the like) was being applied to web pages accessed through the reverse proxy.
I found a few solutions on the web, but I couldn’t figure out how to implement them. They all involved changing the HAProxy configuration, which wasn’t a problem. It seems like most people run these reverse proxies on a separate virtual machine, not on pfSense, so their configuration was done in a text file rather than a web interface. Forum posters were being told to add a couple lines of code to the config files, but I wasn’t able to do that on pfSense. I decided to come back to that later.
After adjusting some HAProxy settings to try to get a reverse proxy going for my local services, I accidentally locked myself out of the pfSense web interface. No problem, I thought, I’ll just roll back to the most recent snapshot. So I did, and I was back in. That also wiped out the new SSL certificate I made earlier in the day. Again, not a problem I thought, I’ll just issue it again. Turns out I had already issued all the duplicate certificates I was allowed for the week, and I couldn’t get a new one until the next week (today).
In the Meantime
While I was waiting for the timer to run out on the certificate, I decided to press on with some other services. I set up a wiki to keep track of my network and installation notes for things. I decided to would be fun to try out an RSS reader, so I got Tiny Tiny RSS going.
A big part of setting up this server was for media serving with Plex, or something like it. I looked into the options, and it seems the three most popular, in descending order, are Plex, Emby and Jellyfin. I was originally planning on Plex, because it has the best name recognition and I knew it could do what I wanted. I tried out Emby before, and I was considering it this time. I hadn’t heard of Jellyfin before, but people seemed to like it. I axed Plex because while most commenters said it did a great job of serving up media, many useful functions are locked behind a not-inexpensive premium tier. The same is true of Emby, and Emby has an additional demerit with its licensing. Basically, Emby used to be open source, and the developers suddenly decided to go closed source. I don’t necessarily have a problem with closed-source software, but going from open to closed, and so abruptly is a problem to me. Jellyfin picked up the pieces and is based on the last open-source Emby code available. It’s not totally 100% quite yet, but it does everything I need.
From the start, this media server was supposed to take over the live TV services from the living room computer. Things were looking promising at first. Jellyfin supports live TV, and using NextPVR as a backend, so I was all set with my stupid Ceton tuner. I put the tuner in (with no drama like the HP server had) and set up PCI passthrough to a Windows VM for NPVR. I connected Jellyfin to that and things were working perfectly, with absolutely no fuss. I seemed too good to be true. And it was.
While the tuner worked perfectly (better than the bare-metal installation on the living room computer) with the Windows VM turned on, things went south when I shut the VM off. Shutting the VM down crashed the whole server. It was an abrupt shutdown, like yanking the power cord. That wouldn’t work. If it was a graceful shutdown, like clicking the button in Proxmox, maybe I could have dealt with it. Maybe not with the frequency with which Windows needs to reboot for updates.
The only solution to letting the new server handle the TV services was to get a new tuner. Pickin’s are pretty slim when it comes to cable card tuners these days. You can either have an old (used) HD Homerun, but they’re really expensive and only have three tuners, or you can have a stupid Ceton which has more tuners, but has very little software support (probably because the company doesn’t exist anymore). I picked a Ceton InfiniTV 6 Eth. This is an ethernet tuner similar in style to an HD Homerun, except worse. It’s gotten a tad less painful to go with a Ceton tuner in the past year because one enterprising person in the NPVR community has authored a piece of software called cetonproxy. This makes a Ceton tuner appear to be an HD Homerun device. That means I can either use NPVR 5.0 to handle tuning, or let Jellyfin do it directly. I’m not sure which way I’m going to go, but the tuner should be in tomorrow I hope, so I’ll post an update soon.
Today
I was finally able to issue a new SSL certificate today. I got HAProxy set up after reviewing a few tutorials and I figured out what I needed to do to make WordPress and my other services work though the reverse proxy. Some services need this code added to the “Backend pass thru” section of their backend configuration:
http-response set-header Content-Security-Policy upgrade-insecure-requestsSome need this added to the same section:
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }I don’t know why some need one or the other, but it works, so ¯\_(ツ)_/¯. I’ll have to read about it some time.
WordPress would load up mostly fine with the the first option, but some sections of the site, like the live appearance customizer wouldn’t load, and Firefox would tell me passwords may not be secure. This said to me that there was mixed http and https content being served up. This happens because WordPress doesn’t know it’s behind a reverse proxy that is handling SSL. All I had to do was add this code to the top of my wp-config.php file:
/** Make sure WordPress understands it's behind an SSL terminator */
define('FORCE_SSL_ADMIN', true);
define('FORCE_SSL_LOGIN', true);
if ($_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https')
$_SERVER['HTTPS']='on'; Next
Coming up this week, I need to get the TV tuner set up and get a torrent client going with Jackett, Radarr and Sonarr. The weather is going to be very hot and sometimes rainy after tomorrow, so it should be a great time to stay inside at the computer. I’m so happy to have the blog back after a month away. I really missed it.