pfsense – Patrick's Journal

The Dust Settles

After too many weeks of struggling and waiting and struggling some more, I finally have a functional server running almost all the services I was planning on, including this blog. Last we left off, about a month ago, I was rounding up the parts I needed to assemble a white label server. So here’s what I got.

The Hardware

Intel S2400SC motherboard: This is a dual socket LGA1356 motherboard with eight RAM slots total and a decent selection of PCI express slots. I picked it because it has PCIe x16 and x8 slots that would be capable of holding a graphics card. I initially planned to use the built-in SAS mini ports to handle my hard drives, but this didn’t end up working out. I used the same Xeon E5-2450 CPUs I bought for the HP server. I also decided to add 16GB more memory just in case.

Dell PERC H310: I flashed this popular and cheap SAS card to IT mode which allows it to be used as an HBA so I can use ZFS instead of hardware RAID.

Corsair RM850x power supply: I spent a tad more than I wanted to really, but I wanted an 80 Plus Gold power supply giving at least 850 watts with semi-modular cables at the minimum. They were all priced similarly, but in and out of stock, probably due to the pandemic.

HP NC365T: A popular and cheap 4-port gigabit network card.

Antec P101 case: I picked this because it billed itself as a quiet case, it came with four fans and it said it should hold an E-ATX motherboard, meaning my CEB motherboard shouldn’t be a problem.

Turned out, the CEB motherboard was a tiny problem. Despite what the internet said would happen, only three screw holes lined up with the standoffs in the case. The standoffs that didn’t line up were touching the motherboard, so they had to be removed. The motherboard was definitely not mounted securely with only three screws, and it was a bit too flexible with no standoffs behind it. There were some foam standoffs on the motherboard, but only two and they weren’t in the right places.

I moved and super glued the foam standoffs into new positions, and then I 3D printed a few more for the rest of the board.

The orange things are the 3D printed standoffs.

With that done, it was no trouble the get the board in and screwed down. I had to use a zip tie in the upper right corner, but it gets the job done just fine. Building in this case was alright I guess. I’m not a fan of the immovable power supply shroud, and I don’t really care for the tool-less drive bays. On the other hand, there wasn’t much choice for an E-ATX case at this price.

With the basic components assembled, it was time to install the hard drives and get going. Flashing the H310 to IT mode was a piece of cake and not worth talking about more. I bought two Seagate 1TB SAS drives to use in RAIDZ for my hypervisor. For some reason, these are incompatible with the H310. It knows the drives are connected, but doesn’t pass them though to the OS. I couldn’t get to them in Proxmox or a GParted live USB drive. It might be the fault of the hard drives, and not the H310, but I don’t have any other SAS equipped devices to test that theory out. So I bought another copy of an old 750GB SATA drive I had laying around. That worked just fine.

The Software

With the computer finally up and running, it was time to get some services installed. I got WordPress installed first, and it was running fine when it was being accessed via port forwarding on my router. I realized I was going to need a reverse proxy to handle access to multiple services on a single domain name. pfSense has HAProxy available in its repositories, so I decided to go with that. It’s a little convoluted to set up if you have no experience with reverse proxies, but after watching a couple videos and reading a few articles, I got it going. There was one problem though; no formatting (CSS and the like) was being applied to web pages accessed through the reverse proxy.

I found a few solutions on the web, but I couldn’t figure out how to implement them. They all involved changing the HAProxy configuration, which wasn’t a problem. It seems like most people run these reverse proxies on a separate virtual machine, not on pfSense, so their configuration was done in a text file rather than a web interface. Forum posters were being told to add a couple lines of code to the config files, but I wasn’t able to do that on pfSense. I decided to come back to that later.

After adjusting some HAProxy settings to try to get a reverse proxy going for my local services, I accidentally locked myself out of the pfSense web interface. No problem, I thought, I’ll just roll back to the most recent snapshot. So I did, and I was back in. That also wiped out the new SSL certificate I made earlier in the day. Again, not a problem I thought, I’ll just issue it again. Turns out I had already issued all the duplicate certificates I was allowed for the week, and I couldn’t get a new one until the next week (today).

In the Meantime

While I was waiting for the timer to run out on the certificate, I decided to press on with some other services. I set up a wiki to keep track of my network and installation notes for things. I decided to would be fun to try out an RSS reader, so I got Tiny Tiny RSS going.

A big part of setting up this server was for media serving with Plex, or something like it. I looked into the options, and it seems the three most popular, in descending order, are Plex, Emby and Jellyfin. I was originally planning on Plex, because it has the best name recognition and I knew it could do what I wanted. I tried out Emby before, and I was considering it this time. I hadn’t heard of Jellyfin before, but people seemed to like it. I axed Plex because while most commenters said it did a great job of serving up media, many useful functions are locked behind a not-inexpensive premium tier. The same is true of Emby, and Emby has an additional demerit with its licensing. Basically, Emby used to be open source, and the developers suddenly decided to go closed source. I don’t necessarily have a problem with closed-source software, but going from open to closed, and so abruptly is a problem to me. Jellyfin picked up the pieces and is based on the last open-source Emby code available. It’s not totally 100% quite yet, but it does everything I need.

From the start, this media server was supposed to take over the live TV services from the living room computer. Things were looking promising at first. Jellyfin supports live TV, and using NextPVR as a backend, so I was all set with my stupid Ceton tuner. I put the tuner in (with no drama like the HP server had) and set up PCI passthrough to a Windows VM for NPVR. I connected Jellyfin to that and things were working perfectly, with absolutely no fuss. I seemed too good to be true. And it was.

While the tuner worked perfectly (better than the bare-metal installation on the living room computer) with the Windows VM turned on, things went south when I shut the VM off. Shutting the VM down crashed the whole server. It was an abrupt shutdown, like yanking the power cord. That wouldn’t work. If it was a graceful shutdown, like clicking the button in Proxmox, maybe I could have dealt with it. Maybe not with the frequency with which Windows needs to reboot for updates.

The only solution to letting the new server handle the TV services was to get a new tuner. Pickin’s are pretty slim when it comes to cable card tuners these days. You can either have an old (used) HD Homerun, but they’re really expensive and only have three tuners, or you can have a stupid Ceton which has more tuners, but has very little software support (probably because the company doesn’t exist anymore). I picked a Ceton InfiniTV 6 Eth. This is an ethernet tuner similar in style to an HD Homerun, except worse. It’s gotten a tad less painful to go with a Ceton tuner in the past year because one enterprising person in the NPVR community has authored a piece of software called cetonproxy. This makes a Ceton tuner appear to be an HD Homerun device. That means I can either use NPVR 5.0 to handle tuning, or let Jellyfin do it directly. I’m not sure which way I’m going to go, but the tuner should be in tomorrow I hope, so I’ll post an update soon.

Today

I was finally able to issue a new SSL certificate today. I got HAProxy set up after reviewing a few tutorials and I figured out what I needed to do to make WordPress and my other services work though the reverse proxy. Some services need this code added to the “Backend pass thru” section of their backend configuration:

http-response set-header Content-Security-Policy upgrade-insecure-requests

Some need this added to the same section:

http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }

I don’t know why some need one or the other, but it works, so ¯\_(ツ)_/¯. I’ll have to read about it some time.

WordPress would load up mostly fine with the the first option, but some sections of the site, like the live appearance customizer wouldn’t load, and Firefox would tell me passwords may not be secure. This said to me that there was mixed http and https content being served up. This happens because WordPress doesn’t know it’s behind a reverse proxy that is handling SSL. All I had to do was add this code to the top of my wp-config.php file:

/** Make sure WordPress understands it's behind an SSL terminator */
define('FORCE_SSL_ADMIN', true);
define('FORCE_SSL_LOGIN', true);
if ($_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https')
$_SERVER['HTTPS']='on';

Coming up this week, I need to get the TV tuner set up and get a torrent client going with Jackett, Radarr and Sonarr. The weather is going to be very hot and sometimes rainy after tomorrow, so it should be a great time to stay inside at the computer. I’m so happy to have the blog back after a month away. I really missed it.

Fixing Internet Speed on Virtualized pfSense

It’s been a few weeks since I set up my pfSense router inside Proxmox inside a HP desktop computer. After I set it up, I noticed my internet speeds weren’t quite what I was getting with the Orbi acting as the router. With the Orbi, I generally got somewhere around 650-700 Mbps for downloads and 700-800 Mbps for uploads. With pfSense, I was getting around 520-550 Mbps for both. My internet service should be 1Gbps in both directions (actually a theoretical maximum of 940Mbps due to the way the network hardware works). I set up pfSense as the Great 2020 Work From Home was in full swing, so I thought maybe Verizon’s network had more concurrent users during the day slowing me down. I didn’t really think anything of it until today, when I was downloading a 100 GB game.

When I first setup pfSense, I told Proxmox to give it two NICs of the VirtIO paravirtualized type. When I get pfSense set up, I noticed it told me the speed of the two interfaces was 10 Gbps, and my web page loading times were very long. I assumed this was a duplex mismatch, and changed the NIC type to Intel E1000. Pages loaded just fine after that. It turns out it was a mistake to change the NIC type. VirtIO was the correct type, and the 10 Gbps speed was referring to the link to the Proxmox virtual switch, not the link to the internet or my physical Cisco switch. I changed back to VirtIO and disabled all hardware offloading in the System > Advanced > Networking settings of pfSense.

I also happened upon a Reddit post describing the same issue I had. I followed the directions to install ethtool and add one line like

post-up ethtool -K vmbr0 tx off

for each virtual and physical interface in /etc/network/interfaces.

I also discovered that while pfSense CPU usage was only in the single digits when doing web browsing, during speed tests and large downloads, it hit close to 100%. I resolved that by adding another CPU core in the Proxmox hardware configuration. CPU usage is now 70-80% during big downloads.

I fixed everything up with these changes. My downloads and uploads now easily hit their maximum possible speeds of 940 Mbps, at least when other internet usage is kept to a minimum. I wish I did this last year when we first got Fios because I never got the advertised gigabit speeds with the Orbi router. I guess the Orbi wasn’t designed to handle a gigabit WAN connection. pfSense handles it with no trouble, at least once it’s properly configured.

Homelab Underway

There’s been a flurry of activity, and a false start in the homelab in the last week. I made my shit-tier vertical mount rack system and put it in my office closet and got just about everything set up. On the other hand, I have some trouble with the HP server I bought.

The first step to get everything working was to run some cables into the office; two ethernet cables and on RG6 coaxial. All three cables go from the basement to the office along the outside of the house. The coax and one ethernet cable are attached to the ONT (optical network terminal, basically a modem but for fiber optic) in the basement. This supplies the main internet connection to the office. The coax is disconnected for now, but I might hook it up and put my TV tuner in the office. The second ethernet cable ends up connected to the Orbi satellite in the living room for wired backhaul.

The new cables are on the left. I would have run them with the electrical service and fiber optic cables, but I don’t have a ladder tall enough. Just ignore all the garbage on the ground. It’s not a crack house, I promise.

I added a wall plate under the office window to nicely terminate the wires coming in. I used a backless retrofit/old work box to hold everything in place.

Not quite straight, but it gets the job done.

The upper ethernet cable is the internet supply from the ONT, and it goes to the new pfSense router across the room.

The HP desktop is the “router.” The Orbi is now acting just as a wifi access point. The gold thing is a Raspberry Pi 4 B 2GB which is currently serving up this website.

The HP ProDesk 400 G1 (what a name 🙄) desktop has an HP NC365T four-port NIC that handles the in and out for pfSense. Speaking of the software, I’m actually virtualizing pfSense. I’m using Proxmox as the hypervisor. Proxmox is a common choice for homelabbers, but it doesn’t seem to be as popular as ESXi. Most homelabbers use the same hardware and software as their work does, and almost no business uses Proxmox. I picked Proxmox because it’s free and open source with no limitations on its capabilities. ESXi places limitations on what you can do with the free version of the software, and I don’t want to pay the yearly subscription to use everything. On the other hand, I probably don’t need everything in the paid version. Anyways, it’s Proxmox for now. I set up pfSense as a virtual machine within Proxmox and assigned it two ports from the NC365T to do the routing.

I’m also running a Pi-Hole on the HP desktop inside a Ubuntu virtual machine. I was initially using Debian, but I ran into problems that I may have erroneously attributed to Debian. I still have trouble with the Chrome browser on my desktop while running on Ubuntu. Firefox on the same computer works perfectly. I never had any problems with any browser when running on a Raspberry Pi. Pi-Hole had a big 5.0 update a couple weeks ago, so I might have to try Debian again sometime over the summer. For now, it gets the job done; the ad blocking is working normally.

The LAN port on the router is connected to a Cisco 3560G switch. I just finished a semester-long networking class with curriculum provided by Cisco (I got an A, by the way), so it seemed like a good idea to get a switch I was already familiar with. The switch basically distributes the LAN (and thus internet) access wherever it’s needed. The Orbi base station is plugged into the switch, and the base station is then connected to the satellite in the living room. These provide the wifi coverage for the house.

The shitty “rack” I put together in the closet. It works though, and the things are so much quieter in there than out in the open.

The network side of things is going great. The server on the other hand, is not. I installed the hard drives I ordered and put it in my rack, and now the RAID card doesn’t work. No matter what I do, I can’t get it to work. It gives me an error like the card itself is defective or not plugged in properly. I initially thought a dead battery for the card was causing the problem, so I bought a new card and battery, but got the same result. The only difference was adding the hard drives and moving the server. It worked perfectly fine two weeks ago on my shelf. The server was pretty cheap, so I ordered another identical one. Hopefully it doesn’t get killed. If you think you might be able to help me with my P420 controller woes, drop me a line here.

Overall, I’m happy with the set up so far. The only thing I’m a tad dissatisfied with is the wifi solution. The Orbi is a great mesh system for the consumer, but I find it a little lacking from my more somewhat more knowledgeable perspective. The big thing that’s missing from it is support for virtual LANs. I’d like to have three wifi networks: one for guests, one for things, like printers and smart speakers and the thermostat, and one for trusted devices like personal laptops and phones. VLANs would make this possible by allowing the three wifi networks to be on separate VLANs with separate routing and firewall rules to keep traffic out of the home network if needed. Commercial wifi gear like Ubiquiti is all about that stuff, and if I hadn’t purchased the Orbi stuff relatively recently, I’d probably look into some of those commercial access points. Maybe I’ll cruise around for some used ones on eBay some time.