It’s been a few weeks since I set up my pfSense router inside Proxmox inside a HP desktop computer. After I set it up, I noticed my internet speeds weren’t quite what I was getting with the Orbi acting as the router. With the Orbi, I generally got somewhere around 650-700 Mbps for downloads and 700-800 Mbps for uploads. With pfSense, I was getting around 520-550 Mbps for both. My internet service should be 1Gbps in both directions (actually a theoretical maximum of 940Mbps due to the way the network hardware works). I set up pfSense as the Great 2020 Work From Home was in full swing, so I thought maybe Verizon’s network had more concurrent users during the day slowing me down. I didn’t really think anything of it until today, when I was downloading a 100 GB game.
When I first setup pfSense, I told Proxmox to give it two NICs of the VirtIO paravirtualized type. When I get pfSense set up, I noticed it told me the speed of the two interfaces was 10 Gbps, and my web page loading times were very long. I assumed this was a duplex mismatch, and changed the NIC type to Intel E1000. Pages loaded just fine after that. It turns out it was a mistake to change the NIC type. VirtIO was the correct type, and the 10 Gbps speed was referring to the link to the Proxmox virtual switch, not the link to the internet or my physical Cisco switch. I changed back to VirtIO and disabled all hardware offloading in the System > Advanced > Networking settings of pfSense.
I also happened upon a Reddit post describing the same issue I had. I followed the directions to install ethtool and add one line like
post-up ethtool -K vmbr0 tx off
for each virtual and physical interface in /etc/network/interfaces.
I also discovered that while pfSense CPU usage was only in the single digits when doing web browsing, during speed tests and large downloads, it hit close to 100%. I resolved that by adding another CPU core in the Proxmox hardware configuration. CPU usage is now 70-80% during big downloads.
I fixed everything up with these changes. My downloads and uploads now easily hit their maximum possible speeds of 940 Mbps, at least when other internet usage is kept to a minimum. I wish I did this last year when we first got Fios because I never got the advertised gigabit speeds with the Orbi router. I guess the Orbi wasn’t designed to handle a gigabit WAN connection. pfSense handles it with no trouble, at least once it’s properly configured.
There’s been a flurry of activity, and a false start in the homelab in the last week. I made my shit-tier vertical mount rack system and put it in my office closet and got just about everything set up. On the other hand, I have some trouble with the HP server I bought.
The first step to get everything working was to run some cables into the office; two ethernet cables and on RG6 coaxial. All three cables go from the basement to the office along the outside of the house. The coax and one ethernet cable are attached to the ONT (optical network terminal, basically a modem but for fiber optic) in the basement. This supplies the main internet connection to the office. The coax is disconnected for now, but I might hook it up and put my TV tuner in the office. The second ethernet cable ends up connected to the Orbi satellite in the living room for wired backhaul.
The new cables are on the left. I would have run them with the electrical service and fiber optic cables, but I don’t have a ladder tall enough. Just ignore all the garbage on the ground. It’s not a crack house, I promise.
I added a wall plate under the office window to nicely terminate the wires coming in. I used a backless retrofit/old work box to hold everything in place.
Not quite straight, but it gets the job done.
The upper ethernet cable is the internet supply from the ONT, and it goes to the new pfSense router across the room.
The HP desktop is the “router.” The Orbi is now acting just as a wifi access point. The gold thing is a Raspberry Pi 4 B 2GB which is currently serving up this website.
The HP ProDesk 400 G1 (what a name 🙄) desktop has an HP NC365T four-port NIC that handles the in and out for pfSense. Speaking of the software, I’m actually virtualizing pfSense. I’m using Proxmox as the hypervisor. Proxmox is a common choice for homelabbers, but it doesn’t seem to be as popular as ESXi. Most homelabbers use the same hardware and software as their work does, and almost no business uses Proxmox. I picked Proxmox because it’s free and open source with no limitations on its capabilities. ESXi places limitations on what you can do with the free version of the software, and I don’t want to pay the yearly subscription to use everything. On the other hand, I probably don’t need everything in the paid version. Anyways, it’s Proxmox for now. I set up pfSense as a virtual machine within Proxmox and assigned it two ports from the NC365T to do the routing.
I’m also running a Pi-Hole on the HP desktop inside a Ubuntu virtual machine. I was initially using Debian, but I ran into problems that I may have erroneously attributed to Debian. I still have trouble with the Chrome browser on my desktop while running on Ubuntu. Firefox on the same computer works perfectly. I never had any problems with any browser when running on a Raspberry Pi. Pi-Hole had a big 5.0 update a couple weeks ago, so I might have to try Debian again sometime over the summer. For now, it gets the job done; the ad blocking is working normally.
The LAN port on the router is connected to a Cisco 3560G switch. I just finished a semester-long networking class with curriculum provided by Cisco (I got an A, by the way), so it seemed like a good idea to get a switch I was already familiar with. The switch basically distributes the LAN (and thus internet) access wherever it’s needed. The Orbi base station is plugged into the switch, and the base station is then connected to the satellite in the living room. These provide the wifi coverage for the house.
The shitty “rack” I put together in the closet. It works though, and the things are so much quieter in there than out in the open.
Pretty lights.
The network side of things is going great. The server on the other hand, is not. I installed the hard drives I ordered and put it in my rack, and now the RAID card doesn’t work. No matter what I do, I can’t get it to work. It gives me an error like the card itself is defective or not plugged in properly. I initially thought a dead battery for the card was causing the problem, so I bought a new card and battery, but got the same result. The only difference was adding the hard drives and moving the server. It worked perfectly fine two weeks ago on my shelf. The server was pretty cheap, so I ordered another identical one. Hopefully it doesn’t get killed. If you think you might be able to help me with my P420 controller woes, drop me a line here.
Overall, I’m happy with the set up so far. The only thing I’m a tad dissatisfied with is the wifi solution. The Orbi is a great mesh system for the consumer, but I find it a little lacking from my more somewhat more knowledgeable perspective. The big thing that’s missing from it is support for virtual LANs. I’d like to have three wifi networks: one for guests, one for things, like printers and smart speakers and the thermostat, and one for trusted devices like personal laptops and phones. VLANs would make this possible by allowing the three wifi networks to be on separate VLANs with separate routing and firewall rules to keep traffic out of the home network if needed. Commercial wifi gear like Ubiquiti is all about that stuff, and if I hadn’t purchased the Orbi stuff relatively recently, I’d probably look into some of those commercial access points. Maybe I’ll cruise around for some used ones on eBay some time.
