It’s been a few weeks since I set up my pfSense router inside Proxmox inside a HP desktop computer. After I set it up, I noticed my internet speeds weren’t quite what I was getting with the Orbi acting as the router. With the Orbi, I generally got somewhere around 650-700 Mbps for downloads and 700-800 Mbps for uploads. With pfSense, I was getting around 520-550 Mbps for both. My internet service should be 1Gbps in both directions (actually a theoretical maximum of 940Mbps due to the way the network hardware works). I set up pfSense as the Great 2020 Work From Home was in full swing, so I thought maybe Verizon’s network had more concurrent users during the day slowing me down. I didn’t really think anything of it until today, when I was downloading a 100 GB game.
When I first setup pfSense, I told Proxmox to give it two NICs of the VirtIO paravirtualized type. When I get pfSense set up, I noticed it told me the speed of the two interfaces was 10 Gbps, and my web page loading times were very long. I assumed this was a duplex mismatch, and changed the NIC type to Intel E1000. Pages loaded just fine after that. It turns out it was a mistake to change the NIC type. VirtIO was the correct type, and the 10 Gbps speed was referring to the link to the Proxmox virtual switch, not the link to the internet or my physical Cisco switch. I changed back to VirtIO and disabled all hardware offloading in the System > Advanced > Networking settings of pfSense.
I also happened upon a Reddit post describing the same issue I had. I followed the directions to install ethtool and add one line like
post-up ethtool -K vmbr0 tx offfor each virtual and physical interface in /etc/network/interfaces.
I also discovered that while pfSense CPU usage was only in the single digits when doing web browsing, during speed tests and large downloads, it hit close to 100%. I resolved that by adding another CPU core in the Proxmox hardware configuration. CPU usage is now 70-80% during big downloads.
I fixed everything up with these changes. My downloads and uploads now easily hit their maximum possible speeds of 940 Mbps, at least when other internet usage is kept to a minimum. I wish I did this last year when we first got Fios because I never got the advertised gigabit speeds with the Orbi router. I guess the Orbi wasn’t designed to handle a gigabit WAN connection. pfSense handles it with no trouble, at least once it’s properly configured.