Dang, about five months since my last update. In my defense, I’ve been pretty busy. I got my first real career job as a help desk tech at a school district. It’s been about three months in this job so far and I think it’s going fairly well. I’m probably 90% up to speed now; I know most of what I need to know and the rest is relatively minor stuff. I felt a little bit of imposter syndrome in the first couple weeks because I’ve never been a tech support person in a professional setting before. Getting a ticket and just going there and clicking around on someone’s computer until the problem is solved and getting paid for it was kind of surreal. I’m used to it now. I think I could probably communicate a bit more confidence to the people I’m helping; I don’t want to say I can fix something immediately for sure until I see the problem, but “I’ll see what I can do” isn’t the best thing to say to someone who really needs their computer to work normally so they can do their job. I might have to see if I can come up with another way to under-promise and over-deliver. Or maybe I should just say “I’ll come over and fix it.”
A lot of IT people, or maybe even most, start out on the help desk. It’s a good learning experience, it’s usually not too difficult as long as the customers are nice, and someone has to do the job. The pay isn’t horrible, but it’s not great either. Help desk staff are definitely the lowest-paid employees under the IT umbrella. I don’t want to stay on the help desk forever (no one does) and help desk has nothing to do with my degree, which is in networking. I’d be much more interested in a network engineer or network admin type job, but I need some experience and probably a certification first.
I’m getting more experience by the day, I have a degree (just an associate’s, but that’s better than nothing), but I don’t have any certs yet. The classes I took in college align roughly with the Cisco Certified Network Associate (CCNA) cert. This is the entry-level networking cert, but combined with the degree and a modicum of professional and personal experience, I think getting this cert will open up the next tier of IT jobs, and thus salaries, to me. I’m planning on studying up this summer and hopefully taking the test by the fall. Afterwards, I’ll have to see what jobs are out there. I’m hoping for a 40-50% salary increase with my next jump.
Somewhat longer-term, in the next year or so I’d like to get at least one more cert. The Red Hat Certified System Administrator (for Linux) or the Windows Server Hybrid Administrator Associate (for Windows), or maybe even both. I’m more interested in Linux personally, but the Windows cert might open more jobs in my area. Maybe I’d wait to move jobs until I can get either of those, but it never hurts to throw applications out there.
Longer-term than that, my career goal for right now is to make $100,000 a year by the time I’m 40-years-old. I have 13.5 years to go. I think it’s probably attainable with some luck and hard (ish) work.
To this end, I’ve started getting some more stuff going on my server to practice for these things. One thing that seems to be pretty important for IT jobs is experience in Active Directory. At the help desk, there isn’t much need for techs to delve into AD for anything other than a password reset. Setting up users, groups and OUs isn’t really under the purview of the help desk; it’s a systems administrator’s job to do all that. To help me get some experience, I got a Windows Server domain controller going and joined a couple virtual machines to it. I’m practicing things like deploying software, creating network shares, using Group Policy settings, and I’m getting a bit of experience with PowerShell. That last one is pretty important for a good sys admin. Windows doesn’t have any built-in bulk import tools or tools for regularly updating user groups, so it has to be done with PowerShell scripts. So far I’m mostly copy-pasting, but I’ll probably check out a Udemy class sometime to learn more.
So that’s basically it. Mostly job stuff these days. I’m going to try to post here once in a while to record what I’m doing. It might make me look like a great candidate for a new job some day.
After too many weeks of struggling and waiting and struggling some more, I finally have a functional server running almost all the services I was planning on, including this blog. Last we left off, about a month ago, I was rounding up the parts I needed to assemble a white label server. So here’s what I got.
The Hardware
Intel S2400SC motherboard: This is a dual socket LGA1356 motherboard with eight RAM slots total and a decent selection of PCI express slots. I picked it because it has PCIe x16 and x8 slots that would be capable of holding a graphics card. I initially planned to use the built-in SAS mini ports to handle my hard drives, but this didn’t end up working out. I used the same Xeon E5-2450 CPUs I bought for the HP server. I also decided to add 16GB more memory just in case.
Dell PERC H310: I flashed this popular and cheap SAS card to IT mode which allows it to be used as an HBA so I can use ZFS instead of hardware RAID.
Corsair RM850x power supply: I spent a tad more than I wanted to really, but I wanted an 80 Plus Gold power supply giving at least 850 watts with semi-modular cables at the minimum. They were all priced similarly, but in and out of stock, probably due to the pandemic.
HP NC365T: A popular and cheap 4-port gigabit network card.
Antec P101 case: I picked this because it billed itself as a quiet case, it came with four fans and it said it should hold an E-ATX motherboard, meaning my CEB motherboard shouldn’t be a problem.
Turned out, the CEB motherboard was a tiny problem. Despite what the internet said would happen, only three screw holes lined up with the standoffs in the case. The standoffs that didn’t line up were touching the motherboard, so they had to be removed. The motherboard was definitely not mounted securely with only three screws, and it was a bit too flexible with no standoffs behind it. There were some foam standoffs on the motherboard, but only two and they weren’t in the right places.
I moved and super glued the foam standoffs into new positions, and then I 3D printed a few more for the rest of the board.
The orange things are the 3D printed standoffs.
With that done, it was no trouble the get the board in and screwed down. I had to use a zip tie in the upper right corner, but it gets the job done just fine. Building in this case was alright I guess. I’m not a fan of the immovable power supply shroud, and I don’t really care for the tool-less drive bays. On the other hand, there wasn’t much choice for an E-ATX case at this price.
Mmmmmm, that dual-socket goodness.
With the basic components assembled, it was time to install the hard drives and get going. Flashing the H310 to IT mode was a piece of cake and not worth talking about more. I bought two Seagate 1TB SAS drives to use in RAIDZ for my hypervisor. For some reason, these are incompatible with the H310. It knows the drives are connected, but doesn’t pass them though to the OS. I couldn’t get to them in Proxmox or a GParted live USB drive. It might be the fault of the hard drives, and not the H310, but I don’t have any other SAS equipped devices to test that theory out. So I bought another copy of an old 750GB SATA drive I had laying around. That worked just fine.
The Software
With the computer finally up and running, it was time to get some services installed. I got WordPress installed first, and it was running fine when it was being accessed via port forwarding on my router. I realized I was going to need a reverse proxy to handle access to multiple services on a single domain name. pfSense has HAProxy available in its repositories, so I decided to go with that. It’s a little convoluted to set up if you have no experience with reverse proxies, but after watching a couple videos and reading a few articles, I got it going. There was one problem though; no formatting (CSS and the like) was being applied to web pages accessed through the reverse proxy.
I found a few solutions on the web, but I couldn’t figure out how to implement them. They all involved changing the HAProxy configuration, which wasn’t a problem. It seems like most people run these reverse proxies on a separate virtual machine, not on pfSense, so their configuration was done in a text file rather than a web interface. Forum posters were being told to add a couple lines of code to the config files, but I wasn’t able to do that on pfSense. I decided to come back to that later.
After adjusting some HAProxy settings to try to get a reverse proxy going for my local services, I accidentally locked myself out of the pfSense web interface. No problem, I thought, I’ll just roll back to the most recent snapshot. So I did, and I was back in. That also wiped out the new SSL certificate I made earlier in the day. Again, not a problem I thought, I’ll just issue it again. Turns out I had already issued all the duplicate certificates I was allowed for the week, and I couldn’t get a new one until the next week (today).
In the Meantime
While I was waiting for the timer to run out on the certificate, I decided to press on with some other services. I set up a wiki to keep track of my network and installation notes for things. I decided to would be fun to try out an RSS reader, so I got Tiny Tiny RSS going.
A big part of setting up this server was for media serving with Plex, or something like it. I looked into the options, and it seems the three most popular, in descending order, are Plex, Emby and Jellyfin. I was originally planning on Plex, because it has the best name recognition and I knew it could do what I wanted. I tried out Emby before, and I was considering it this time. I hadn’t heard of Jellyfin before, but people seemed to like it. I axed Plex because while most commenters said it did a great job of serving up media, many useful functions are locked behind a not-inexpensive premium tier. The same is true of Emby, and Emby has an additional demerit with its licensing. Basically, Emby used to be open source, and the developers suddenly decided to go closed source. I don’t necessarily have a problem with closed-source software, but going from open to closed, and so abruptly is a problem to me. Jellyfin picked up the pieces and is based on the last open-source Emby code available. It’s not totally 100% quite yet, but it does everything I need.
From the start, this media server was supposed to take over the live TV services from the living room computer. Things were looking promising at first. Jellyfin supports live TV, and using NextPVR as a backend, so I was all set with my stupid Ceton tuner. I put the tuner in (with no drama like the HP server had) and set up PCI passthrough to a Windows VM for NPVR. I connected Jellyfin to that and things were working perfectly, with absolutely no fuss. I seemed too good to be true. And it was.
While the tuner worked perfectly (better than the bare-metal installation on the living room computer) with the Windows VM turned on, things went south when I shut the VM off. Shutting the VM down crashed the whole server. It was an abrupt shutdown, like yanking the power cord. That wouldn’t work. If it was a graceful shutdown, like clicking the button in Proxmox, maybe I could have dealt with it. Maybe not with the frequency with which Windows needs to reboot for updates.
The only solution to letting the new server handle the TV services was to get a new tuner. Pickin’s are pretty slim when it comes to cable card tuners these days. You can either have an old (used) HD Homerun, but they’re really expensive and only have three tuners, or you can have a stupid Ceton which has more tuners, but has very little software support (probably because the company doesn’t exist anymore). I picked a Ceton InfiniTV 6 Eth. This is an ethernet tuner similar in style to an HD Homerun, except worse. It’s gotten a tad less painful to go with a Ceton tuner in the past year because one enterprising person in the NPVR community has authored a piece of software called cetonproxy. This makes a Ceton tuner appear to be an HD Homerun device. That means I can either use NPVR 5.0 to handle tuning, or let Jellyfin do it directly. I’m not sure which way I’m going to go, but the tuner should be in tomorrow I hope, so I’ll post an update soon.
Today
I was finally able to issue a new SSL certificate today. I got HAProxy set up after reviewing a few tutorials and I figured out what I needed to do to make WordPress and my other services work though the reverse proxy. Some services need this code added to the “Backend pass thru” section of their backend configuration:
I don’t know why some need one or the other, but it works, so ¯\_(ツ)_/¯. I’ll have to read about it some time.
WordPress would load up mostly fine with the the first option, but some sections of the site, like the live appearance customizer wouldn’t load, and Firefox would tell me passwords may not be secure. This said to me that there was mixed http and https content being served up. This happens because WordPress doesn’t know it’s behind a reverse proxy that is handling SSL. All I had to do was add this code to the top of my wp-config.php file:
/** Make sure WordPress understands it's behind an SSL terminator */
define('FORCE_SSL_ADMIN', true);
define('FORCE_SSL_LOGIN', true);
if ($_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https')
$_SERVER['HTTPS']='on';
Next
Coming up this week, I need to get the TV tuner set up and get a torrent client going with Jackett, Radarr and Sonarr. The weather is going to be very hot and sometimes rainy after tomorrow, so it should be a great time to stay inside at the computer. I’m so happy to have the blog back after a month away. I really missed it.
It’s been a few weeks since I set up my pfSense router inside Proxmox inside a HP desktop computer. After I set it up, I noticed my internet speeds weren’t quite what I was getting with the Orbi acting as the router. With the Orbi, I generally got somewhere around 650-700 Mbps for downloads and 700-800 Mbps for uploads. With pfSense, I was getting around 520-550 Mbps for both. My internet service should be 1Gbps in both directions (actually a theoretical maximum of 940Mbps due to the way the network hardware works). I set up pfSense as the Great 2020 Work From Home was in full swing, so I thought maybe Verizon’s network had more concurrent users during the day slowing me down. I didn’t really think anything of it until today, when I was downloading a 100 GB game.
When I first setup pfSense, I told Proxmox to give it two NICs of the VirtIO paravirtualized type. When I get pfSense set up, I noticed it told me the speed of the two interfaces was 10 Gbps, and my web page loading times were very long. I assumed this was a duplex mismatch, and changed the NIC type to Intel E1000. Pages loaded just fine after that. It turns out it was a mistake to change the NIC type. VirtIO was the correct type, and the 10 Gbps speed was referring to the link to the Proxmox virtual switch, not the link to the internet or my physical Cisco switch. I changed back to VirtIO and disabled all hardware offloading in the System > Advanced > Networking settings of pfSense.
I also happened upon a Reddit post describing the same issue I had. I followed the directions to install ethtool and add one line like
post-up ethtool -K vmbr0 tx off
for each virtual and physical interface in /etc/network/interfaces.
I also discovered that while pfSense CPU usage was only in the single digits when doing web browsing, during speed tests and large downloads, it hit close to 100%. I resolved that by adding another CPU core in the Proxmox hardware configuration. CPU usage is now 70-80% during big downloads.
I fixed everything up with these changes. My downloads and uploads now easily hit their maximum possible speeds of 940 Mbps, at least when other internet usage is kept to a minimum. I wish I did this last year when we first got Fios because I never got the advertised gigabit speeds with the Orbi router. I guess the Orbi wasn’t designed to handle a gigabit WAN connection. pfSense handles it with no trouble, at least once it’s properly configured.
I’ve acquired a everything I need to start my set up and I’ve been playing with it for about a week now.
The HP server is great so far, except for the noise. I know enterprise servers are probably designed with no thought given to noise levels, but Jesus, this thing is ridiculous. During the entire minute-plus POST process, the six fans run at their maximum speed of something like 12,000 rpm. It’s loud. If I was near a rack full of them all day, I’d definitely be wearing some ear protection. Once the fans settle down to 35-40% when the thing is idling, they’re bearable, but still too loud to have on an open shelf in the office. I’ve decided to make a redneck “rack” to suspend the server and the switch vertically in the office closet. I’ll be making it this week, so I’ll be sure to post some pictures. In other news, the server turned out to take 3.5″ drives, not 2.5″, which really pleases me. I found some used 3TB HGST SAS drives and got six of them, plus some drive trays. I’ll be doing a RAID 6 array, so I should have 12TB of total storage while being able to recover from two drives failing simultaneously.
I got a Cisco 3560G 48-port switch to connect everything. I decided I had to update it, and killed it somehow. The flash memory appears to be wrecked. I used the web admin page to try to update it, and that image might have been too large for the flash memory. I tried to format the memory and install a new OS over the serial connection, but I had no success. So I bought another switch. The same model, but this one has already been updated to the latest supported version of IOS and it has a one year warranty. I won’t even be thinking about updating this one.
The SFF HP desktop as a router is coming along just fine. I dug up an old hard drive from a MacBook Pro I flipped a few years ago to use as the storage for that. At 250GB, it should be more than plenty. The system came with only 4GB of RAM, so I ordered another 4GB stick that should be in this week. I could probably get by on 4GB, but why not double it for like $12? I installed an HP NC365T NIC to give that computer a total of five gigabit ethernet ports. I’ll be using the built in port for the Proxmox admin console, then three of the four ports on the HP card for pfSense and Pi-Hole. I also need to get a VPN running, and I’d like to use WireGuard, which I may be able to do right in pfSense. If not, I’ll get a third VM going on this router box to handle VPN duties and use up the last ethernet port.
I’ll be running some wires to get wired internet and cable TV from the basement to the second-floor office. It shouldn’t be too difficult, but I’ve never run wires on the outside of a house before. Theoretically, all I need to do is add a couple holes to the area where the electric service and cable/fiber connections enter the house from the outside, put the wires through there and then drill a couple holes in a wall of the office. I’m hoping to not have to terminate my own ethernet cables because it’s a real pain. Monoprice has some outdoor rated cables with RJ45 connectors already attached for a great price, so I’m going to try to drill a hole big enough to let the connector through. The coax cable for the TV signal is no problem though, I’ve used compression-fit connectors on those a million times.
I’ll be ordering all my cables early in the week, so hopefully I can get drilling on the weekend.
A little while ago, I stumbled upon the r/homelab subreddit. There, users gather to discuss their home network setups, often used for experimentation like a laboratory. I finally have enough money to get started on a modest set up, so I ordered some pieces this week to get started.
I’m currently taking classes to get a degree in computer networking. One class is an introduction to computing class that has us using virtual machines for something. Another is a networking class that should prepare me for the Cisco CCENT certification exam. I’ve become a lot more interested in the subjects thanks to the classes. At the same time, my home network needs have changed and I could use some more power and storage.
Enter the homelab. Following some guidance from the r/homelab wiki, I decided on a basic set up. For hardware, I decided to get an HP DL380e Gen8 server, a Cisco Catalyst 3560G 48-port switch, and an HP ProDesk desktop. The plan is to use a hypervisor (probably Proxmox) on the ProDesk so it can act as a pfSense router, Pi-Hole ad blocker, VPN (hopefully with WireGuard), and reverse proxy (probably Caddy) all at the same time. This router will be connected to the Cisco switch, where I may set up some virtual LANs. I’ll have to see how everything works together. I got the DL380e as a barebones thing, so the specs were up to me. I decided to go with dual Xeon E5-2450 processors, and I’ll be getting 48 GB of RAM (the maximum is 384 GB). These are both pretty cheap options. I think I found a good deal on some 1TB 2.5″ SAS hard drives, so I will probably start with six of them in a RAID 6 configuration and add more as needed. I’m not totally certain about this though. The server will be home to a few virtual machines. I’ll be moving my TV tuner card there, so there will be at least one Windows 10 VM. I’m also going to be running my normal website from there, and I’ll probably run an OctoPrint setup so I can control my 3D printer, so I’ll need at least two Linux VMs for that. I’ll probably also throw in one for Arch Linux, just because I like to tinker with it. Beyond that, I’m not sure what else I’m going to do with it yet. Maybe host some game servers for Minecraft or something or get a media library going. I’d really, really love to put my spare RX 480 graphics card in it and stream some games. This is totally possible, but I need a very specific PCI riser card to fit a double slot graphics card, plus a power wire adapter and maybe an extra power supply. We’ll see how it goes and maybe I’ll try to track down the special riser in the future.
There’s no real purpose to doing this other than I want to. The experience with Cisco networking and virtual machines might help me get a job some time in the future. It’s not super expensive at least, and it’ll be fun to have a ton of computer power at my disposal.
