Tried OPNsense to replace old cisco router. Failed

I have been using a second hand Cisco 3800 series router running very old IOS software as my home gateway for the last few years. It's been very reliable and always performed extremely well. Apparently this model can switch almost 500,000 packets per second (with no other features configured) so it was always more than adequate for my needs.

Unfortunately it was very loud and took a lot of power to run, and since Cisco stopped providing updates for this model of router years ago I decided to look elsewhere for something a bit less bulky, a bit quieter, and a bit more modern.

It weighs as much as one of my children and it sounds like a jet engine
when you turn it on.

Naturally I wanted to be trendy and use an open source product! I wanted the power of being able to customize and control what my router was doing and not have to worry about my router secretly sending "troubleshooting data" (wink wink) back to the manufacturer.

Although I'm not a complete amateur when it comes to this sort of thing (I use OpenWRT on my access points and most of the little devices in my house run some kind of pre-hacked linux) I didn't want to build something from the ground up. I wanted some kind of "integrated" system that let me get things up and running and go from there.

I considered a few alternatives including OpenWRT, DD-WRT, pfSense, ipFire and others. All of these have a nice web based gui and lots of useful "add ons" and "plugins" that allow you to perform a great deal of customization.

I eventually settled on OPNSense which is based on the FreeBSD operating system.

I've never had much to do with the FreeBSD operating system before so I was a little hesitant, but after hearing about how strong and powerful the FreeBSD networking stack was meant to be, I decided to go with it. I was particularly convinced by a smart looking guy called George Neville-Neil who mentioned that the FreeBSD networking stack is what's being used by the Mars Pathfinder mission to communicate with Earth!!

George has written textbooks, spoken at conferences, has a cool beard and his
code is on a space ship....he's everything I want to be. 

Now for the exciting part of the post. The bit where things went wrong and I tell you why I failed. There are two reasons why OPNsense didn't work for me.

1) OPNSense is nothing like Cisco IOS

When I say this I don't mean "The commands weren't the same" (which is true). I mean that Cisco IOS is a truly integrated routing system where each part of the router software knows about all the other parts whereas OPNSense is nothing like that. In addition the commands and tools on a Cisco router generally follow a consistent style and pattern whereas each plugin in OPNsense is a kingdom unto itself.

OPNSense is essentially a shell that fits atop dozens of different programs that don't natively know anything about each other and were probably not written with many of the other programs in mind. This applies equally to other open source style routing systems like OpenWRT and pfSense. DD-WRT seems to be better integrated than its cousins, but consequently it has far less functionality and ability for customization.

In most cases this is fine because if you run a standard combination of services, then they'll have been tested adequately with each other and the OPNSense system can successfully integrate their functionalities together. However, if you stray off the beaten path a little, then that's when you run into trouble.

For example, in my network I need to route multicast traffic. It took me over a week to get it working with OPNsense and this is why. In some circumstances the OPNsense firewall component (pf) is programmed to transmit routed multicast traffic with an INCORRECT DESTINATION MAC ADDRESS WHICH MAKES IT ABSOLUTELY BLOODY USELESS (See OPNsense issue 3629). It seems like the maintainers of this part of the code has absolutely no plans to fix this issue.

This picture isn't here for any reason. I love
everybody and I am at peace with the universe.

I suspect that it isn't being fixed because multicast routing is quite rare in home networks and isn't necessarily super common in enterprise networks either so why would the maintainers spend time fixing this problem?

I came across about a dozen other feature interaction issues like this one. Most of them were trivial but a few were very tedious to overcome indeed.

Now I'm not saying that products made by big proprietary network vendors don't have problems, especially when weird combinations of features are used but I can guarantee that if something like multicast routing wasn't working then it would be fixed very very quickly.

I actually know a guy who worked for the now defunct Bay Networks router company.
You wouldn't believe some of the bugs he found!

2) FreeBSD is the OS/2 of the modern open source world

Are you old enough to remember OS/2? One of my professors introduced me to it back when Microsoft Windows was pretty much completely dominant in desktop computing.

It was great! It performed better and more reliably than Windows did, and it even ran (some) Windows applications! It's nowhere to be seen now.

If it's made by IBM then how can you go wrong?

The issue I had with it was that in those rare instances where I had a problem with OS/2, there was every possibility that I was the first and only human being on the planet to have experienced that particular issue because the user base of OS/2 wasn't what you'd call "vast".

And even though there were hundreds of brilliant people on forums and mailing lists who could probably help me, I was a "newb" and my requests for help weren't looked upon with much respect.

FreeBSD has kind of the same problem. Even though there are some very appealing aspects of FreeBSD, such as the package and ports system, and the extensive documentation of the base operating system, getting help isn't always easy. That seems to be because the user base is numerically much less than that of Linux, but also the FreeBSD user base all seem to be universally brilliant people who aren't always interested in helping "newbs".

These people aren't worthy to smell an experienced FreeBSD user's dirty socks!!

Now obviously there are also many arrogant people that maintain Linux features and core components as well, but since there are always plenty of new users being drawn in to the linuxverse, there's always someone out there willing to help, and likely many others who have had, or are having, the same problem as you!

Linus Torvalds, the creator of Linux, is widely acknowledged to be one of
the kindest and most patient men in the world.

What will I do?
To anyone who says I didn't give OPNsense enough of a chance, I actually ran it as my home's main gateway for a whole month as I tried to incrementally enable features I needed and troubleshot problems. I eventually had to give up after weeks of my OPNsense system's Ethernet interface randomly losing it's link about once every 36 hours.

I promise I did extensive troubleshooting. I replaced cables, moved switch ports and went through a large permutation of settings, driver updates and debugging but I could find no sign of why the problem was happening or what to do to fix it.

All of that being said, seeing how OPNsense works has given me some great insight into the tools that open source routing and firewall systems use. For that I am grateful.

I basically plan now to run a very basic linux system as my gateway (maybe on a Raspberry Pi?) but without a unifying Gui/web interface sitting above all the tools. I plan to just use the individual base programs like ipfw, dnsmasq and quagga and take control of the interactions between them myself.

Obviously this won't be as pretty as using a nicely integrated GUI but you have to admit, it will be very hardcore!

You know this guy edits his firewall config files with "vi" instead of "nano".

I will probably fail and go back to using my old hulk of a Cisco router, or buying a second hand TP-link router and installing OpenWRT on it, but I'm sure I'll learn a lot on the way. I think that's what my main goal is anyway.