Tag Archives | war stories

War Stories: Backup NICs, DNS and AD

A return to our sporadic series of networking war stories. This time it’s fun with dedicated backup networks, DNS auto-registration, and Active Directory. Thank God it’s a lot easier these days with virtualisation. But back then… Backups suck, but you need to do them somehow Back in the olden days we had a dedicated tape […]

Continue Reading

War Stories: Closing out Projects

We put a lot of energy into new projects. We argue about the design, we plan the cutover, we execute it…and then we move on. But decommissioning the old system is critical part of any project. It’s not over until you’ve switched off the old system. Years ago I was involved in the buildout of a […]

Continue Reading
256px-FireIcon

War Stories: ITIL Process vs Practice

Our irregular War Stories returns, with a story about a network I worked on with strict change control, but high technical debt. What should have been a simple fix became far more pain than it should have been. Lesson learned: next time just leave things alone. I’m sure the ITIL true believers loved their process, but did […]

Continue Reading

War Stories: Unix Security

A different kind of war story this time: Unix security blunders. Old-school Unix-types will mutter about how much more secure Unix systems are than Windows, but that glosses over a lot. In a former life I worked as an HP-UX sysadmin, and I saw some shocking default configurations. I liked HP-UX – so much better […]

Continue Reading
256px-FireIcon

War Stories: Cursed VLANs

I’ve written before about switch ports being permanently disabled. This time it’s something new to me: VLANs that refuse to forward frames. A Simple Network The network was pretty straightforward. A pair of firewalls connecting through a pair of switches to a pair of routers: Sub-interfaces were used on the routers and firewalls, with trunks […]

Continue Reading
256px-FireIcon

War Stories: Gratuitous ARP and VRRP

Continuing our theme of ARP-related war stories, here’s another ARP/switching behaviour I’ve come across. This particular problem didn’t result in any outages, but the network wasn’t working as well as it should have, and started flooding frames unexpectedly. Here’s what was going on: The Network Breaking the network down to its simplest level, it looked like this: The […]

Continue Reading
256px-FireIcon

War Stories: Dual-Vendor Firewall Strategy

Networks of the early 2000s often used multiple firewall vendors. This was done with the best of intentions, but it could make troubleshooting far more complicated. Here’s a weird bug we came across with one firewall, that was obscured by data we saw on another firewall. Dual-Vendor Good, Right? Maybe Not Years ago, we used to […]

Continue Reading
256px-FireIcon

War Stories: Check Point Meltdown

Firewalls are usually deployed as a cluster, to provide failover capabilities. Protocols such as VRRP are used to that traffic is normally routed via one node, but if that node fails, the other one automatically takes all traffic. Connection synchronisation ensures that the backup firewall is always aware of all active sessions, so that a […]

Continue Reading