In system design, be it a network, application or any other piece of automated infrastructure, we eschew single points of failure.  We know that in the real world, things go wrong, and consequently we design systems that have redundancy built in.  If something fails, the system can transfer operations over to a redundant subsystem, and keep on going.  That’s why, for example, websites have back-up load balancers or data servers.  These apparently redundant elements are there to keep the system online if the primary subsystem breaks.

Unfortunately, in the I.T. world, it is common to neglect that the human operators of automated systems are also effectively part of that system.  They can also constitute a single point of failure, and we should be avoiding this problem with humans as well as machines, for the same reasons.

No single operator (be they employee or principal) should ever control exclusive passwords or knowledge about a critical system. To do so makes the system fragile, and sets it up for the kind of snafus that are currently plaguing San Francisco.

In the linked article, it’s described how the network administrator in question was unwilling to allow anyone else to work with the network because he felt they were incompetent, and the configuration for the network was extremely complicated.  I can’t help thinking that centralizing control over the network was a band-aid on a tumor, however.  Essentially the city was running a network that was too complicated for them to staff properly, and was relying on a bad management decision in order to cover for it.

Essentially this is trading risk for cost.  However, it raises the question about whether it was acceptable risk and whether the decision was made consciously, or if it just happened through the ignorance of the network administrator’s supervisors.  I’m betting the latter.   This was sweeping the problem under the carpet, and the city of San Francisco is now paying the cost.  Bad policy.