Well, That Was Fun....

     Except it wasn't fun.  I was watching the Sunday morning political shows, mostly, and dozing, a little, and eventually decided to make some brunch.

     It was going to be pretty good brunch, too -- roast beef hash with canned mild chilies, a little onion, a dab or more of Chicago-style Giardiniera, a bread-crumb crust and eggs cooked on top, served with cheese.  But it wasn't to be.  I turned on the kitchen TV and the channel I wanted wasn't there.  It was one of the ones I'm responsible for, though not one of the big ones.

     So much for breakfast!  After a little more checking, I fired up my computer and started troubleshooting remotely, texting my boss at the same time.  He was aware of the problem, but busy with other things.

     I didn't find anything easy, so I brushed my teeth, got dressed, packed some snacks and went to the transmitter.  A redundant communications link had half-failed, which should be a minor problem, something that makes no impact and you solve during regular business hours.

     Yes, it's no problem -- unless the automatic switch that makes it redundant happens to have been hooked up backwards.  I looked and looked, and it didn't occur to me for a long time, until I was peering at the back of the conglomeration, seeing green lights where there should have been red lights, and red lights where they should have been green!

     You see, if you have two identical widgets, call them A and B, and a smart switch that selects between them, it's all well and good; if A is running and it conks out, the switch goes to B, and vice versa.  But if it was connected backwards?  A keels over, and the switch, automatically, selects what it thinks is B -- but is in fact A, and A, being dead, does nothing.  And the switch, while it is pretty smart as such things go but not quite as smart as one might hope, assumes it is not being lied to, and B must be as defunct as A.  So it does nothing.

     We have backups on our backups, and that kicked in -- losing a few minor channels in the process.

    I moved the connections to where they should have been all along, the two ends did a complicated handshake, and hey, presto! Everything was working, with the failed part safely sidelined for later.  And it only took three hours.  Well, four.