A little less than two hours from the end of my shift -- and ten minutes before it was needed -- we had a critical widget fail. No, that's not exactly true: the darned thing was found twenty minutes before it was needed and the users spent ten minutes spinning their wheels and making oblique comments on the intercom instead of yelling for help.
It was ten minutes we really could have used.
There were several possible causes; the device moves around and uses fiber-optic connections to the controller, via a kind of connector that it easy to unplug but shouldn't be -- and yet occasionally is, by helpful users trying to untangle the traveling cable from four more just like it. So we cleaned 'em, a procedure that takes two and a half minutes per connector, five minutes for each end of the cable or ten minutes overall. That didn't work, and by then it was too late.
We kept checking, staying out of the way of the users as best we could, and after two hours, it was pretty obvious that the fiber "modem" on the device itself had failed. When I departed, the next shift was just finishing swapping the entire device for a near-substitute, and won't this be expensive fun to get get fixed!
4 months ago