Tuesday, March 07, 2017

Lurching From Crisis To Crisis

     A little less than two hours from the end of my shift -- and ten minutes before it was needed -- we had a critical widget fail.  No, that's not exactly true: the darned thing was found twenty minutes before it was needed and the users spent ten minutes spinning their wheels and making oblique comments on the intercom instead of yelling for help.

     It was ten minutes we really could have used.

     There were several possible causes; the device moves around and uses fiber-optic connections to the controller, via a kind of connector that it easy to unplug but shouldn't be -- and yet occasionally is, by helpful users trying to untangle the traveling cable from four more just like it.  So we cleaned 'em, a procedure that takes two and a half minutes per connector, five minutes for each end of the cable or ten minutes overall.  That didn't work, and by then it was too late.

     We kept checking, staying out of the way of the users as best we could, and after two hours, it was pretty obvious that the fiber "modem" on the device itself had failed.  When I departed, the next shift was just finishing swapping the entire device for a near-substitute, and won't this be expensive fun to get get fixed!

6 comments:

Old NFO said...

And I'm 'sure' maintenance will get yelled at for 'not' fixing it in a timely fashion... Sigh...

wheelgun said...

Only 10 minutes wasted? I guess that comes from having a deadline. I used to get handed problems at 3:15 that *somebody* had been working on since 10 AM. It was a few hour fix, but they didn't know that. (They only included me because they wanted to go home one time.)

Old NFO, "Sigh" indeed.

6 phases of project management:

1 Enthusiasm,
2 Disillusionment,
3 Panic and hysteria,
4 Hunt for the guilty,
5 Punishment of the innocent, and
6 Reward for the uninvolved.

wheelgun said...

Redundancy is your friend. Anything critical... but that costs money.

Will said...

Perhaps it's time for a locking latch, or box, for those fibre connectors to live behind/inside. Do they at least have a warning tag?

It sounds like an optical modulator. Wouldn't be a Harmonic's head box, by chance? I built the Nd:YAG laser and modulator systems for those, back in the 90's. Quit when they switched to using diode lasers.

Roberta X said...

Will, it's a very specific thing which I can't get too specific about here, but the connectors in question are very large, industry-standard Lemo types over an inch in diameter, carrying two fibers, two power conductors and two "communications" wires. One end (where we suspect the trouble lies) connects to a device that moves.

Will said...

Whoops, forgot about this post.
Worked with LEMO connectors on fibres for Nd:Yag surgical lasers in the late 80's. Sounds like the same size. Lousy connectors for that application. Not precise enough for accurate alignment of fibres to handle the power that the systems were capable of producing. That was with 600micron core fibres, and they wanted to use smaller ones.
There seemed to be alignment variables that the in-house LEMO fibre assembly people couldn't control, judging by the high failure rate during acceptance testing. (Failure of an SMA mounted fibre was a rare event.)

The surgical laser industry was moving toward SMA connectors for fibres, so I designed a setup to convert customer's machines to use them, that allowed the machine to put full power into a 400micron fibre. The company sat on it for several years while they struggled with a much more expensive fix that down-graded the capability of the system. Cost them millions of dollars for that office politics screwup. Eventually they hit panic mode, and decided to dust off my version. When I saw the package ready for shipment, I discovered someone had decided to use a down-rev version of it, that made it more awkward for the maintenance tech to clean and adjust. Idiots. Company is long gone.