Wednesday, January 23, 2008

Sometimes You Shoot Trouble. Other Times, The Trouble Trees You, Barking

And let's just say the cuffs of my jeans are well-frayed right about now. The "trouble" here is in the main RF cabinet of one third of what we shall call the stardrive array. Each RF cabinet is about two and a half refrigerators in size and considerably less easy to get inside.

So, there we were, Handsome Dave an' me, stepping our way through Using The Stardrive's 36kV Supply To Locate Its Own Problems. It should be cut'n'dried but somehow it never is. There was, of course, one step not in the manual, and the part it would test was something that just should not be a problem, a thing very overbuilt for the task. And every step takes forty minutes or more. You have to talk about what you're going to do before you get into the very noisy compartment (Hearing Protection Required For Your Safety) where the stardrive is mounted and before that, you have to figure out what it is you'll be doing; there are mandatory cool-down and warm-up times and so on.

Step by slow step, we approached this one item; and when we got there, lo! Oh, the shock and wonderment! Scarcely a mark on it, yet-- It had failed!

Briefly, there was much rejoicing. Then... Well, you don't stock parts unlikely to fail, do you? H'mmm. Oh, but wait; we'd upgraded every part of the original stardrive except the phantasmajectrode[1] and its so-called "trolley" back in '01 and I had, over the objections of Wiser Heads, removed every part small enough to lift and stashed them away in the structure. Could it be?

Twenty minutes later, it seemed like maybe not, and then I lifted away a very nearly complete Hydrogen Thyratron Crowbar assembly[2] out of a nest of plastic packing, the sharpest-lookin' (but one of the least vain) tech-d00d in the biz reached into the packing to see if any interesting things had fallen off, and put his hand directly on the exact component we needed. The factory techs had called it the "dynamite stick" and it does look like a fat stick of TNT. (It's a custom high-voltage part).

We went back to the stardrive compartment, I wriggled my way into the crowded, awkward innards of the device (poking away at all the high-voltage-while-operating spots with a shorting hook, 'cos all live stuff is always live until tested, in the exact same way all guns are loaded all the time) , unhooked and dismounted the old part (nice little side-trip into "oh, that's a metric fastener, then?"), got the new one all in, clicked the large (and ever so fragile, easy, easy...) isolation resistor back into its brackets, eeled back out, disconnected the shorting hook, we reinstalled the cover plates with most of the proper bolts, kicked all the breakers back on, flipped the big earthing[3] switch, engaged the smaller "isolation" switch that pins it in place and enables everything but the 36kV, went around to the front and started the ten-minute warm-up cycle.

...A dozen minutes later, the "READY" light came on, my erstwhile peer hit the "BEAM" button and our hours of step-by-tedious step were rewarded when the high voltage came up, step one, and a couple ticks later later, the second step kicked in and it was back at full beam voltage... Hoo--

--ray? Click-THUD! The "BIAS OK" led on the status panel goes out and the 'drive shuts all the way back down to "PREHEAT" Won't even go back to "STANDBY" until we kick it to "OFF," at which point we can start over from scratch and it proceeds exactly as before, right up to the point where it shuts itself off. Third try for luck and...same thing. Went back to the workshop, called the factory[4], hit the books while waiting to hear back, and found nothing at all to explain this behavior. When the factory called us back, they agreed. "Never heard of that," said the tech, "and I've seen about everything these gadgets can do wrong." Ummm, thanks?

Tried a number of things and ran out of time. No joy. But hey, the high voltage does come up; it wasn't doing that at all before! Progress must have been made.

The next day, a bold plan: We'd lie to the status circuit; with the phantasmajectrode out of the circuit (and it had been from the moment high voltage troubleshooting began), there was no need of the bias-checking thingamabob. So we tried it. No dice. BEAM ON. Handsome Dave says "Flame on!" (I flinch and plead, "Pleeeze don't put it that way." He grins ever so slightly). Count about three and..."BIAS OK" goes out ("But it can't go out!" "It just did. Fourth time.") Rats.

Found and cleaned some suspicious-looking spots on the circuit board. Maybe...?
This takes us to lunchtime. Came back from lunch found, about three hours in, one (1) tiny difference between this particular model and the other two stardrives in the main drive array. Rechecked. Looks wrong.

Made the change, restarted. Success! Now it's back to step-by-step and each step is 40 minutes but each one works!

Ah? Ah-aaaah! I've been holdin' out on you. Since even before all this happened, there has been another problem, this one in the serious bigtime primary power regulator that provides the juice for the stardrives, and the factory that made it seemed to not be answering their ansible. Until we get it either A) happy or B) bypassed, we can't run all three 'drives at the same time. We can't bypass it hot; we have to power down the entire drive array and if we are extremely lucky, it will take a minimum of 30 minuted to accomplish it and get the drive back on.

So of course we call the bridge and of course the Captain says, "Go black? At this time? D'you realize how many complaints we're going to get? The disruption?"

...It's about the end of another day anyway, so we schedule a better time and knock off. All that work and we're still not at full power! I think we're not gonna be at tau Ceti in time for lunch, kids.

So, how was your day?
1. Hey, does it not sound a zillion times niftier than "Inductive Output Tube?" It does.

2. You're not gonna believe me when I admit that's the real name of the device and subassembly, are you? But it is. And it handles atomic-physics-lab energy levels for a very short time when it's triggered, too.

3. As in, "Oh, that's a metric fastener?" "Earthing" = "grounding." Our stardrives were designed in the UK and built in the US using parts from all over. Most of the fasteners are SAE but the tuned cavities and support assembly for the phantasmajectrode and a few other special items still bolt up British.

4. If it wasn't for the
ansible, working on these things would be way worse!


Tam said...

Why do I keep having visions of Vinnie the Volt lurking in ambush at your every turn?

RE: Customer support.

Back when I was with SKB's flight department, our mechanics ran into a peculiar maintenance issue with one of our high-time Cessna 310's. They called the guys at Twin Cessna to ask for some troubleshooting tips. Which led to the following:

"You've got how many hours on the airframe? Uh, that's kind of outside of our experience. There's an outfit down in Georgia that has more time on 310's than anyone on the planet. I can give you their number..."

"Is it 555-5454?"

"Uhm, yeah. How'd you know?"

"That's us."

Anonymous said...

Be careful.
Remember Murphy's Third Corollary: "A part for which you have a spare will never break."
British, eh? I bet there's some Whitworth hardware in there somewhere. And maybe a left-handed, coarse-threaded, Imperial castellated nut or two!

Anonymous said...

Congrats! You're at the top of the Google list for "shorting hook"!

Rob K said...

I would rather a machine be all metric fasteners than a mix. I hate that when I run into those.

Carteach said...

Troubleshooting on vehicles..
An old saying we use sometimes.

Some days we troubleshoot and win.
Sone days the trouble shoots back.

Dr. StrangeGun said...

Reminds me of my '84 Tbird with the Essex V6... imperial for everything but all the motor bolts (metric) and the interior (torx).

Roberta X said...

Things are often a mix -- my '74 MGB is nearly all SAE where it matters, metric in a few places and has Strange Coincidence fasteners for various covers: it's a British Association machine screw, 2BA, that is so very almost a 'Murrican 10-32 that you can use them: slightly (.001) smaller o.d. and 31.4 threads per inch instead of 32. Spooky!

The scooter had me investing in metric ignition wrenchs, spanners and sockets. Well worth it.

Anonymous said...

Sounds much like a former job I had. Replace a part, power one part up, wait for it to say GO, power up the next part, get a RDY, then restart the test you were doing in the first place. That gets old in a mighty hurry when it is -20 (celcius) and dark.