Stop Shouting!

     Alternative titles: "Get Off The Bus!" and "Partyline."

     It has nothing to do with politics.  I went to work on a transmitter yesterday; it's been behaving oddly, slow to "wake up" after power hits, and the first, simplest step was to replace the transient suppressors on the incoming power.  They're conveniently socketed, and somewhat inconveniently mounted -- you have to remove the front panel from a module and it exposes the incoming mains voltage, 208 Volt 3-phase.  So I had to shut it all the way off.

     I proceeded to do the task, transmitter off, breaker off, panel off, unplug and replace three little widgets, then put it all back together, look it over, flip the breaker, power up, wait for the hearts and brains of the thing to get happy (okay, the RF modulators and controllers, each nicely redundant) and then the ON switch, a little power-up cycle of yellow and green lights, and--  And--  And--

     And the darn green lights went nuts.  Multiple faults, amber and blinking red in the "Christmas tree" status lights down one side of each amplifier; on the other, where three power supplies should light up green and steady, instead it showed blinks, green and off, chasing one another down the front panel in waltz time, 1,23; 1,2,3; 1,2,3; 1,2,3.  One amplifier showed a few more faults than the other three.

     I started digging through the front-panel menus on the controllers.  They had apparently signed up with the Wobblies: one should be Active while its pal is on Standby.  Instead, they were both on Standby!  Checking the computer-screen GUI confirmed it.  They weren't a-gonna work.

     The manual recommends setting the power output to zero and rebooting.  So I did.  Nope.  The next step is to try shutting things off -- it will run with one controller and one exciter, after all -- and rebooting.  Nope.  At that point, I sat down and put in a request for tech support with the manufacturer,* then went back to trying things like shutting off each amplifier module one at a time, and trying to take off with one prop feathered.  No luck.

     Eventually, a note came back from the factory.  They were reaching out to their subject-matter expert -- who just happened to be the contract engineer who had supervised our installation.  I remembered him as a guy who'd known all the quirks of the transmitter.

     Back to trying things.  My phone lit up with a message: Their guy was deep in another crises, and suggested putting the old transient suppressors back in. That's right, he'd said those things could be flaky, back when we put the transmitter in.  So I did that.  No change.

     I tried a few other things, got a message saying a call to me was next in the queue, and spent some time going back through the manual.  There's no tutorial, but the English translation has all the drawings and implies quite a lot.  I rebooted the internal network switch, even though it mostly communicated with the outside world; internal communications uses a different system.

     Man, still no call.  It had been an hour and a half since the initial contact.  I sent another email to the factory support coordinator, suggesting I'd been waiting long enough to hire a truck and make a good start towards driving the thing back to their parking lot.  That got a reply.  "What?  He hasn't called you?"  By this point, I'd also told my bosses, and they were sending increasingly worried text messages.

     The expert eventually did call.  He was mystified by the symptoms.  We wasted a half-hour trying to connect to the amplifiers directly.  They've got USB jacks, but it takes a stone-simple terminal emulator; it pops up a blinking cursor and we got that far, at least.  From there, a menu should appear in response to a tap on the ENTER key, but none of 'em were answering the phone.

     This made no sense to me, and, at first, not much more to him.  Then he asked if I'd tried shutting things off and restarting.  I had.  "Have you tried shutting off two things?"  That, I had not. So, with the most-complaining amplifier and one of the controllers off, I did yet another reboot.

     It's a slow process, a power-up self-test that steps through each part.  Green lights came up, flickered amber, came back green one at a time.  Green lights that stayed green.  I tried turning up the output power and my analog meters flickered.  (They're external.  It bugged me that the thing didn't have a nice big "I'm happy!" indication I could see across the room, so I added them.)  I told the expert, and kept on turning up the power, a little at a time.  "You're not going to make full power, of course," he told me, "But it'll do at least fifty percent."

     With one of four amplifiers off, you might think 75 percent, but that's not how the math works; the combining uses clever physics tricks that only add up nicely when everything is balanced, and when it isn't, some energy is lost.  Fifty is still greater than zero, and that's where we ended up.

     So what happened?  The internal communications is a kind of "party line," a shared serial data bus.  The expert figures the bad amp has been yelling for help -- but not flagging any alarms -- for a long time.  That's why the transmitter's been so slow to recover from power hits.  When I shut the transmitter down for half an hour, that's longer than it has been off since we put it in, and one of the controllers woke up unhappy.   I'm told that's a known problem.  And the controller was spamming the party line, too.  That's way too much traffic: neither controller could promote itself to active, the good amplifier modules couldn't report themselves ready for duty, and that transmitter wasn't going nowhere nohow until we shut off all the racket on the line.

     If it's not one thing, it's two things.  And it was.  The amplifier modules -- eighty pounds or so! --  are factory repair only, so they're shipping me a box and we'll cycle it through the repair process.    
________________________
* Or as close as I can get.  It's actually imported, then fitted out to U. S. standards by the importer.  But both the overseas manufacturer and the U.S. importer are owned by the same multinational, headquartered in yet another country, so they're actually doing business with themselves.  Happily, all three are mostly geek-driven.  Unhappily, all three got downsized as the last of the U.S. digital cutover process met the COVID-19 pandemic.  There are a handful of engineers and techs left, in three languages.