Saturday, May 06, 2023

The Blame Game

      Some days, I swear I'm gonna go Sledgehammer Barbie on a recalcitrant piece of electronic equipment.  User-hostile support only makes it worse.

      We've got a gadget that takes a bunch of meter readings and status indications (off or on?  Okay or acting up? and so on) and hands them off via a network connection to a number of distant locations, while outputting on-off commands from the same locations.  A lot of companies make them.  Over 40-odd years, I have installed four different models from four different companies and worked on at least six varieties.  It's not unusual to total a couple of hundred "channels" of telemetry, status and control.

       The details of how they interface with external hardware vary and I have learned to always build a connection panel; barrier-type screw-terminal blocks used to be standard but the last couple have used "Eurostyle" connector blocks and crimp-on ferrules, a combination that works well with solid and stranded wire in a wide range of sizes.  The reason for this is that with one exception, every last one of them has used some quirky, high-density connection system for inputs and outputs.  They are awkward to change connections on, especially without disturbing other connections, and most accommodate only a few sizes of wire.  So you set things up so you only have to make the difficult connections once and all subsequent changes are made using something simpler and easier to get at.

      The most recent such device has jumpers (with SCSI-bus obscure high-density connectors at each end!) from their devices to panels with custom 110 blocks, a punch-down, insulation-displacement connection developed for telephony.  They are compact, secure and extremely reliable -- if you use the one exact right kind of wire (#24-26 solid) with the right kind of insulation and the proper spring-impact tool to shove it into place and trim off the excess.  So I did; I bought 25-pair telephone wire, put the 110 bit in my punchdown tool and wired the 110 blocks out to Eurostyle blocks, where I was able to connect the mad array of different wires that go to all the widgets this thing has to talk to.  I followed the proper color code for 25-pair cable, making the wiring relatively easy to follow -- despite the OEM having installed their color-coded 4-pair 110 blocks backwards, so each group of four is marked in ascending numerical order but color-coded for pairs 4, 3, 2 and 1.  (That's brown, green, orange, blue, for those keeping track at home.)

      This same device turned out to have a software "gotcha:" if you followed the instruction manual in naming, setting parameters, calibrating and assigning each channel of the interface device, that channel would become "invisible" to the other end.  There's a hidden command level to make it visible again (by entering all the information again) and end-users cannot access it; only factory support can make those changes.  The manual doesn't tell you that, so if you're me, you set the system up and check it before handing it off to factory support -- or you try to check it.  I spent a month messing around with it, trying to figure out what was wrong, before I admitted defeat.  The manufacturer was cagey; all they'll say is, "The manual is out of date," but they won't tell you anything more.  In four and half years, no updates to the manual have been forthcoming.  It looks like they just don't want us in there.

      As a result, when we make changes or have trouble, factory support has to be involved.  So when the network connection with one of the devices became unreliable and we'd eliminated other causes, they sent out a new device, (supposedly) ported all the settings from the bad one to its replacement, and once that was done, I moved the connectorized jumpers over.

      Half of the indications were wrong.

      The first question from factory support, working remotely: "Will you check the punches?"  Yes, sure.  I reminded him that I had not changed that part of the system in any way, got out a meter, checked and reported back.  All was as it should be.  I already knew that.  When I installed this system, I did so in parallel with the one it was replacing, and left it running.  The old system has a nice interactive computer display of all the relevant parameters, essentially a web page of geekery.  At least I think it's nice -- probably because I laid it out and did all the fiddly work to set the appearance, calibration and colors.  It was still okay.

      I suggested that perhaps not all of the settings for the individual channels had been ported over from the flaky device to the new one.  I cannot see those settings; after the debacle of the original setup, I don't have a computer at the site able to access the system at that level, since anyone making the slightest change, even by accident, will make it "disappear" from the other end.  Factory support wanted me to re-check the punches.  I sent him a photograph of them, and followed up another set of measurements.  If you use skinny probes with sharp-pointed ends, you can check directly at the contact terminals of the 110 block, affirming that the electrical connection between the wire and the block is okay.

      The factory tech was assuming we had punched whatever kind of wire we had directly into the block.  This is a known source of intermittent trouble -- and it's why I used genuine telephone-type wire to run from the 110 block out to a more tolerant kind of connection.  It's a lot simpler for their tech to blame me -- "The punches/grounding/jumper settings must be messed up." -- than dig back into the individual channel settings of the device, with multiple parameters to be looked at and possibly reset for each one of 96 channels.

      In the end, I moved the plug-in jumpers back to the original device and -- what a wonder! -- all of the readings went back to normal, just as if the punches, grounding and jumper settings were okay.  We'll have another go at it next week.

      I am hoping the next attempt will not involve a round of mutual finger-pointing.  I'm more than happy to show my work.  So far, the factory has not been.

7 comments:

Robert said...

Your story makes this old field service guy cringe. I am glad I am not you (nuthin' personal). Good luck.

Roberta X said...

It's about par for the course. The support guy keeps wanting to do a phone call, and I keep on using e-mail that Ccs his boss and mine. If he does call, I'll do summary e-mails after each conversation. I'm detail-oriented and I get paid by the hour. They're not hanging me out to dry on the hook of "inept local tech has screwed up and we're washing our hands of it." I've sent them close-up photographs of the 110 punchdowns; I have nothing to hide.

They want this to be our problem, and not in their hardware. And it might, in fact, be our problem -- but at the data-network level, not the hardware I/O level. There is heavy security, managed network switches, restricted paths and an invasive system of network scans, all overseen via a loosely-coordinated group of in-house local and Corporate level IT gurus plus external providers that could be creating glitches. Successfully swapping out the hardware will help locate such problems if they exist. The IT guys are *buried* in troubleshooting everything from rained-on laptops to bug-laden updates to critical systems and the more possibilities we can eliminate, the simpler their part of this will be.

grich said...

Yikes! The last two I've installed were brand "B", and I've never had that kind of trouble with making changes (is yours brand "D"?) I get humbled often enough, but I've been in the biz 40 years, and have never appreciated factory techs that are quick to blame the customer when poor equipment design is the culprit.

The latest brand "B" stuff I've installed has eurostyle-block connectors on the interface chassis, so connectivity usually isn't an issue. I've had to build opto circuits to get status indications from the transmitter to talk nice to the remote control...that's been the weirdest issue for me.


grich said...

Just read your reply to Robert, and the mention of "invasive" network scans hit a nerve. Our brand "B" interface chassis WILL lock up hard when hit with the scans, requiring a power-cycle to restore. They are now on an exclusion list.

Roberta X said...

Our old system was brand "B." I liked it a lot, especially the GUI -- as I liked the late-90s Moseley it replaced, in both the slick MS-DOS-native flexible text UI it came with and the slightly less flexible but workable customizable Windows GUI that replaced it.

The present system seems to be favored by big corporate owners and the precise name of it is vague -- the support consultants have one acronym, the hardware (apparently built by or for them) has another and the software that runs the whole thing has an entirely different name. Since I am of the opinion the hardware is average at best and the undocumented software bugs intentionally exist to A) keep local fingers out and B) enable ongoing support contracts, I'm not going to mention any of the names it's known by. The UI is remarkably ugly and inflexible as such things go, and if you've got this system, you'll know it. The most favorable comment I have heard about it is, "It's adequate."

Roberta X said...

Grich: I ran the connectorized Phoenix plugs on our Brand B out to terminal blocks because I had so darned many channels to hook up, and you can't get a screwdriver on the connector without pulling it and taking down multiple inputs or outputs. I wanted it all out where I could get at it. I was having to re-use all of the original wiring to the Moseley in order to minimize downtime, and it was a lot easier to move cables to Eurostyle blocks on rack panels with lots of space between the rows.

The current system got the same treatment. As soon as I realized they'd used 110 blocks, that was a foregone conclusion: most of my connections to the big air-cooled equipment were made with stranded wire, due to the vibration-fatigue problems with solid wire. That stuff doesn't punch worth a darn and about half of it was going to be too big.

Those wire-end ferrules are worthwhile. The crimping tool is one-size-fits-all, and mashes them down square with multiple high-pressure points. They go right into a Eurostyle block, or a screw-clamp Phoenix connector with no fooling around securing loose strands or misclamping of solid wire.

Robert said...

Every time a customer called my boss saying I was incompetent, it turned out they had made changes that were invisible to me and they felt no need to inform me.

Roberta: your cyber-paper trail is an outstanding CYA procedure. CCs to the boss types is even better. I absolutely detest having someone try to make me the fall gal or guy. I look forward to hearing the resolution of your adventure.