The Blame Game

      Some days, I swear I'm gonna go Sledgehammer Barbie on a recalcitrant piece of electronic equipment.  User-hostile support only makes it worse.

      We've got a gadget that takes a bunch of meter readings and status indications (off or on?  Okay or acting up? and so on) and hands them off via a network connection to a number of distant locations, while outputting on-off commands from the same locations.  A lot of companies make them.  Over 40-odd years, I have installed four different models from four different companies and worked on at least six varieties.  It's not unusual to total a couple of hundred "channels" of telemetry, status and control.

       The details of how they interface with external hardware vary and I have learned to always build a connection panel; barrier-type screw-terminal blocks used to be standard but the last couple have used "Eurostyle" connector blocks and crimp-on ferrules, a combination that works well with solid and stranded wire in a wide range of sizes.  The reason for this is that with one exception, every last one of them has used some quirky, high-density connection system for inputs and outputs.  They are awkward to change connections on, especially without disturbing other connections, and most accommodate only a few sizes of wire.  So you set things up so you only have to make the difficult connections once and all subsequent changes are made using something simpler and easier to get at.

      The most recent such device has jumpers (with SCSI-bus obscure high-density connectors at each end!) from their devices to panels with custom 110 blocks, a punch-down, insulation-displacement connection developed for telephony.  They are compact, secure and extremely reliable -- if you use the one exact right kind of wire (#24-26 solid) with the right kind of insulation and the proper spring-impact tool to shove it into place and trim off the excess.  So I did; I bought 25-pair telephone wire, put the 110 bit in my punchdown tool and wired the 110 blocks out to Eurostyle blocks, where I was able to connect the mad array of different wires that go to all the widgets this thing has to talk to.  I followed the proper color code for 25-pair cable, making the wiring relatively easy to follow -- despite the OEM having installed their color-coded 4-pair 110 blocks backwards, so each group of four is marked in ascending numerical order but color-coded for pairs 4, 3, 2 and 1.  (That's brown, green, orange, blue, for those keeping track at home.)

      This same device turned out to have a software "gotcha:" if you followed the instruction manual in naming, setting parameters, calibrating and assigning each channel of the interface device, that channel would become "invisible" to the other end.  There's a hidden command level to make it visible again (by entering all the information again) and end-users cannot access it; only factory support can make those changes.  The manual doesn't tell you that, so if you're me, you set the system up and check it before handing it off to factory support -- or you try to check it.  I spent a month messing around with it, trying to figure out what was wrong, before I admitted defeat.  The manufacturer was cagey; all they'll say is, "The manual is out of date," but they won't tell you anything more.  In four and half years, no updates to the manual have been forthcoming.  It looks like they just don't want us in there.

      As a result, when we make changes or have trouble, factory support has to be involved.  So when the network connection with one of the devices became unreliable and we'd eliminated other causes, they sent out a new device, (supposedly) ported all the settings from the bad one to its replacement, and once that was done, I moved the connectorized jumpers over.

      Half of the indications were wrong.

      The first question from factory support, working remotely: "Will you check the punches?"  Yes, sure.  I reminded him that I had not changed that part of the system in any way, got out a meter, checked and reported back.  All was as it should be.  I already knew that.  When I installed this system, I did so in parallel with the one it was replacing, and left it running.  The old system has a nice interactive computer display of all the relevant parameters, essentially a web page of geekery.  At least I think it's nice -- probably because I laid it out and did all the fiddly work to set the appearance, calibration and colors.  It was still okay.

      I suggested that perhaps not all of the settings for the individual channels had been ported over from the flaky device to the new one.  I cannot see those settings; after the debacle of the original setup, I don't have a computer at the site able to access the system at that level, since anyone making the slightest change, even by accident, will make it "disappear" from the other end.  Factory support wanted me to re-check the punches.  I sent him a photograph of them, and followed up another set of measurements.  If you use skinny probes with sharp-pointed ends, you can check directly at the contact terminals of the 110 block, affirming that the electrical connection between the wire and the block is okay.

      The factory tech was assuming we had punched whatever kind of wire we had directly into the block.  This is a known source of intermittent trouble -- and it's why I used genuine telephone-type wire to run from the 110 block out to a more tolerant kind of connection.  It's a lot simpler for their tech to blame me -- "The punches/grounding/jumper settings must be messed up." -- than dig back into the individual channel settings of the device, with multiple parameters to be looked at and possibly reset for each one of 96 channels.

      In the end, I moved the plug-in jumpers back to the original device and -- what a wonder! -- all of the readings went back to normal, just as if the punches, grounding and jumper settings were okay.  We'll have another go at it next week.

      I am hoping the next attempt will not involve a round of mutual finger-pointing.  I'm more than happy to show my work.  So far, the factory has not been.