Thursday, November 08, 2012

"Look Upon My Works, Ye Mighty..." And Falter

     I'm so proud: tonight I was a part -- I daresay a key part -- of a team that threw away I don't know how much money.  $20K?  30?  More?  I  may have even helped my employer lose more than they'll pay me this year.

     Why?  Well, we all trusted something to be right; even gave it a cursory check.  Missed a very key parameter, though (heck, who would have preset that wrong?  It can't even be changed without digging deep, deep into firmware and doing a very scary reboot!), and when it came time for it to be there...it wasn't, not quite, and we faffed around for ten minutes of very high-priced time trying to sort it all out.

     So proud.  Contributed so materially to not catching the problem before it bit us where we sat. Slunk out of there at the end of my shift and I'm giving serious thought to the merits of debarking to Patagonia first thing tomorrow.

     Geez.

20 comments:

Hammerbach said...

You would not be the first- and your attitude does you credit. I wish you worked for me.

Dave H said...

You described a typical week at my job.

"To err is human; to really screw things up you need Design Engineering involvement."

jed said...

I feel your pain. I've unfortunately been involved in projects where much much higher sums were flushed down the toilet. Both, BTW, involving govt. agencies. But I digress. It's always disheartening, even depressing. Yep, we're humans. Hell, where I work now, we're filling up a rolloff with money -- mis-ordered doors, parts, countertops ... 2nd time in 4 years.

Trust is a bitch. But where would be without it? I rely on it all the time, and there's only so much verify you can do.

Over on the Arduino forum, there've been some threads along the lines of "what's the most expensive board you've fried?" Some pretty good stories there from the old hands of the bunch.

Anyways, Patagonia. Same objection as pretty much anywhere other than USA. Can't own guns (well, not without greasing some palms, at least). Sure is pretty, and they have penguins too.

Carteach said...

Been more than a few days where I was looking through my pockets for that matchbook advertising 'truck driving school'.

Anonymous said...

So Ozi, what exactly would the difference have been if you'd caught "the problem before it bit us where we sat"? Would it, perchance, have still needed that expensive therapy but just without, maybe, all the rush and self-flagellation?

Just because we might think you're perfect, doesn't mean you are! ;-p

Oh, and as an employer I'd rather have someone who recognises they missed something (as most would have done in similar circumstances, don't deny it) and won't do it again than the usual 'twern't my fault, twas the manufacturerer'.

Just sayin'

Roberta X said...

Able: had we caught it beforehand, there would have been no excitement at all. Nor any loss of revenue. :(

Cincinnatus said...

Ah, that's nothing, I lost about $2 million for a client in about 10 minutes once ...

B said...

$60 Grand in 4 seconds once, back in '92. Told my boss expecting to be fired.

He didn't mind the money so much as the impact on delivery time.

Suggested I get to work making a list of parts we would need to replace the fried ones. And plan on working overtime to rebuild.

Many have similar stores. If you work long enough, you'll have another one. Humans err.

Opinionated Grump (Rich in NC) said...

Isn't that the reason that God invented asbestos shorts?
Rich in NC

New Jovian Thunderbolt said...

Coulda been worse. You coulda inadvertently killed someone.

Roberta X said...

Pretty much my worst nightmare, driving downtown in the dark.

og said...

This is not the end, and you have learned from it, so neither is it a complete loss.

yeah, you feel likc crap for a long time after but eventually you heal.

Anonymous said...

"An expert is a man who has made all the mistakes which can be made, in a narrow field."
Niels Bohr

"Eight years involved with the nuclear industry have taught me that when nothing can possible go wrong and every avenue has been covered, then is the time to buy a house on the next continent."
Terry Pratchett

After twenty-mumble years I can honestly say the only reason I haven't killed someone by omission is that we're all paranoid and act as if everyone we work with is a lazy, incompetent idiot, double and triple checking everything. And yet still things get missed. It happens. Sucks bigtime, but the same never happens twice. I'd say I'm well on the way to being a Bohr defined expert by now.

I'd guess boss is pissed but realistic. The only one beating you up is.. you.

“A clever man commits no minor blunders.”
Johann Wolfgang von Goethe

I guess this means you're REALLY clever ;-)

mikee said...

Having spent months building a multi-technique ultrahigh vacuum analytical system on a large cylindrical vacuum chamber, when all the electron, ion, x-ray and other spectroscopy finally was working, I was chagrined to learn the chamber was not exactly cylindrical, the attachment ports were misaligned and none of the spectroscopies pointed at the same spot in the darn chamber.

Many adjustments to the x,y,z coordinates of the sample holder got us through initial research work, but months would have been saved had we spent $1000 or so laser alignment checking everything first rather than trusting the chamber manufacturer and our eyeballs.

Learned that lesson some 25 years ago, and have annoyed any number of folks with retelling it pointedly in meetings and demanding more confirmation of "known" stuff since then.

rickn8or said...

Stevie Ray Vaughan missed a note here and there, but people still think he was a middlin' good git-tar player.

Anonymous said...

I once got a call from super secret .gov facility that some equipment we sold them was not working. Big confernce call with several PHds and engineers to trouble shoot problem with no luck.

I loaded the replacement into a car and drove 12 hours to super secret facility and went through everthing short of anal probe to get onsite.

I found out they never turned on the power on installation. We assumed they would have checked that first.

Gerry

Anonymous said...

What a coincidence. We just had a customer complaint about a batch we made several weeks ago. Guess whose name is on the COA, proudly proclaiming that it was just fine? And guess who spent the past two days trying to figure out, "How the hell did I miss THAT????"

Retired Spook said...

Don't let it get you down! We all have those days when the company would have been money ahead to pay us to stay home! Take comfort in the fact that while you may make other mistakes in the future, you won't make THAT one again!

Retired Spook said...

Don't let it get you down! We all have those days when the company would have been money ahead to pay us to stay home! Take comfort in the fact that while you may make other mistakes in the future, you won't make THAT one again!

Ritchie said...

Due to the unfortunate nature of reality and complex systems (more than 1 part), it is necessary to make assumptions at some point. Gravity and time are probably good to go, after that...