Worst Professional Blunders...

SIZE=4]Wondering how many people will be willing to post some of their worst professional technology blunders...[/SIZE]
#1) While designing a targeting system for the main gun of a Frigate....

We were on sea trials, a videeo camera was loadede into the main gun, and drone aircraft were sent flying around. If the drone stayed in the center of the cross hairs, all was good.

Afrter about 2 hours, the ship suddenly reverberated with a heave metal on metal BANG, that rattles the entire ship. The was mild panic, since this was the first time the ship was out at sea, and anything upto and including a hull fracture could have been the cause. Nothing was found.

About an hour later, the same thing, less panic, but more concern. Again nothing was found.

Another 1.5 hours passed, and it happened again. This time a sailor saw the main gun move from the targed possition, slam down against the bottom stop, slam up agains the top stop, then return to tracking the target.

The cause was found...but why???

Looking at some of the machine logs the reason was determined. A gun must correct for "actual" wind. But the ships sensors (anonemeters) measure relative wind. You need to subtract out the ships speed and heading (the induced wind). Now when dealing with angular measurements, you typically use either 0-360 or +/-180. I had chosen the latter.

There was a filter algorythm that smoothed the data. If the wind was coming in over the stern (very rare), then sometimes you would get numbers in the 170-179.99 range other times in the -179.99 to -170 range.

Now what happens if you average these two numbers. YOU GET A NUMBER NEAR ZERO.

The computer interpreted this as an instant hange from a tail wind to a head wind!. The gun went berserk...... :eek: :eek:

[June 1991 abord an RNLN M-Class Frigate]

Well, the technical blunders that I did, were int most cases because of the limitations imposed by client, as in one of my major project, I've to integrate different 3rd party SDKs, and some of them were not complient with each other (for example, integration of GameSpy SDK with DirectPlay, which isn't supported by GameSpy), but I was told to "Just Do It" :confused: , therefore, making layers over layers, so that everything remains intact totally shattered the design and degraded performance. :cry:

One of the biggest blunder that I did, was, I used atoi(), where I should use atof(), infact I shouldn't be using any of them at all, I should go with Automatic Type Conversion,
the expected value truncated :ehh: and luckily, the result returned, it ceased the system, otherwise, it could be a real big news at media. :eek:

Another big blunder that I remember (it wasn't because of me, I just joined the company and the very first day that happened), was because of a design flaw. It was like, clients connect to a proxy and send an order request, which further send it to the main server. I don't know how, but somebody, executed 2 proxies at same LAN, and when sending the orders to the main server, they also send it to the other proxy, which take it as another order and again send it to the main server but also send it to the other proxy again (I guess it was sort of broadcast type of thing). This way, one single order got executed, I guess 13K times :o Imagine, if you are a stock market broker and this happens with you, how would you feel? :sick: The bug couldn't identified for about 2 months, when one day, it got replicated by me while I was giving demonstration to a client, coz I didn't knew the sequence of executions and by chance I executed two proxies at my LAN.

#1 ) Not sure if this can actually be termed as a blunder, i would term this as in-experience, as it was my first project, and about in total 2 months of experience (barring the fact that my educational backgroud is of Computer science ;)).

The application was a real time tracking system, on a hardware, embedding a processor with a processing speed of only about 25 MHz. I proceeded with implementing it as an multi-threaded application, with 4 threads. Now all these threads were transfering data among themself's with a means of shared memory. And :( i totally forgot the concepts of OS. :( *cry* thread synchronization. :eek: and to top it all, it goes undetected through all the testing phase. finally during the customer demo, Every time the application was started on the hardware.... it caused the hardware to freeze completely :( :( :( :GOSH:

I remember my biggest mistake till date was a typo, I used 2 same named variables (with a little difference in spelling) and used them all over assuming I had only one..... Then the thing started showing up bugs in variety and I wasted 2 days looking for problem, went mad as there was nothing wrong.

Luckily, I was debugging with some other debugger and I browsed debug symbols for the module for some reason and saw 2 names for that variable in the list. I was so ashamed to tell that to my boss, I had to make up something else. :D

My first real job, out of school. Working in a huge office, doing WordPerfect on the Novell Lan. One day, a worker's machine crashed, and I told her to check for her backup copy. She couldn't get to the BACKUP folder.

I looked on my machine, and saw that each user had THE SAME FOLDER for all the document backups! I mentioned it to my boss, and he almost had a fit! They called in the big (blue) guns to come in and rectify the situation over the weekend.


They had to determine who owned the files! I made sure that I didn't try to open any of them :)

2 weeks ago, I happen to saw a comment and source code I wrote 2 years ago, because I was reusing the class.

Something along these lines.

Code:

---

// have to allocate on the heap, as we do not know how big the vector
// will be, might overflow the stack if allocate on the stack.
vector<string> *pvecStr = new vector<string>;

---

*LOL* I must be having a brain fart that day. Vector's array is always allocated on the heap! Later I removed the comments and changed the code to below,

Code:

---

vector<string> vecStr;

---

I think bugs and software coding errors should not be classified as "professional blunders".

A professional blunder is (IMO) something on lines of... Taking up a wrong job, or firing the wrong email to the wrong people... Losing an opportunity, etc... Basically, actions that can hinder professional progress.

Bugs - well, software is one of the few professions where bugs are taken for granted and developers even get paid for fixing them. :D

(Think of Win 98 and the number of blue screens it came with - can you think of buying a car that came anywhere that close in quality?)

Quote:

---

Originally Posted by Siddhartha

I think bugs and software coding errors should not be classified as "professional blunders".

A professional blunder is (IMO) something on lines of... Taking up a wrong job, or firing the wrong email to the wrong people... Losing an opportunity, etc... Basically, actions that can hinder professional progress.

Bugs - well, software is one of the few professions where bugs are taken for granted and developers even get paid for fixing them. :D

(Think of Win 98 and the number of blue screens it came with - can you think of buying a car that came anywhere that close in quality?)

---

Well, lets think about this... A "Bug" is a programmer writing code that does not do what is intended...

A surgeon cuts off a persons arm because of knee pain... If that a "bug" or a MAJOR Professional Blunder???

IMHO, one of the biggest problems with the software industry is the lack of responsibility that is placed on developers. If you hire an accountant and that accountant makes a mistake, they are typically libel for any criminal charges/penalties/fines. However if you write an accounting package, and distribute it, it is generally held that the programmer is NOT accountable.

I seriously wonder how much actuality would go up if people were held legally responsible for any and all effects that arose from bugs within their programs.....

[Note: I am niot actually suggesting it, just making an observation on the mindset such as you describe].

Quote:

---

Originally Posted by TheCPUWizard

A surgeon cuts off a persons arm because of knee pain... If that a "bug" or a MAJOR Professional Blunder???

---

It definitely is only a 'bug'. At best a high-priority one that ought to have been detected and resolved either in the reviews or in the pre-delivery testing stage.

Compiled output of a programmer should never be a direct deliverable to a surgeon's computer. If ever a surgeon cut the wrong part of a body thanks to software, then it is a flaw in the process that deployed evidently un-tested software for such a critical operation. The tragedy is definitely not the programmer's fault. It is probable that the bug lies in a generic software module that interacts with the database, and does not deal with body-parts at all!

This is precisely why mission-critical software products often have testing teams that are as big or bigger than the development teams!
(You can include Cardio-Pulmonary Machines, etc in that category.)

Quote:

---

Originally Posted by TheCPUWizard

I seriously wonder how much actuality would go up if people were held legally responsible for any and all effects that arose from bugs within their programs.....

---

Software is therefore usually well-protected by the EULA.

In a worst case scenario, the software-making company is held responsible and has to pay-up, but individual developers are rarely at fault. You can take the anti-trust rulings against Microsoft as an example, or other issues raised by its competitors against software developed by Microsoft.

I my analogy to a surgeon, I was not referring to him using any technology. Just doing the wrong thing. Maybe he thought it was a different patient. Maybe he was just distracted.

The point is he has to take responsibility! Most developers don't. Not even employment responsibility. Imagine simply being fired if you checked 5 bugs into the repository over a 5 month period.

Regarding the programming/testing ration. It sdould be heavily weighted on testing.

I budget my "Hand-On Technical" time (not including administrative task)

1. 10% Analysis Architecture
2. 20% Programming/Coding
3. 40% Testing (and Test Development)
4. 10% Debugging
5. 10% Documentation

That means out of a 40 hour week, 8 hours coding, and the remaining 32 performing other development tasks. I very rarely go over-budget, or miss ship dates...

In a larger team (say 20 people)

1. 2 Architects
2. 4 Product Developers
3. 8 Test/QA Personal
4. 2 Troubleshooters
5. 2 Technical Documentation Writers.

Software is inherently different when compared to other professions and in ways a more privileged profession. :D

When comparing software to surgery...

Software is rarely life-critical and when it is, it probably is tested by more eyes (and other software) than a surgeon's task.

Secondly, a surgeon who does a bad job is usually easily replaced by another that can do the job well. Software ironically is a profession where the person that codes the bug is usually the best candidate also for fixing it - for he knows 'his' module.

Thirdly, all software has bugs. No software developer can promise bug free code.
Good surgeons however can promise and deliver great results, consistently.

Fourthly, some surgeons make mistakes and some do get away without people knowing it. Bugs in software are easier to detect, to reproduce and to measure.

Last but not the least, big software is a lot more complicated than big surgery. There was a parallel I read somewhere that compared developing an Operating System to the complexities of building a Space Shuttle.

Quote:

---

Originally Posted by TheCPUWizard

Imagine simply being fired if you checked 5 bugs into the repository over a 5 month period.

---

Yes, imagine the only guy who is capable of fixing the bug is the one getting fired. :D

Any company (except a one man shop) that allows the situation where only one person is experienced in a peice of code DESERVES to immediately go bankrupt.

At a minimum the person who tests (white box) the code needs to know how every line of the code functions to ensure code coverage and boundary condition checking.

Ideally every peice of code should be written by a team of at least two programmers working in tandem. Some shops take this to the extreme that they share a single computer workstation (I dont go quite that far).

Also, I engourage a company with multiple developers to establish a process where the person who wrote the code (
and checked it into tthe repository) is not allowed to perform any of the maintenance work on that module. It MUST be done by a different person.

This encourages people to write the cleanest possible code. Otherwise their coworkers will rapidly come to hate them.

The teams that have established these simple guidelines do find it works!!! :thumb: :thumb:

(And egotistical developers who think they can not be easily replaced, soon find themselves on the street)). :thumb:

Quote:

---

Originally Posted by TheCPUWizard

Any company (except a one man shop) that allows the situation where only one person is experienced in a peice of code DESERVES to immediately go bankrupt.

---

Actually, multiple people knowing one piece of code is not possible in any big organization producing a colossal bit of software.

What is proposed above may work only in systems that you so correctly called - "shop".

Most (all that I know of) successful companies build themselves on the capabilities and talents of their employees - and derive from factors that makes the talent unique. Any model that proposes that developers be easily replaceable resources and liable to be fired when they write bugs would ensure employee un-happiness, low productivity and a high attrition rate.

I agree with the point you make about more people knowing and being responsible for one software. But, this is not with the perspective that the resource be easily fired when he delivers bugs, rather that sharing knowledge increases interaction between people, raises team competencies and reduces pain when the employee quits.

Redundancy costs a lot of money by the way, and few companies can afford to let two people know every line of code on an organization-wide basis.

I figured this as the worst blunder i've done...

I also do some network instalations, and when istalling the switch-hubs, when making the mini flyleads, I plug both ends into the switch to check them.. (1 or 2 often needs a little extra press with the crimpler) after doing a few dozen of them, linking all the sites, and switching a few leads over, The system ran to speck..

needless to say with initial testig the system ran fine.. Although half an hour later i was called back, because the network had bogged down, and connectivity was imposible..

Looking at the hubs.. Activity was at 100%... after a little time, i found the problem..

I accidently put two flyleads to link two switchs... with smart packet switch's this did not cause immidiate problems but as soon as one of the systems sent a broadcast, the two hubs kept on bouncing this packet back and forth, also seding it out too all nodes... thus effectivly killing the network...

Quote:

---

Originally Posted by Siddhartha

Redundancy costs a lot of money by the way, and few companies can afford to let two people know every line of code on an organization-wide basis.

---

In my experience [30 year primarily in the USA], the cost of "redundancy" is actually very low. Considering that nearly every developer knows multiple pieces of code. There is bound to be a time when he/she needs to work on both at the same time, while another developer may sit idle. Having significant cross-training addresses this issue, as well as issues such as vacations, illness and other reasons why one person may be unavailable.

My "emergency service" contract costs clients approximately 1.5 times my normal rate, since I have to step into an environment where they do not have a knowedge expert available.

Coming back to the topic under discussion - I have seen people get fired or demoted for making false disclosures, going against corporate policy, even for having porn on their company-supplied hardware (knowingly or unknowingly), or make too many personal phone calls on office time - but, I have not seen anyone face dire consequences for writing a bug...

So, I'd say that the probability that one might compromise his job by writing a bug is a lot lower than many of the other mistakes that one might do.

Quote:

---

Originally Posted by Siddhartha

- but, I have not seen anyone face dire consequences for writing a bug...

So, I'd say that the probability that one might compromise his job by writing a bug is a lot lower than many of the other mistakes that one might do.

---

Very true... But there are two type of bugs, and for one of them you can face dire consequences...

1) Unintentional Bugs... : Simply put, these are overlooked, Acidental, Typo (My top bug), Incorrect application of (someone elses) module(s) (mostly due to incorrect spec's) and Oversite...

2) Intentional Bugs... : Back door code (Bypass Security), Event logging (listing critical data such as usernames and passwords), Usage of redundant code to slow the application.

Any of these intentional bugs can land the programmer in a disaplinary hearing..

I've signed a contract that specified that any [b]intentional bugs[/i] (with a long list simular to above) will result in imidiate termination of contract and will lead to legal preceedings to recope losses incured, including Cost of 'Repairs' ..

Gremmy...

:D :D

"Intentional bug" (is that a bug, or a crime?) is akin to one ramming one's car into one's neighbour's home just because one doesn't like the neighbour... :D Chances are that the insurance wont cover damages, if it understood the motive...

Those that program viruses, or trojans that steal personal data while offering the user a seemingly cool utility / service could potentially end up in jail for really long periods... This is a felony.

My Worst Professional Blunder has to be the first business program that I wrote.
The entire project and the business relationships developed from it have been akin to setting myself on fire while eating acid.

I worked for this company as a machinist and they knew about my computer skills. I already did a few programs and some VBA stuff for them. Not only was I a machinist I handled a little paperwork and did tech work for the computer stuff.

They asked to write I program. I didn't know much and let them know.
After I started they cut my time on the project saying that if I didn't complete it with 80 hrs that I have to finish it on my time.
Of course this only amplified the piece of **** program to be an even bigger piece of ****. Though it ended up full of holes and bugs, it worked.

Even after I left the company 3 years ago. I still do a little consulting for them. Every now again I tweak the program a little. But every time there is huge argument about being paid. The new General Manager doesn't believe I should be paid becuase it's my program. LOL. I've gone over it with him several times about why I'm being paid to do this.

I still have the source for this project. They have a copy of source (or they should) from the first working copy.

I've venting a little bit here, they have been calling me for about a week now.

I have only lost one other customer, who I consulted for becuase of this bad deal. I think this time I'm just going to make it very clear to them I will not work for them.

Quote:

---

Originally Posted by Sabin_33

The new General Manager doesn't believe I should be paid becuase it's my program. LOL. I've gone over it with him several times about why I'm being paid to do this.

---

You startyed the program while an employee. Employee's are work for hire, and get paid for their labor (I am referring to in the USA), but do not own their work product.

Therefore you have every right to get compensated once you are a consultant.

Now comes the tricky part. If you modified the program (or created a new one), and were not clear (in a written contract) wether it was a "work-for-hire" or a "work product", then you could be legally libel (if is is deemed a "work product" to make it function properly without additional compensation (unless there was a written contract detailing limits of liability).

Rule #1 for ANYONE developing software outside of a strict employeer/employee relationship is to make sure that there are detailed contracts (my base contract runs about 17 pages) in place.

Quote:

---

Now comes the tricky part. If you modified the program (or created a new one), and were not clear (in a written contract) wether it was a "work-for-hire" or a "work product", then you could be legally libel (if is is deemed a "work product" to make it function properly without additional compensation (unless there was a written contract detailing limits of liability).

---

This indeed is the tricky part. Since I worked for the company as an employee, we never made a written contract. We had a few verbal contracts but they we all met by the time they started to use the program.

I do have copies of invoices issued in the past though. On those invoices is a disclaimer that I am not liable for the work.

I don't think something like this could escalate that much to a court order. I think I'll look for somebody else to take my place consulting for them.

I was once asked to fix this totally non-critical feature in a very critical application. The argument was "we really don't need it, but since it's documented you may as well make it work".

Bug #1 (my fault):
I added:

Code:

---

i = -1;

---

where I should have added:

Code:

---

i -= 1;

---

Bug #2 (my boss' fault):
Due to "too much to do" at the QA department I was told to test the application myself. My answer was "I'm not sure that's a clever decision. There is a good reason why we got a QA department, and there is probably a good reason why they've got too much to do...". Anyway, I was told to "think" like they did down at QA, and to do my best... I didn't find any bugs.

Bug #3 (the techies fault):
When done testing the application I went to the guy responsible for the software releases. He had the final word on any software that was released into the production environment. They had ofcourse strict rules on all software releases, planning, scheduling, avoiding peak hours etc, etc.. But apparently that didn't apply this day. The software went straight into production (at peak hour).

Bug #4 (operational departments fault):
It took about 2.4 seconds before the bug manifested itself and the CPU went straight to 100%. The first thing to be knocked out was ofcourse the software based KVM. Now we were blinded and in an early state of panic. The server then stopped resposning to network requests, with one exception. It was still sending 'OK' heartbeats to it's $100.000 passive failover server, so the failover server did nothing. In a higher state of panic they reasoned that "we need to pull the plug to terminate those heartbeats". So they did. All went silent.

After a couple of minutes they found the real reason why the active/passive failover solution didn't work. After a hardware test a couple of months earlier they never performed the manual failback routine, so we had been running on the failover server all the time while the main server was totally shut down.

Alot of phones went dead that day... but luckily I wasn't blamed :rolleyes:

- petter

petter,

That is exactly the type of tale I was looking for. A wonderful example of "the rule of 10's". I wonder if anything was done to the underlying business processes to reduce the rsik of this type of error happening again...

My worst blunder was ...

I was to write to a file a record id where I stopped printing the report for each day. And as each day begins, the program would start printing from the record id and stored where it stopped, in that same file. Unfortunately, I was told to store that file in some obscure places the operator wouldn't know, and I stored it in a system folder! As I was testing it, everything went fine because I was logged in as an administrator. However The operator logged in as a standard user, and you know the rest of the story. Well, that was long time ago.

Nowadays, to be compliant with Vista's UAC, programmers store their program-specific data or user data in those UAC recommended folders.

CBasicNet,

Thank you for your contribution. :wave: :wave:

For those who may be looking at this thread, there is a motive behind my posting....

I am currently putting some material together for publication. Part of this is a discussion on the "rule of 10's", which I firmly belive in (within an orer of magnidute, of course ;) )

All of the "blunders" listed so far, could have been easily avoided by following "standard" processes, yet these processes were either not in place, or not followed.

The eternal question is "How to get upper management to understand that placing and enforcing these policies SAVES MONEY!!! (and if you avoid a serious phyical injury, that involves a LOT of money!)

Sorry David...I don't make blunders as they tend to turn into fireballs...so I try my best not to do that, it has a tendency to ruin someones day.