Software Production Defect — Who not to blame
Published: 2022-11-05 06:25:32

“We have a defect in the production” — Try to take a trip down your memory lane and retrospect how you and your team reacted to this. You can imagine the reaction and effects of this situation from the Early ’90s till today has not changed alot. Technology has changed a lot, we are in the Agile, SaaS, DevOps, Cloud era today but we continue to react, address and continue to make testing teams the root cause for this.


In the era of AI, ML, Blockchain, SaaS, cloud the reasons for production defects are many.

How did the testing team let this defect leak — this has been echoing and still echos in many company meetings.

Production defects could be anything ranging from functionality not working as expected, software being unable to cope with the load, security vulnerabilities discovered, accessibility defects, and many more.

Their severity would depend on how much it's going to cost/dent in terms of customers' success. Ultimately it's all about customer success and the kind of impact it could have. Customer success would drive product success, revenues, and the future of the company

When a critical production defect is found, teams scramble to see how quickly they could fix this. Effective incident management kicks in. Many people get paged, they will be pulled into meetings and a lot of collaboration & coordination that would cut across teams across organizations depending on the defect. In today’s world SRE team owns and drives these.

Postmortems would be conducted, one of the key items is to include all sources/details that contributed to this situation. Today everybody is pushing for blameless postmortem culture and which is very important for the teams to succeed.

SRE plays an important role in this era to ensure software work as expected and drive customer success

Let's analyze the statement “How did the testing team let this defect leak”

After root causes for the defects are found, it boils down most of the time to leaders asking “why did we not conduct adequate testing to catch this defect? It rarely gets asked how did product management miss on this, how could development team induce this, and why did Business analyst/ Scrum master not highlight and do something about this.

Testing cannot catch every defect that is out there due to the sheer volume of effort that is required and having to work under a time crunch, budget constraints, and many others.

Everybody ranging from product management, R&D, Services, and Cloud is equally responsible for the defect.

You might be thinking why and how?

Let's look at Product management

SLA, SLI, and SLOs are not part of the product requirements. It's the responsibility of SRE / cloud, product management thinks they don't have anything to do with this
Non-functional requirements are not fully considered nor planned
Product management things once the product is pushed to production, it isn't our responsibility how it works and runs
The extensive testing that needs to be conducted is not factored in as part of the product requirements
Tools, platforms, and solutions that are needed to test the requirements are not budgeted and factored in as part of the product requirements
Product management thinks once we deliver working software, the job is done
How can feature/development teams carry out effective testing, what training, tools, and budget needed, are never discussed in/with product management
Working software is not functionally working, it's beyond that. Scalability, availability, reliability and others don't come up until the product goes out
Customer success is not completely understood by product management. Pushing products out of the door should not be the only success criteria
Many other reasons
You might be thinking what has production defect got to do with all of the above points? The fact is any of the above could be the reason for production defects.

Next lets look at R&D which comprises of development team. To give a context development teams are the scrum team which comprises of scrum master, developers, testers, business analyst. Depending on the maturity of the orginization there could Devops, Documentation, Non functional testing, translation, release management and others that would operate as part of common services team Or part of the development teams.

Many development teams proudly say they follow scrum without realising they are running waterfall methodology inside scrum. The development teams are so much dependent on other common services team to actually deliver value in each sprint and release a product feature.

Without a strong leader with a background of testing, testing is being planned, perceived and executed to suit each of the individual needs. Sometimes an individual to cover his/her incapabilities he/she defines and executes testing in his/her own way which is easy, irrelevant and inefficient. And he concinves the development team, product management to belive this is the right way to test. And they proudly say we have automated 100% and we have provided 100% test coverage.

Executive leaders many a times fall into this trap and also there are leaders who define in their own ways how testing should be planned, conducted and executed.

On one side technically weak people defining how testing should be performed, on another side testing folks fighting battle around underpay, being dictated, demotivated, underskilled and others.

Everybody looks at an easy way, that is all green on dashboard which says testing is 100% completed, 1000 test cases have been executed, and nobody digs to find out if 1000 test cases are all redundant and provide very low coverage.

In today’s world speed is everything, and whole orginization would be happy if teams shows dashboard where they display testing 100% automated and executed and ready to release.

There is no doubt defect would be found in production with all these inefficient process and planning. Who to be blamed, everybody who are part of R&D to be blamed.

Likewise other parts of the orginization like Cloud, SRE, services all share their bit towards nurturing, growing and releasing defects to production.

Who is not to be blamed…………??

Keep thinking as I write part 2

Thanks for reading.

Am open to writing, please feel free to contact me @nutanrp1@gmail.com