If you have ever been involved in software testing as a Quality Assurance (QA) engineer, you must have asked yourself, “How did I miss that?” Although the role of a QA engineer is to find as many bugs in the application as possible before releasing for production, defects found by the customer, referred to as “leaked defects,” should be seen as a gap in the process and not necessarily just identified in the final stage. From analyzing the requirements to coming up with a design, implementation, testing and finally, deployment, the Software Development Life Cycle (SDLC) is designed to ensure bugs are caught throughout the process. Keep in mind, humans are managing these steps and, therefore, defects can be inadvertently introduced at any time, even early on when gathering the requirements. Therefore, testing is everyone’s responsibility and must be embedded in every stage of the SDLC. Risk analysis, adopting proper test techniques and setting priorities are instrumental in delivering a quality product.
Most software practices use leaked defects as a quality metric that speaks to the integrity of the testing. Some practices have a zero-defect objective. While leaked defects may not cause a life-or-death situation, there are financial implications, as well as delays, to consider. It is therefore imperative that software testing becomes a mindset, rather than a task, with the entire team committed to a zero-leaked defect culture. This means the product manager, software developers, QA engineers and product owner all embrace this approach. Everyone involved needs to abide by guiding principles defined by the team, so ultimately this becomes a standard practice. In addition to not tolerating defects, the team needs to focus on how the defects are introduced, rather than who introduced the defect.
Why Do Defects Happen?
Testing not done early enough. In traditional software development models, testing is not done until the end of the cycle. The team adheres to a timeline to release the product and there is never enough time to test everything. Also, most people have the notion that codes must be in place to begin testing. Leaving testing to the last step is practically an invitation for leaked defects. It becomes more complex and costly to address defects found late in the cycle with a higher risk of introducing another defect. More often than not, QA engineers are not involved early in the SDLC – the requirement analysis stage where static testing (the process of testing a work product without code) is done. This is a mistake, and a costly one. Missing out on something as fundamental as analyzing the requirements sets things off on shaky grounds. Finding and correcting requirements defects in production can cost 100 times more than the cost to correct if found during requirement analysis.
Limited test scope. Lack of understanding the requirement makes it impossible to correctly define the scope of the test. Consequently, the test cases created will fail to cover the required preconditions (prerequisites), procedures (inputs/actions) and expected results. The complexity of the product also determines how much testing can be done. Trying to simulate the real environment can also be challenging as some issues can only be reproduced in certain conditions. It is important to consider the QA engineer’s level of experience as this could also play a significant role in designing applicable test cases.
Time constraints. Because of project deadlines and limited resources (specifically QA engineers), it is virtually impossible to test every possible user scenario or replicate every found defect. Testing every possible data value, path through the code, combination of inputs is an infinite task – and can never be completed. Even small projects could require significant testing resources that could yield little added value and likely a delay in the product release.
Working Toward Zero Defect
Needless to say, zero defect is technically impossible. However, teams can change their way of thinking that reinforces the notion that defects are unacceptable and do the right thing the first time. It is imperative that every “t” is crossed and every “i” is dotted.
Here are some ways to achieve this approach:
Adopting Shift Left Testing
This popular technique adopted by the agile software development approach ensures testing starts early in the SDLC, to the left side of the curve, when the requirement is being analyzed. Involve the entire team including the QA engineers in the requirement review. Discuss possible failure modes with the design and their effects. These forms of static testing help ensure the requirements are understood and, in some cases, could get changed if gaps (bugs) are identified.
Adopting Multiple Testing Techniques
- Integration testing, manual testing, and exploratory testing are fundamental as is the need to test features in parts, units and the entire system holistically.
- Perform static testing – testing with no line of code, An example is conducting a Failure Modes and Effects Analysis (FMEA). FMEA involves a review of as many components as possible to identify potential failure modes in the system and in turn, the causes and impact of these failures.
- Verify and validate features. While verification helps to ensure the feature is correctly developed, i.e., works as it should, validation goes a step further to ensure the product meets the stakeholder’s requirements. It helps look at the overall system and potentially uncover system related bugs.
- Identify the fragile components in the code – any change could break them.
- Perform a field test in the environment or at least as close as possible to the environment.
- Automating high-level and routine tests keeps you on a sustainable path – it is key for early defect detection and coping with time constraints. In an agile testing practice, automation of tests at lower levels of the pyramid is practiced, improving testing efficiency and effectiveness.
This involves two team members working on the same computer to test the same application. One of them plays the role of the driver and executes the tests (clicking) while the other, called the navigator, directs the session. Although there is a common notion that pair testing is for developers and QA engineers, it can and should involve anyone in the team including the end user, where possible. While there are different outcomes for each partner, everyone has a common goal-verify and validate the requirement that includes finding defects.
Perform Root Cause Analysis for Every Leaked Defect
The objective of the root cause analysis must be clear to all the stakeholders. It is a process to understand how the defect was introduced and not intended to point fingers. This helps the team to be open-minded, identify the root cause and subsequently gaps in the process that need to be filled to prevent a recurrence.
Backup for the Backup
Since defects cannot be completely eliminated, mitigation measures, such as backup contingencies need to be put in place, especially for defects with severe consequences. This will minimize the impact of defects to as low as possible.
- Quality is the responsibility of every team member and should be embraced by all involved.
- Root cause analysis of every leaked defect is not optional.
- Have a backup of a backup for key processes to reduce the impact of leaked defects.
- There is indeed no software without defects as to err is human and testing is inexhaustive. Having a zero-defect mindset makes the difference. In a true zero-defect approach, there are no unimportant items.
Headline photo: Software Development Life Cycle
Olubunmi Adeyemi has more than 15 years of experience in the oil and gas industry with the last 10 years in process and operations management, where she has focused on developing, testing, maintaining and delivering state-of-the-art tools, technology and software for drilling and reservoir evaluation. She has held various roles in software development – technical, quality and project management – across three continents (Asia, Europe and North America) for one of the largest oilfield services. Adeyemi is a certified project management professional and holds a B.Sc. in Chemical Engineering and a testing certification from the International Software Testing Qualifications Board.
Oil and gas operations are commonly found in remote locations far from company headquarters. Now, it's possible to monitor pump operations, collate and analyze seismic data, and track employees around the world from almost anywhere. Whether employees are in the office or in the field, the internet and related applications enable a greater multidirectional flow of information – and control – than ever before.