Back in 2016, I wrote an article about the reasons that motivated us at Locaweb to extinguish the QA function in our product development team. At Locaweb we had 12 QAs, some with a developer profile and others with a SysAdmin profile. In proposing the extinction of the role of QA, some of the QAs became devs while others have taken on the role of sysadmins. The reasons that motivated us to extinguish the QA function at Locaweb are:
When I joined Conta Azul, they had also just extinguished the QA role, and the former QAs became business analysts, mainly helping product managers.
I’ve seen other companies also discussing the need for QAs and in some cases a heated debate emerges around this topic. However, having or not having QAs should not be the center of the discussion. Having or not having this role is a solution to a problem, normally stated as “how can we improve the quality of our product?”, and this problem should be the center of the discussion.
A simple Google search about software quality will yield tons of definitions normally around meeting functional and non-functional requirements. When software does not meet a functional or non-functional requirement, it has a defect, a bug. So, to improve the quality of a software product, we need to work on two things:
A good way to control this is having a weekly measurement of your bug inventory and new bugs per week and discuss this every week with the team. We did this at Gympass. We defined at the beginning of every quarter what’s the target for bug inventory and average new bugs per week.
The image above shows the evolution of our bug inventory for Q2 of 2019. We started the quarter with 215 bugs in our inventory and we aimed at a target of less than 166 by the end of the quarter, a decrease of almost 23%. We were able to end the quarter with an inventory of 136 bugs, a 36% decrease. We did this by focusing not only on resolving bugs in our inventory, but also controlling the number of new bugs per week.
In Q1 2019 we had an average of 26.2 bugs created per week. During Q2 we reduced this average to 17.4 new bugs per week, to a total of 226 new bugs during the quarter. That’s a 33% decrease in the number of new bugs per week.
This looks like a very good improvement, right? But there’s plenty of room for improvement there. Let me explain the math of bug management:
If we were able to reduce our bug inventory from 215 to 136, this means we solved at least 79 bugs. However, we created 226 new bugs (17.4 new bugs per week x 13 weeks) during the quarter. So we solved 79 + 226 = 305 bugs during the quarter, that’s a lot of bug correction work. If we had generated 90 new bugs during the quarter, an average of 6.9 new bugs per week, instead of the 226 new bugs, we could have zeroed the bug inventory.
An additional aspect of the bug resolution to be measured is the resolution SLA, i.e., how many days the team took to solve a bug from the day the bug was first identified. To do this, we classified the bugs by its severity, which is the impact it causes to the users and to the business. Highest severity bugs are the ones that we need to solve in the same day. High severity bugs in 7 days and medium severity in 14 days. The chart below shows how we were at Gympass in Q4 2019.
This is not the ideal visualization because it only shows a picture of the moment, and not an evolution. In order to understand the evolution of any metric, you need to see how it performed in different points in time.
Any user prefers to use a product with good quality, i.e., that behaves as expected. This is front and center to provide a good user experience.
Besides the user experience, there’s also another important aspect to consider when we talk about quality and bugs.
Whenever someone needs to work on solving a bug that was found in a software product, this person needs to stop working on whatever she is currently working to be able to work on the bug. This is an interruption in the workflow. If this person were able to deliver the software without that bug, she could continue to work on new things without interruptions, which will make her more productive.
Do you work with digital products? Do you want to know more about how to manage a digital product to increase its chances of success? Check out my new bundle Digital Product Management with my 2 books where I share what I learned during my almost 30 years of experience in creating and managing digital products.