QA or no QA? That’s NOT the right question…

Back in 2016, I wrote an article about the reasons that motivated us at Locaweb to extinguish the QA function in our product development team. At Locaweb we had 12 QAs, some with a developer profile and others with a SysAdmin profile. In proposing the extinction of the role of QA, some of the QAs became devs while others have taken on the role of sysadmins. The reasons that motivated us to extinguish the QA function at Locaweb are:

  • When there is a QA function separate from the software development function, it is common to hear phrases like “the feature is ready, now it is in the QA phase”, which denotes a waterfall culture. This culture can considerably increase development time and negatively affect the quality of the software.
  • When there is a QA function separate from the software development function, it is also common to hear phrases like “why didn’t QA catch this bug?”, Which denotes a culture of finding the culprits. This culture can be very harmful to the team’s engagement and motivation and, consequently, negatively impact the quality of the software.
  • Quality should not be the concern of one person or one team, it should be the concern of everyone who is working on creating the software.
  • Quality is a non-functional requirement, that is, a requirement that specifies a criterion to assess the operation of a software product, which is different from the functional requirement, which specifies a behavior of the software. Performance, scalability, operability, monitorability are some examples of non-functional software requirements that are just as important as quality. Even so, there are no defined functions for performance assurance, scalability assurance, operability assurance, and monitorability assurance. Why is quality the only non-functional requirement that has a specific dedicated function to ensure it?
  • QA focuses on ensuring the quality of the software development process. If a separate role is needed to ensure this quality, why is there no need for a separate role to ensure the quality of the product management process, the UX design process, the product marketing process, the sales process, the financial process of a company?
  • There was a concern among devs that, if the dev herself will now have to test, deliveries will take longer to get ready. This concern existed because the devs considered that their work was finished – and the delivery was ready – when they passed the story to the QA to test. However, this dev’s readiness concept is incorrect, as she just passed the story on to the next phase, the testing. From the user’s perspective, the story is only ready when the user can use the new feature. So the time the delivery stays in QA is still dev time, even not being in the dev’s hand anymore. And this time gets even bigger when the story goes through QA but is rejected and has to go back to development.

When I joined Conta Azul, they had also just extinguished the QA role, and the former QAs became business analysts, mainly helping product managers.

I’ve seen other companies also discussing the need for QAs and in some cases a heated debate emerges around this topic. However, having or not having QAs should not be the center of the discussion. Having or not having this role is a solution to a problem, normally stated as “how can we improve the quality of our product?”, and this problem should be the center of the discussion.

How can we improve the quality of our product?

A simple Google search about software quality will yield tons of definitions normally around meeting functional and non-functional requirements. When software does not meet a functional or non-functional requirement, it has a defect, a bug. So, to improve the quality of a software product, we need to work on two things:

  • reducing its existing bugs;
  • not generating new bugs.

A good way to control this is having a weekly measurement of your bug inventory and new bugs per week and discuss this every week with the team. We did this at Gympass. We defined at the beginning of every quarter what’s the target for bug inventory and average new bugs per week.

For instance, say the team started Q2 with 220 bugs in our inventory and we aimed at a target of less than 150 by the end of the quarter, a decrease of almost 31%. Setting targets and discussing it every week helped the team understand and decrease our bug inventory. We do this by focusing not only on resolving bugs in our inventory, but also controlling the number of new bugs per week. In our example, let’s consider that in the previous quarter we had an average of 25 bugs created per week and in Q2 the team was able to reduce the average to 15 new bugs per week and got to the end of the quarter with a bug inventory of 135 bugs, 15 below the target. This looks like a very good improvement, right? But there’s plenty of room for improvement there.

Let me explain the math of bug management:

If the team were able to reduce the bug inventory from 220 to 135, this means the team solved at least 85 bugs. However, during the same quarter the team had an average of 15 new bugs per week, they created 15 new bugs per week x 13 weeks in the quarter totaling 195 bugs. So the team solved 85 + 195 = 280 bugs during the quarter, that’s a lot of bug correction work. If the team had generated 60 new bugs during the quarter, an average of 4.6 new bugs per week, instead of the 195 new bugs, the team could have zeroed the bug inventory.

An additional aspect of the bug resolution to be measured is the resolution SLA, i.e., how many days the team took to solve a bug from the day the bug was first identified. To do this, we classified the bugs by its severity, which is the impact it causes to the users and to the business. Highest severity bugs are the ones that we need to solve in the same day. High severity bugs in 7 days and medium severity in 14 days.

Why quality is so important?

Any user prefers to use a product with good quality, i.e., that behaves as expected. This is front and center to provide a good user experience.

Besides the user experience, there’s also another important aspect to consider when we talk about quality and bugs.

Whenever someone needs to work on solving a bug that was found in a software product, this person needs to stop working on whatever she is currently working to be able to work on the bug. This is an interruption in the workflow. If this person were able to deliver the software without that bug, she could continue to work on new things without interruptions, which will make her more productive.

Summary

  • Questioning if a product development should or should not have a dedicated QA team is not the right question.
  • The underlying problem you are trying to solve is how to improve the quality of your product.
  • A good proxy metric for quality is bugs. Bug inventory, new bugs per week and bug resolution SLA.
  • A product development team should have all of its members following these metrics and acting on how to improve them
  • Quality is front and center to provide a good user experience. But it is also key to increase the speed of your product development team. The fewer bugs a team has to fix, the more time it will have to focus on new things.

Digital Product Management Books

Do you work with digital products? Do you want to know more about how to manage a digital product to increase its chances of success? Check out my new bundle Digital Product Management with my 2 books where I share what I learned during my almost 30 years of experience in creating and managing digital products.

No alt text provided for this image

Be the first to like.

Leave a Reply

Your email address will not be published. Required fields are marked *