Measuring is good. Metrics are essential for knowing deeply your product and for making more accurate decisions. But you must be very careful. The excess of metrics can get in your way and overshadow the full knowledge of your product.
Instead of excessively collecting metrics, you should follow a data-driven approach. When you manage a product in a data-driven way, you run experiments that generate data that feed your decisions for the next experiments. It is this simple, and it seems to be an excellent way to manage a product. But there are a few problems with this approach.
Instead of data-driven, we need to be data-informed, in other words, using data as one more input to decision-making, not the only input. Take the experience, the intuition, the judgment, and the qualitative information into consideration, along with the metrics to increase the quality of decision-making.
One of the best examples I can quote has to do with the website hosting product from Locaweb. Through the years, in a reasonably informed way, but always counting on a lot of intuition, we altered our hosting plans in order to have more space on disk, data transfer, and the number of sites that could be hosted in each plan sold. In 2011 we noticed that more than 90% of our clients were choosing the basic plan because it attended to the needs of most people who needed a website.
We wanted that the largest plans would play a bigger role in sales, but with the limits we had, there was no motivation for the clients to buy them. We thought of changing the plans for new subscriptions, decreasing the limits to incentive clients to get bigger plans. However, as this was a significant and very sensible change, we brought in a consulting expert in pricing that helped us collect and analyze several data, to suggest which was the best chance to be done.
We implemented the changes suggested in 2012. There was a little variation in the distribution of plans, but the number of plans contracted per month didn’t change. Moreover, it even decreased a little, which resulted in no alteration in the amount of monthly revenue. That is, we spent time and money collecting and analyzing data that made us take a decision that didn’t change the company’s result. Maybe if we had defined the changes more intuitively, we would have saved money and would know the result of the decision more quickly.
There are two great tools for taking A/B tests — the Visual Website Optimizer and the Optimizely. I used the services from Visual Website Optimizer, that gives you one free month to test some hypotheses about the home of ContaCal:
In less than 30 minutes, I was able to create 4 versions and began to run the test. I decided to test two things. One was for the color of the create account button if it was going to make a difference in the number of people who would click on it:
The second test: if changing the explanatory video for a photo would increase or decrease the number of people who would click on create account button:
The result was:
After seeing this result, I got the impression that if I put the green button with the healthy family picture, I was going to increase the conversion even more. So, I decided to run this test and the result was:
Therefore, take care, appearances can be deceiving! Make experiments before taking conclusions!
Lastly, aside these precautions, it is necessary to take care with the analysis paralysis effect, that is, analyzing data all the time and not taking any action. As seen in the picture from xkcd.com (2014), analysis paralysis can cost you a lot:
The metrics are not the reason for the existence of your product. The users, their problems, and the strategic goals of the business are the reasons for its existence. In other words, metrics are a mean and not the end, the results, and the goal. Therefore, use them more like one of the tools to help you to drive your product in the right direction.
With this, we close the theme on the growth stage of a software product life cycle. We understood how to deal with client feedback, what it is, and how to prioritize a roadmap. We also saw several types of metrics, including the conversion funnel, engagement, churn, global and individual financial metrics, revenue and negative churn, NPS, the loyalty metrics, and we also approached some considerations on metrics.
In the next chapter, we will better understand the next stage of a software product life cycle: maturity.
I’ve been helping several product leaders (CPOs, heads of product, CTOs, CEOs, and tech founders) extract more value and results from their digital products. Check here how I can help you and your company.
Do you work with digital products? Do you want to know more about how to manage a digital product to increase its chances of success, solve its user’s problems and achieve the company objectives? Check out my Digital Product Management bundle with my 3 books where I share what I learned during my 30+ years of experience in creating and managing digital products:
You can also acquire the books individually, by clicking on their titles above.