[3/10] How to set up a growth measurement system

Understanding metric pairs, improving growth models, and root cause analysis methods

Jan 24, 2023

Welcome to the 3rd essay of the 10-part series, Firestarter, on the set of skills to lead growth function at a startup.

In the 1st essay, we went about turning a business idea into its growth formula, and then building its growth model. In the 2nd one, we learnt how to turn the high-level growth model into an actionable growth roadmap.

In this essay, we will learn how to set up a growth measurement system. That is:

how to continuously measure how our growth model and roadmap is doing out there in the real world,
common techniques to identify reasons for any divergence, and
the frameworks to change the growth model or roadmap accordingly.

At the end of this essay, we would have completed the first section of this series on growth planning, and can give ourselves a pat on the back for making it so far.

Why & how to set up a growth measurement system

Startups often oscillate between the two states of not consuming enough data or trying to consume a lot of data.

You are in the ‘not consuming enough data’ state if you know the revenue numbers, the marketing costs, the cash in the bank, and, yeah, that’s about it. Now don’t get me wrong. This is good too. That is to say, not knowing them would be worse. But the problem with tracking just these numbers is that these are output metrics.

You are in the ‘trying to consume a lot of data’ state if one day you felt bad about not tracking enough metrics, and went the opposite extreme: you opened a spreadsheet and wrote down every metric you could think of and then asked someone to update the sheet with all those numbers every morning by eleven. But it has been a few days since you last went through it.

But why is the second state bad, in theory? Isn’t being on top of numbers supposed to be a good thing?

The problem is that such a laundry list of metrics will not be effective in achieving the objective of a measurement system. So, let’s start with the why. Why set up a growth measurement system?

The core objective of setting up any measurement system is threefold:

To know the overall health of the business at a glance with the output metrics
To be able to do a quick diagnosis, if some output metric is off the trend, with hierarchy of input metrics
To be able to make predictions and catch things going off the rails early, with the lead indicators

Let’s understand them with an example. Suppose we are running a call center, with 10 agents. Every morning we come in, we want to, first, track overall health of the operations. How do we do that?

Any system will have two complementary metrics – a volume metric and a quality metric – that capture the overall performance at the highest level. You will see this pattern everywhere, once you internalize this concept. Such as, pages read versus knowledge gained. Calories ingested versus nutritional quality. GDP growth versus quality of life. In fact, negative externalities often arise because there is no counter-balancing quality metric being tracked.

So, for this call center, the top-level volume and quality metrics can be the number of tickets cleared and the customer satisfaction score respectively.

But do these numbers mean anything by themselves? Is 720 tickets cleared the previous day with a 57% satisfaction score good or not? How do we know that? This context can be provided by adding how it has trended over time i.e. a time series trend. And/or adding the targets or projections against these actuals.

Alright. So, we set this up and have a high-level picture of the 2 most important metrics, as well as their context. We look at this in the morning, are happy with the numbers, and can move on with our day, without having to go through a laundry list of every data point the business has generated.

But what if the metrics are significantly deviating from the past trend, or from the projections we had made? How do we do a quick diagnosis?

This is where the metric hierarchy, that we covered in the previous essay, comes into play. Both the volume and quality metrics will need to be split into their respective input metrics. We can quickly look at these level-2 metrics and understand the reason.

For example, let’s say we have started a new content app. What should the top-level volume and quality metrics to track its growth be? It can be the number of unique people opening the app and average engagement per user respectively. We are also tracking this over some time instead of just the previous day’s numbers, so that we are aware of the past trend and any pattern of minor variation (e.g. higher app opens on weekends than weekdays).

Now let’s say, the number of unique people opening the app (i.e. number of active users) is dipping. How do we know what’s happening? From the ideas of the previous essay, we will have its metric hierarchy. For level-2, the formula can be the sum of new (first-time) users and returning users. For level-3, the number of new users can be split into different marketing channels, and the number of old users is split into different cohorts.

So, before jumping into investigating myriad possible reasons every time the top-level metric seems off (did a new competitor launch, is the new app version crashing, etc.), we can first make a quick diagnosis by going down the metric hierarchy, and have a clear sense of the specific direction to go in for identifying the problem. We will come back to some common ways to do root cause analysis in the last section of this essay.

So far, we have talked about setting up a top-level tracking of volume and quality metrics, and their level-2 and level-3 input metrics for quick diagnosis. Is this all? The measurement system defined above looks at the past and catches any aberrations early. But can we have a system which predicts if these metrics are about to deviate from the trend? Enter the concept of leading indicators.

Let’s say that the content app has good user engagement. However, that engagement is coming from content created by creators in the near past. Maybe in the last 48 hours. So, even if the content creators suddenly face a problem in creating content, there will be enough content liquidity in the system for some time that the top-level engagement metric on the consumer side might not show a dip. However, if the problem continues, the engagement metrics will dip in future. Hence, drop in the number of new content items is a leading indicator of drop in the consumer engagement metric.

How do we identify the leading indicators of any metric? It will come from the deeper context of business drivers for that metric. For example, employee attrition rate is one of the key metrics for the people system in an organization. However, it is a lagging indicator of employee engagement. While that number might seem okay at the moment (against time series trend, or against projected numbers), it does not really predict what the attrition rate will be in the next quarter. The leading indicator for it can be employee satisfaction score. That is, even if the attrition rate is stable currently, but if the satisfaction score has started trending downwards, it is likely the attrition rate will increase in the future.

Another example is retention rate of customers for a platform. While it is the most important independent metric in the growth model, it is a lagging indicator of customer satisfaction. In that, a stable retention rate does not guarantee the rate to be stable in future too. Declining customer satisfaction scores (or Net Promoter Score - NPS) will mean the retention rate, and therefore revenue from the existing customers, will trend downward in future.

So, in summary, a good growth measurement system needs to have:

Complementary volume and quality metrics, for a quick overview
Trend of these metrics, and/or comparison against projections, for context
An optional view of metric hierarchy, for quick double-click and diagnosis
Early indicators of independent metrics, for quick prediction

How to improve the growth model with the growth measurement system

Now, let’s say we have a situation on our hands where the actual numbers are consistently veering away from the projected values. If we do not change the model being used to make the projections, it makes the projections meaningless. But if we keep changing projections every time it’s different from actuals, it makes the act of building projections futile. So what do we do?

Let’s go back to the growth model we had built for the mobile game app business, in the previous essay. It had, in its growth formula, 10 million potential customers and a net value of $10 per customer ($12 LTV minus $2 CAC). In its growth model, there was 6k of net revenue projected in the 1st month, 11k in the 2nd month, and 15.8k in the 3rd month.

In this model, the independent metrics that are driving rest of the calculations were:

Number of new users per month. Especially the degree of acceleration in that number i.e. 1k new customers in the first month, 1.5k customers in the second month, and so on.
Cost to acquire new users
% of users returning in successive months i.e. retention
Revenue per ad impression

And so, once we dig into these input metrics to find the reason for actual numbers diverging from what was projected in the growth model, we might find one (or more) among the following things happening:

Number of new users per month is increasing but not at the pace that was expected
Cost is not constant as we are trying to accelerate the number of new users
Fewer than projected % of users coming back in successive months
Revenue per impression being paid at lower rate by the ad platforms than was projected

At this point, we have to dig into the business context of the original assumption. For example, why did we assume that the number of new users per month will keep accelerating? Was it as a result of more channels to acquire the user being opened up? Was it because of more supply levers being opened up? Was it based on industry benchmarking of similar stage apps? Similarly we have to look into the business logic for quantitative assumptions for other metrics too.

We can reflect on the assumptions and see if they have been proved false. If yes, we have two options:

Modify the long-term growth model by changing the independent metric to reflect the actual number. That is, if month-1 retention is actually 12%, use that in the growth model rather than, say, 15% which was originally used.
Modify the short-term growth roadmap by making it a key result to improve the independent metric by picking the right lever against it. In this case, increasing month-1 retention from 12% to 15% within a certain time period using certain levers (we will cover the frameworks to pick the right levers for different independent metrics in a growth model, in the next section of this series - essays# 4 to 7).

After one of the options has been chosen, we can operate for some time, collect new data, and re-visit these steps.

This is similar to the build-measure-learn loop described in the book Lean Startup by Eric Ries. We are building the growth model and roadmap, with some business assumptions. As the business gets operational, we are measuring the actuals against the projections. And, we are accordingly learning from the difference between actuals and projections and the root cause for it. And subsequently, either changing the assumptions in the growth model or changing the growth roadmap to focus on the errant assumptions.

We can run through this loop every quarter (or the short-term planning horizon being followed at the startup). However, in order to be able to tweak one or more assumptions in the growth model, without having to re-do the entire thing, it should be fairly modular.

A well-compartmented growth model will have 3 parts:

Input: with all independent metrics and business logic for these assumptions
Processing: where all calculations take place, but with only transformations (formula in case of spreadsheets) and no hard-coded values
Output: a simplified view of projections for top-level metrics

If there is such a clear separation of concerns in the growth model, one can look at the input sheets and get a clear picture of what the business drivers are. And such an abstraction will help non-specialists look at assumptions in the input and the simplified output views, without getting bogged down by all the transformations in the processing part.

Common Root Cause Analysis methods

We saw how divergence between actuals and projections between output metrics can be looked into, by traversing down the metric hierarchy and finding the independent metrics causing it. For example, retention % dropping and causing a drop in revenue against projected revenue. But how do we investigate a drop in the independent metric itself? Retention %, in this case.

I will share with you three frameworks which will serve you well in most root cause analysis across multiple growth problems. They are:

Funnel thinking
Segment thinking
Cohort thinking

Let’s say the default rate for a bank has gone up massively. How do we look into it and understand the root cause of the problem?

One way to do this is to dice the data by funnel i.e sequential steps. An account is considered to have defaulted if, say, 3 successive payments are missed. So, we can first look at what % of people missed the 1st payment. Of them, what % missed the 2nd payment. Of which, what % missed the 3rd payment. And try to see which step of the funnel has had a sharp increase.

If the increase is across all parts of the funnel, it is not the root cause. This is the crux of root cause analysis, or discovering any data insight in general: you have an insight only when you have arrived at point where the patterns are diverging.

Since it’s not a problem with funnel steps, we can dice the data by segments. A segment of a population is sufficiently similar to each other and dissimilar to other segments. We know of common ways to segment individuals (race, gender, age, etc.). In this context, relevant parameters to segment would be the size of business. Are the large businesses defaulting, or the medium ones, or are the small ones? Another parameter could be sector. Are the primary industries defaulting, or are manufacturing industries defaulting? Or it could be a combination of parameters: while other businesses seem to be fine, it’s the service industries with a turnover more than 100 crore that are defaulting.

And, if there is no pattern emerging either from funnel or segment (increase in default rate is across all stages of funnel, and across all segments), we could slice the data by time. Also known as cohort, or vintage. That is, all loans that were granted in 2020 form one cohort, all loans that were granted in the 4th quarter of 2021 form another cohort, and so on. So, if we find that default rates were high only for the loans which were granted in 2019, and not for the other cohorts, we have zeroed in on an insight.

To be clear, cohort is not the same as segment. For example, a hotel which had its loan granted in 2020 will be in a different cohort than a hotel which got its loan in 2021, but they will be in the same segment (i.e. hotels) in this analysis. While two companies with different turnover and operating in different industries, but which got the loan together would be in the same cohort, albeit in different segments.

Another example of the difference is that generations (e.g. Millennials, Gen-Z) are cohorts, while income brackets (e.g. HNIs, Middle Class) are segments.

Let’s go over these techniques, one more time, but in the context of a growth model. Let’s go back to the example of the lifestyle brand we had modeled in the first essay. We had arrived at an LTV of 4200 with 2.1 orders per customer and 2000 as margin from every order.

However, after 3 years of operation, we notice that the LTV has decreased from 4200 to 3200, with the decrease coming from average orders per customer dipping from 2.1 to 1.6. As we saw above, we have two courses of action:

Change the growth model: by changing the underlying assumption of orders per customer
Change the growth roadmap: prioritize identifying and executing on levers to increase orders per customer

But, before that, as the first step, we need to understand why it decreased from 2.1 to 1.6. To identify the root cause for it, we have to do funnel thinking, then cohort thinking, and then segment thinking.

In funnel thinking, we have to look at sequential steps for all customers. So, we will look at what % of customers were coming back to transact in the 2nd month originally and what the number is now. Then in the 3rd month, and so on. Is it a dip across all the months? Or in a certain month of the customer's journey.

In cohort thinking, we have to group the customers based on when they were acquired. So, we look at LTV of customers who were acquired in January of 2021, February of 2021, March of 2021, and so on. Is the dip across all the cohorts? Or for a few specific cohorts.

In segment thinking, we have to group the customers based on their attributes. So, we can look at LTV of different customer persona, or by segmenting based on the marketing channels or campaigns they were acquired from. Is the dip across all segments? Or have we over-acquired over time from a persona or channel which had low LTV.

Often, the search for a divergent pattern is not so clear cut: it’s not 100% this or that; but 80% due to segment and 20% due to one funnel step; or 70% in this funnel step and 30% in another. However, in most cases, following these steps will lead you to the major driver for the change.

Summary

In this essay, we learnt the following concepts:

How to set up a growth measurement system for any business, using
- Volume and quality metrics as complementary pair
- Trends and hierarchy of input metrics
- Leading and lagging indicators
How to improve a growth model, using
- Separation of concerns
- Build-measure-learn loop
Common root cause analysis techniques, especially
- Funnel thinking
- Cohort thinking
- Segment thinking

With this 3rd essay, we have reached the end of the first of three sections of the Firestarter series. This section was on growth planning, and by now you should have the skills to build a growth model for an idea, build its growth roadmap, and set up a growth measurement system, along with tweaking the model after finding the root cause of deviations.

With the planning base covered, we will now dig into the juicy executional bits of growth over the next 4 essays (published on Wednesdays starting February 1st). Answering questions like: How do we decide which growth levers to pursue? How do we decide the growth channel strategy for a business? How do we plan a campaign? And many more concepts and frameworks which form the pareto of executional aspects of different functions within growth. See you there :)

If you have any suggestions on this article, or need any clarifications, I am reachable at sudhanshu@skilletal.com.