I have always wanted to learn Spanish, and after multiple attempts to learn it another way, I signed up for Duolingo. The app has a lot of behavioral design built into it, and enough has already been written about Duolingo’s product practices elsewhere: the icons that change expression if you miss a lesson, the streaks that track your record, the characters, Lily, Oscar, and others who are both endearing and entertaining. Duolingo reportedly ran over 600 experiments on streaks alone, and figured out how to make users come back every day. I was one of those users. If you had a 264-day streak as I did, would you give it up to try another language app?
The thing is, after 264 days, finishing lessons like a pro and earning XP points, I couldn’t fluently speak Spanish to order my dinner. The gamification is great, but conversational fluency was the actual point, and despite being their ideal user on paper, they lost me.
For Duolingo and apps like Instagram, Engagement is one of the core metrics that tie directly to revenue. More time in the app drives ad inventory, revenue, and subscriptions. That’s the business model. Engagement, however, is not a core metric for every app. The entire job of my LG app is to notify me when my wash or dry cycle is complete, or when my oven is done pre-heating. The Ring app’s function is to tell me when someone is at my door. If these apps were designed to keep us engaged, then their core purpose would fail. It would add friction to finish the task. Success in these apps looks like leaving as quickly as possible.
Engagement is not inherently a vanity metric. It becomes one when there’s no real link between the metric and the value it’s supposed to represent. The apps that don’t chase Engagement alone, like LG and Ring, make an intentional choice of what the right metric looks like for the success of the business and its users. That said, I keep hearing about measuring Engagement for apps whose sole purpose is to get the users to complete a task and get in and out quickly. Do we keep measuring Engagement because it’s the easiest metric to pull? Or is it a means to an end, not the end in itself, like a renewal, upsell, or subscription?
Here’s a useful way to think about it: if a metric can’t map to top-line revenue or bottom-line savings, it can’t answer the ‘so what’ or ‘then what.’ It’s a vanity metric. Duolingo’s ‘so what’ is retention, and its ‘then what’ is a subscription.
Then some metrics can map to revenue or savings and still miss the outcome they are expected to achieve. Take a goal to reduce billing-related calls by 20%, as an example. This objective is directly tied to lower cost-to-serve. We could hit the number by solving the actual customer problem, making the bill easy to understand, or having the same bill every month so there are no variations or surprises. We could also make the phone number harder to find. So we could achieve the objective of reducing calls but not meet the outcome of changing the user behavior at all. That’s the gap the test has to catch.
What’s the ‘so what’ and the ‘then what’ of the metric you’re tracking right now?