Mobile apps are hot. As of July 2015, the iTunes and Google Play stores combined for a total of 3.1 million apps available for download. Even if the vast majority of those apps are simply duplicates of one app for each platform, that’s still enough apps to make up the country’s fifth largest city (if apps were people). But, even with so many products available, today’s users are actively engaging with between 20 and 30 apps a week.

For product owners, this means two things: you have an insanely large pool of competition and an extremely small window of opportunity to win and retain users. Sooner or later, this truth will cause you wonder why more people aren’t using your platform and brainstorming new features that could turn the tide in your favor. Before you start grooming your backlog, engage in a very brief self-assessment by answering the following question:

When features are added to the app, are they added because they seem like the cool thing to do or because they’re useful?

From experience, many brands tend to develop apps and judge their success within the same framework they apply to their web project. While there are many valid reasons for building apps this way, one major factor that can hinder the long-term quality of a product is the approach to analytics.

The challenge with web-first metrics

No matter what the app is or is supposed to do for users, the primary metrics that are most commonly referred to when talking about the success of products are page views and unique visitors. These numbers are very often supplemented by total downloads and more micro-versions of usership, such as daily unique visitors or app launches. Generally speaking, these four or five metrics, when used together to calculate additional “engagement” figures, can provide product owners with a snapshot of their app’s user base.

And that’s the problem.

Brands who rely principally on these metrics to determine future user engagement strategies will ultimately struggle to exceed their stretch goals. That’s because this approach prioritizes company-centric numbers over real user activity. Apps do not exist in a vacuum, and therefore, product owners cannot divorce measurement from the context of the user. To gain true insight into an app’s performance, brands have to ask the hard questions and expect that the app will fall short. Luckily, failing to meet more stringent, user-first measurement criteria means product owners get the opportunity to work with users to build real value into their platforms and decrease the likelihood that their apps will get lost in the increasingly populated app universe.

Apps do not exist in a vacuum, and therefore, product owners cannot divorce measurement from the context of the user.

Four Better Questions

In order to measure the right variables, product owners need to start with the right questions. Like a well executed experiment, beginning with the right questions means product owners can devise measurement tactics to more accurately prove or disprove their hypotheses. Here are four questions that prioritize the user’s experience.

Why are people using this app?

As a product owner, you should have recorded documentation of the app’s reason for existence. This product strategy guide will be meaningful for kicking off development, but may eventually be out of phase with what the user expects of the app once it’s available. This is especially true of brands that created the product strategy internally, with little input from users throughout design and development. Even if the strategy was developed in response to a real audience-based problem, product owners must commit to regularly answering this question beyond development. Users and their context will change, and the product may not be as relevant or worth the user’s time as it once was.

Users and their context will change, and the product may not be as relevant or worth the user’s time as it once was.

Where are users failing to convert?

When an app has users, chances are they’re using the app to complete a few specific tasks. It’s also true that there are users who are failing to complete those tasks — at least as much or as often as intended. Not only do you need to find those users who are struggling to convert, but you need to figure out why they’re struggling. Using analytics, product owners can create a “lookalike” segment—a group that’s using the app in the same way as those who are failing to convert, but is actually converting. Using this segment, you can begin to analyze the success factors that lead to conversion and tailor the experience to better expose those elements to users. Another tactic is to look at “bouncebacks”—users who download your app, leave almost immediately, but return some time later to become loyal users.

What are the most common paths taken?

A lot can be learned about the success of user flows just by looking at path metrics in your analytics suite. One of the tactics that can be powerful in helping you determine an optimization strategy is to look at paths that ultimately terminate in an exit. If you realize certain paths cause users to exit prematurely, you have a great foundation to begin asking why users are leaving and propose very focused in-person user testing to determine the root cause and possible solution. Additionally, you may uncover red routes taken by users that are different than what the app was designed around in the first place.

What do users actually think about the app?

Finally, there’s no better way to understand how target users engage with the product than receiving feedback directly from their mouths. While online reviews and direct feedback channels can surface real opinions, remember that more than 90% of customers are silent about the issues they have, and would prefer to just leave, rather than speak up. Product owners who develop a sustainable way to communicate with their users more regularly might be surprised at how much insight is gained through real conversations. Remember, users have chosen to use this app from the millions available and the very few they truly have time for. Don’t be afraid to build a direct relationship through feedback channels.

Product owners who develop a sustainable way to communicate with their users more regularly might be surprised at how much insight is gained through real conversations.

Putting the Questions into Practice

To show how changing your analytics approach can provide better strategic value for product owners and their teams, let’s walk through a hypothetical example.

You and your team have built a universal mobile application, running on both Android and iOS phones and tablets. After a year’s worth of time in the wild, you begin to pull numbers on adoption and usage and discover an interesting difference between platform users. For some reason, Android users are spending close to half as much time per session using the app compared with your iOS users.

Using a web-first analytical mind, you immediately conclude that something must be wrong with the Android app. Session length is a prized engagement metric — the longer a user spends with the platform, the more deeply they’re engaging. In fact, the automated analytics reporting platform you use may even generate messaging to make you feel that your iOS users are objectively more valuable than your Android users.

But, before building a reactive strategy around this conclusion, think through the differences in how your apps are presented to different audiences. Ask, “What are the most common paths taken by users?” To figure it out, you begin to dig deeper into the metrics, and it appears that users on all platforms are taking the same basic paths to complete the same basic actions. In fact, as a percentage of total sessions, each group is doing the same things the same amount of times.

As you continue to dig into one primary red route, an interesting pattern emerges. Android users are completing the action quite a bit faster than iOS users. You compare these results across other paths and discover similar results. Overall, Android users seem more efficient at accomplishing the app’s primary actions.

You’re still not out of the woods, even after discovering this unique insight. At this point, you need to decide what to do next. You could propose conducting sophisticated user research that would allow platform users to walk through why it takes so little time to finish an action. Although this could be effective and lead to the discovery of new insights, it can also be costly in time and resources, especially if you haven’t eliminated all the variables in play. One of your team’s Android developers has an idea: Material Design. Halfway through the year, the team redesigned the app to be on parity with Google’s Material Design standards. You updated multiple app components, one being the app’s primary action button. You uncover that Android users are much more likely to take action quicker once the app loads, rather than exploring the interface and thinking about what to do first. And, by comparison, your iOS app has not been built with the same design considerations in mind.

With this hypothesis, your team is able to make some subtle design shifts to your iOS app, prototype and test a few different user flows, and ultimately make both apps equally efficient, and (according to the latest reviews) all users bases are more satisfied with the result. This is just one hypothetical example of how asking harder questions of analytics can change the strategic direction of an app. It’s also important for product owners to regularly take stock of the underlying subjective assumptions they bring to conversations about app analytics. As in all research, removing bias from the conversation can be a powerful force in helping to uncover useful insights about an audience.

A Final Challenge

As a product owner, it’s critical that you figure out how to communicate with users about how the product is performing and the ways it impacts brand perception outside the company’s walls. Analytics provide a powerful tactic to begin the process, but doing analytics right requires a creative approach centered in asking the right questions.