David Bland, Principal Advisor at Neo, stopped by today to chat about Pirate Metrics and the importance of testing our assumptions.
I found it very interesting to hear his thoughts on how quantitative metrics like the Pirate Metrics (Acquisition, Activation, Retention, Referral and Revenue) goes together with qualitative metrics like a user being aware, hopeful, satisfied and passionate.
Pirate Metrics are often referred to as a funnel, but another way to think of it is a system where your users move around from being aware of your product, hopeful that you have as solution to their problem, satisfied about your solution to the problem and then passionate enough about your product that they refer their friends.
Quantitate metrics are helpful to monitor and segment where your users are in this system, but you need qualitative research to validate assumptions about the reasons for their movement pattern. For instance it's a common mistake to think that satisfaction is equal to retention, or that activation is a result of awareness. David gave an example of a situation he'd seen where users had activated a feature even without knowing they had, and retention was zero because of that.
I can remember a similar awareness example from Jetpac. In the app, at the end of each city guide you could take a fun photo quiz. From looking at the data it seemed like nobody were interested in taking quizzes. Then later, from usability testing I realized why. The call to action button wasn't styled the same way as the other blue buttons, this one had blue text on a white square (on white background..), so I found that people either though 1) it was a 3rd party mobile advertisement 2) didn't register it at all.
So it turned out our users were not aware that taking a city guide quiz was a feature. We would never have figured out the real reason from just looking at the data.
Another interesting point David brought up was asking yourself "What is the experience that you would want your users to come back to do?" when you're trying to optimize for retention. If this is clear it's easier for you to design a tight loop with emails or push notifications or maybe you will realize that the thing you want people to come back to do is not related to the need that attracted them to the product in the first place and you will have no success trying to engage them that way.
Check out David's blog or find him on Twitter.
Also, David's colleague at Neo, Ash Maurya, is coming out with a new book soon about crafting effective experiments that lead to breakthrough insights called ”The Customer Factory”.
A book that is already out is called "Running Lean: Iterate from Plan A to a Plan That Works".