Earlier this week I was discussing with my friend, Tom Graber, about the problem with metrics and I thought I would jot down a summary of the conversation. It all started with a discussion about social game companies focusing on metrics to determine whether or not they were successful with their game.
I was asked in an interview how I would know something was successful. While I can say that with some of my projects I know I have been successful because we increased engagement. Say for example, CBSNews.com with the redesign we had an increase of 40% on traffic across the site and 60% with video consumption. Sounds successful right? But how successful was it really? Did it increase the bottom line? Was this an ongoing increase or did it drop off over time? Take for example the story, “The Brake Gets Pulled”, which was in Louis Rosenfeld’s book, “Search Analytics for your Site“, the author writes of how after a site redesign the traffic had increased but actually upon deeper research of why they had discovered that the site search experience they had redesigned had actually become worse! Thus you can’t always be assured based on numbers that you were successful.
What Tom and I agreed to is that most people confuse engagement and KPI. What the game companies really should be exploring is the engagement alongside the KPI. Your metrics will only tell you one story. Interviewing your users and finding out why and watching them play the game will tell quite another story.