Wrong Analytics
Customer: What do you mean you have used all the budgeted developer hours? There's so much stuff missing from the product! How much of the original spec is here? 50%? 25%?
Developer: I hear your pain. Many of the original features had to go because of budget constraints, and yeah, some of these features that did make the cut could use some more work. But you are not taking into account our new analytics features that were not in the original spec. We built those features so that we can get real data from real users. We shouldn't measure our progress in lines of code written but instead look at the impact we create. And how can we judge this impact if we have nothing to measure it with?
Customer: But this... this is just embarrassing. I can't believe you shipped this to our users. What am I going to say to my managers and my team?
Developer: Don't worry. We kept the core features. With this product we were still able to test out your idea with real users. And thanks to those analytics we added, we got actionable insights into what's working and what's not!
Customer: Yeah? I guess that's better than nothing. So what's the main insight?
Developer: People hate it.
Customer: Hate? Who hate it?
Developer: The users, they don't like using our product. Do you see this number here? It takes roughly ten seconds for the average user to realize this product isn't working for them and after that they just leave and never come back. It's actually pretty great that we get to learn these things with our analytics reports.
Customer: How on earth is this great? First you tell me that I have to go to my colleagues with this unfinished piece of junk. And now you are saying that there are also real numbers for pointing out the epicness of our failure? What is wrong with you! I'm going to be the laughing stock of the whole department.
Developer: I'm sorry, I got distracted by this analytics dashboard. Did you say you wanted to show these numbers to your team? It's really easy to export the data from here. Would a CSV file work for you?
Customer: Get the hell out of my sight!
***
What's wrong with this scene?
Let's start with the obvious problem: our developer has been building stuff without keeping the customer up-to-date. The fact that the status of the budget and the prioritization of different tasks comes as a surprise for the other party is clear evidence of miscommunication between the developer and the customer.
But what I'm trying to describe here is a situation where people might be better off by not allocating any serious resources to analytics than to get real data on how their product is being used.
How is this possible? First, when a team is delivering more features instead of doing analytics work, other people in the organization might look at the team and applaud their productivity ("this team is delivering new features so fast!"). Feature development is easier to measure and more visible than business impact and because of that features can get more positive attention. Features are where the focus of our customers, managers, and peers naturally go.
Second, when there are no analytics there are also no negative reports. It is true that when you can show your product is a hit with analytics, you get more leverage with others. However, it's also true that disappointing results take some of that same leverage away. Since time and time again we overestimate the upside of our projects, it is more likely that the end results of our projects fail to live up to expectations. It's more likely that you get to report bad news instead of good news. So why would you want to report any news at all?
How could you prevent that scene from happening at your work place?