How to produce useful team reports

As someone who started their career in Business Intelligence, reporting is life.
Dashboards and visualizations are key to understanding how your team is doing.

So if we should use story points? What KPIs should we track?
As we discussed on Monday, there are 3 KPIs that will give you the overview you need and this is how we should visualize them.

Scatter Plot

circles are features, triangles are bugs

By using a scatter plot with the cycle or lead times, we can see:

  • How long a feature will take on average to be delivered, and what is the confidence interval you can use.
  • Team performance/ticket complexity over time.
  • Clusters of tickets, by ticket type or priority.
  • How long does it take for your bugs to be repaired?

Bugs Over Time

Nothing is simpler than a single number, but it also can provide a lot of information.

The arrow represents a bug that moved to the next iteration and how many.
  • How many bugs are you getting?
  • Are they bleeding over to the next iteration?
  • Do you have more or less bugs than less iteration?

Add this information to the scatter plot and you have a really good overview of the stability of your system. Also how easy bugs are to fix in your system

Simple reports provide a lot of information without being overwhelming.You can quickly inspect and adapt your approach to provide the best value you can to your stakeholders.



Who should use story points

One of the biggest pitfalls of any agile transformation is to make story points the centerpiece for long term planning. Even worse, when companies try to standardize velocity across teams for the sake of long term planning.

The biggest advantage of being agile is throwing away long term planning and estimation. We understand that the world is ever-changing and we cannot predict the future.

Welcome changing requirements, even late in development.
Agile processes harness change for the customer’s competitive advantage.

So who should use story points and why are they useful?

Story points are only useful for the team itself.
The purpose of story points is to help the team understand what was delivered during this sprint and plan the future one. We call it predicting by using yesterday’s weather.

Using story points in any other way is to invite failure. It’s going back to the old way of doing things. Ways that have been proven time and time again that won’t work are a risk and invite failure.



Why are story points about time

Unlike this cat, it is impossible to escape the wheels of time…

It doesn’t matter if you use T-shirt size or Story Points or something new, in the end, a year has 365 days, a week is 7 days and a day 24 hours.

The world runs on time. It is a currency we cannot exchange. A commodity we cannot buy more of.

So now that the grim part is out of the way, let me explain why I view SP’s like time. Let us start with the…


If you are using SCRUM, and you probably are, you know the concept of the SPRINT. There is also a high chance you have been using story points.

Sprints, if you don’t know already, are time-boxed windows of time. It starts with planning and ends with a review. Whenever one Sprint ends, a new one starts.
Depending on the needs of your stakeholders and how short your feedback loop, your sprint could go from 1 week to 4.

You can look at the sprint as a division of time on your calendar.
During the sprint, an increment of working software will be worked on and delivered in the end.
There are also other important features to the sprint, like Goals and Events, but for the sake of this argument let’s keep them away.

One way we estimate the amount of work we can pull into the sprint and deliver the increment is by using…

Story Points

Story points are the most used agile estimation metric.
One of the reasons why they are so famous is because they can be summed up (to extrapolate time). It is much harder to do if we are T-shirt sizing (S, M, L, XL).

It’s one of the hardest things to master and something gets in the way of what is important.

I’ve seen many teams get lost in the discussion of “the difference between a 2 and a 3”. You may disagree with me but I find this discussion very unproductive.
One thing I always do with my teams is to explain why it is unproductive and we should drop the 2’s and just focus on 1’s and 3’s.

The distance between 1,2,3 is only 1.
If that equates to around 1 or 2 days, who cares? We are doing a relative estimation, not a precise one.
By removing it, if its smaller than a 3 its a 1, if its bigger than a 3 its a 5. Much simpler.

So why do we estimate?

We do it to get a perception of the amount of work.
We do it to try and understand if the cost is bigger or smaller than the value that we expect to produce.

Over the years I have seen many explanations about what makes a good estimation. We need to include:

  • The amount of work to be done.
  • The complexity of the work.
  • The risk/uncertainty of what needs to happen.

Both the amount of work and complexity can be correlated to time.
Look at it like this:

  • Amount of work: Taking 100 coffees will take more time than taking only 1 coffee.
  • The complexity of the work: Taking 1 coffee will take less time than building a coffee machine.

Risk/Uncertainty is difficult to measure, so let us leave it for now and I will come back to it later.

It is Time

Now the interesting part.
If Story Points are time and, they are a relative measure of time, how does it work?

This scale is relative to a 2-week sprint. Your mileage may vary.
In working days :
1 = between 0 and 3
3 = between 2 and 5
5 = between 4 and 8
8 = between 6 and 10
i.e I expect this to take me between 2 to 3 days to deliver. Then this story is a “3”.

So where does the risk/uncertainty come into play?
You may have noticed that there is an overlap between the SP’s. This is where uncertainty and risk come into play.

  • How sure are you that you can complete this story in 2 days?
  • How risky is this story to do?
  • How familiar are you with what we need to do?

If either the uncertainty or risk is high, maybe you should go for a 5 instead of a 3.
Remember that the goal is to have a relative estimation, not an exact one.

The conclusion

Story points are about time. A range of Time.
We need to take into consideration both the risk and uncertainty of the feature.

During these years I have come to believe that there are better ways to measure team performance and do an estimation.
Better ways to provide more value to the teams and their discussions.
But… Let’s be honest. In the end, estimation or no estimation, the only thing that matters is : Are you delivering value or not?

This agile principle says it all: Working software is the only measure of progress.



Why do we even use KPIs?

Do we just track KPIs in an attempt to be like Big Brother from the 1984 book of George Orwell?

No. KPIs are extremely important, if you choose the right ones.

If we don’t measure what we do, it is like driving with our eyes closed. There is a small chance you are going to arrive at your destination safe and sound, but most likely you are not.

Transparency is extremely important. Whatever you measure, your team must be aware. No hidden performance KPIs. No hidden tracking for bonus purposes or career progression. Everything must be out in the open and explained exactly WHY we are measuring it.

On one of my teams we are now working on simplifying our workflow and we will put some KPIs along the way to measure our progress, find points of improvement and points of pain. This is an exercise that must be done with everyone and everyone should be able to contribute.



What KPIs should you be tracking?

Whenever we start a new project, it is extremely important to track the progress.
Sure, you can just go with the plan and hope for the best, but we all know that doesn’t work.
It doesn’t matter which methodology or framework you are using, if you don’t track your progress you are going to fail.

So what should we track?

  • Hours?
  • Man/Days?
  • FTE?
  • Lines of Code?
  • Story Points?

In my years working in IT and helping teams find their best performance, I’ve found that there are 3 key metrics that matter the most.


We use cycle time to calculate the amount of time that takes, since a request is made (by stakeholders, customer or product owner) till it is delivered. This includes all the waiting time in the backlog.

This metric helps us understand, on average, when you can expect a feature to be built and delivered.



This one calculates the amount of time since your team picks up the item, until it gets delivered.

Lead time will give you as a product owner/product manager an overview of how long, again on average, your team requires to deliver the feature.



Also can be called the CYCLE TIME for bugs.
This represents the amount of time since the bug was found until it was resolved.

Repair time allows us to understand the complexity of both your bugs and your system.