QI for DevOps Metrics Tree

Quality Intelligence for DevOps metrics tree

You don’t need to measure everything, though. A lot can be achieved by just keeping the release quality in control. The essential logic of the DevOps Value Creation Model has been codified into the DevOps metric template that creates this kind of DevOps metrics tree skeleton with one click on the Metrics page:

devops metrics tree skeleton

DevOps Metrics Tree displays all your DevOps metrics in a hierarchical structure. Each metric can contain sub-metrics and results (indices) of sub-metrics are aggregated to the top-level parent metric.

Quality in Use is measured by the test pass rate of “shift-right” tests run in Production. QI for DevOps approach is based on Flow Framework and Flow of Value is primarily measured by Flow Velocity and Production Deployment. Pull Requests and Commits act as leading indicators of how software gets done and value delivered.

Release Quality is the most critical indicator for release readiness. In addition to QA / staging test pass rate you should track critical bugs at least.

Happiness of the team is a ‘soft’ indicator that affects the team’s outcomes big time. That’s why it has been added to the DevOps dashboard.

When you create this DevOps metrics skeleton from the metrics template, Qentinel Pace creates the required data sources for the metrics, too. Personal and Pro plans support only manual and test results data sources of Qentinel Pace itself. If you have a Team or Team 24x7 subscription, you can also set up supported Pull Services to pull data on work items, commits, pull requests or deployments from supported DevOps tools such as Azure DevOps or Jira.

Using DevOps metrics tree

Each metric has the following information in addition to the metric name:

  • Value (v): the latest actual, measured value of a metric. The values can represent a wide range of different units ranging from milliseconds to the amount of bugs etc., depending on what is measured.

  • Index (i): the normalized index value in the range 0 – 100 (target) – 200 (max)

  • Status: The "traffic light" status color represents the current status of a metric compared to its target. This helps you to understand if the results are in an acceptable range, even if you are not familiar with possible deviations in a metric. Status indicators can have the following colors:

    • Grey (metric does not have a configured data source / we can’t get a value, or the index value is zero)

    • Red (the latest index value is below 75)

    • Yellow (the latest index value is between 75-99)

    • Green (the latest index value is 100 or above)

Index values allow you to compare and aggregate metrics which have completely different units. Normally it’s hard to compare or aggregate for example 54% test pass rate and 3 new critical bugs. Which one is more important to be reacted on immediately? Which one is an outlier to what we normally see? To make the meaning of an actual value easy to understand and to make it possible to aggregate different types of (sub-)metrics, we give each metric a target and bottom limit. Then we turn the actual value to index value based on how it relates to the target and bottom value.

Normalized index values

The normalized index values are calculated based on the two limits configured for each metric: target and bottom.

Target value defines the goal value we aim at. Bottom (optional) defines the point where the index value will be 0. Together these two limits define the range for the index. We can calculate an index score (0 – 100 – 200) where 100 represents the intended target level. If our actual value is above the target , our index value will be more than 100. If our actual value is exactly the same as the target value, the index will be 100. If our actual value is at our bottom value or less, then the index value will be 0. Anything between the bottom and the target value will provide an index value between 1-99.

Let’s have an example. Our target is that we will have only 3 major bugs open at any point in time. We set the bottom value as 10 open major defects. Let’s consider a few examples:

  1. If we have 3 open major defects today, our index value would be 100

  2. If we have 12 open major defects today, our index value would be 0 (actual value is less than the bottom value)

  3. If we have 10 open major defects today, our index value would be 0 (the actual value is the bottom value)

  4. If we have 0 open major defects today, our index value would be 200 (actual value is better than our target / maximum achievable)

  5. If we have 5 open major defects today, our index value would be 71 (actual value is between our target and bottom values)