How To Track DORA Metrics in an Internal Developer Portal
DORA metrics are a key barometer for engineering performance and software stability. But collecting and interpreting DORA metrics isn’t always straightforward: Data needs to be pulled from various sources, such as git providers, CI/CD pipelines, incident management tools and more. This can be a complex task and is different for each organization.
As an internal developer portal already integrates with an engineering team’s underlying platform and centralizes everything that an engineering organization needs to do and know, it is the ideal platform to provide leaders with all the metrics they need to manage their teams and operations efficiently.
In Google’s recent DORA report, the company emphasized that internal developer platforms required user-centeredness, developer independence and a product-oriented approach to get the most out of platform engineering. This is where the portal takes away the complexities of the infrastructure and offers direct interaction through a unified interface. This ensures an improved developer experience.
Let’s explore how an internal developer portal can be used to track DORA metrics within your organization.
Before You Start
The prerequisite for this guide is to have completed the onboarding process for the internal developer portal, as well as linking the repository of your choice (in this case, we’ll be looking at either GitHub, GitLab or Azure Repos — but you can bring your own data to the portal too).
How To Track Your Team’s Deployments
To track the specific data required for deployment frequency, change failure rate and lead time for changes, you need to create a deployment blueprint. Blueprints are the foundational building blocks of the internal developer portal, holding the schema of the entities that you would want to represent in a centralized repository. They are completely customizable and can support any number of properties the user chooses. In this case, their creation will ensure that the essential information for each deployment is captured.
Step 1: Set up your data model
The first step is to create a new blueprint, which is as easy as clicking on the “+ Blueprint” button on the builder page. Then, name it “deployment” and add the schema below:
For those of you who do not have the lead time configured, you can follow the integration guide for your git provider to add this property.
Step 2: Pick a deployment tracking strategy
You’ll then have to pick your way of tracking deployments. Here are the options:
- Pull Request/Merge Request Merge
The recommended approach for tracking deployments and calculating lead time is by monitoring when pull requests (PRs) or merge requests (MRs) are merged into a branch. The lead time for these merges is calculated as the difference between when the PR/MR was created and when it was merged.
When a PR/MR is merged, a deployment entity is created within the portal to represent the deployment that took place. The lead time for that PR is calculated and added to the blueprint’s makeup.
- Workflow/Job
You can monitor workflow runs in your pipeline to track deployments. This captures all workflow runs from the main branch that match a certain name or label and maps them to deployment entities. The deployment status is set dynamically depending on whether the concluded workflow run was successful.
For example, when a GitHub Actions workflow, GitLab CI/CD pipeline, Jenkins job or CircleCI job runs on the main branch with the name “build,” a deployment entity is created in the portal to represent the deployment.
- CI/CD pipelines
CI/CD pipelines provide a reliable method to track deployments. With Jenkins, for instance, you can create and update entities in your portal dynamically using the portal’s API as part of the pipeline execution. Other CI/CD pipelines such as Octopus Deploy, Circle CI, Azure Pipelines, Codefresh and GitLab can also be used.
- Releases/Tags
You can track GitHub deployments by linking repository releases and tags directly to deployment entities in Port. These repositories carry essential details like service versions, commits and release information, making it easier to keep tabs on your deployments.
To implement this, you can add the configuration below to the data sources page in your portal, and select your GitHub integration.
- Custom API
For those of you whose tool or workflow is not natively supported in the portal, you can create custom integrations by directly interacting with the portal’s API. This enables you to track deployments from any system that can initiate HTTP API calls.
Here’s an example of how you can use the API to create a deployment entity in Port:
You can also track services or components within a monorepo through custom integrations (API) and advanced mapping.
Incident Management
Incidents play a key part in tracking DORA metrics such as change failure rate (CFR) and mean time to recovery (MTTR). By improving the way you track incidents, you’ll have a better handle on how frequently deployments fail and how quickly teams can resolve issues.
This means having the capability to use incidents to calculate CFR and MTTR, link incidents to specific services to gauge the impact of failures, and aggregate metrics across incidents for more effective monitoring.
Here’s how you can do this:
Step 1: Setting Up Your Data Model
First, you have to make sure your PagerDuty incident blueprint is properly configured to map incidents to the correct services. This includes defining the appropriate properties and relations for incidents. You can do this by following the guidelines on this page.
To add properties to capture incident resolution time and recovery time, use the instructions below:
Then, you can add the below mapping config to the PagerDuty incident data source:
Ingest Incidents From Tools
To sync incidents from PagerDuty, simply follow the steps outlined in this PagerDuty guide, which provides a detailed walkthrough for setting up integrations to track deployment-related incidents.
For other incident management tools, refer to these specific guides:
Relating Incidents to Services
You can link incidents to your service, whether it is in a GitHub, GitLab or Azure DevOps repository. Here’s an example of the code you would need for GitHub:
You then update the mapping config to your PagerDuty incident data source.
Aggregating DORA Metrics
The next step is to aggregate the DORA metrics at the service level, enabling teams to track deployment frequency, change lead time, change failure rate (CFR) and mean time to recovery (MTTR) for each service. For broader insights, these metrics can be aggregated at higher levels — such as team or domain — by ensuring relevant blueprints are in place and relationships to the service blueprint are defined. Metrics are aggregated monthly by default, but this timeframe can be adjusted to weekly, daily or even hourly to meet your needs.
To set up aggregation and calculation properties in your service blueprint, start by navigating to the Builder in your Port portal. Find and select the service blueprint, then click the options menu (the “…”) in the top-right corner and choose “Edit JSON.” Add the relevant properties under the aggregationProperties or calculationProperties sections within the JSON schema. Once added, save your changes to activate the new aggregation configuration.
The aggregation properties should look like the following:
Deployment Frequency
Change Lead Time
Change Failure Rate
Add the following to the aggregated property in service:
Add this calculation property to calculate the CFR from the aggregated properties:
MTTR
Visualization
The most powerful way to use these metrics is to present them on custom dashboards and widgets within your portal.
To set up a dashboard:
- Go to your software catalog.
- Click on the “+ New” button in the left sidebar.
- Select “New dashboard.”
- Name the dashboard (such as DORA metrics), choose an icon if desired and click “Create.”
Then you can add the widget of your choice:
- Click on “+ Widget” and select Number Chart.
- Fill in the respective title.
- Add an icon that makes the most sense for your team.
- Select
Display single propertyand choose Service as the Blueprint. - Select an entity and choose MTTR as the property.
Acting on the Metrics
Once you’ve put your metrics in place and you’ve set your benchmarks, the next step is to put this information to practical use. You can:
- Use scorecards to evaluate the team’s performance against your chosen benchmarks.
- Set up alerts for managers when their teams fall below thresholds.
- Initiate changes to improve your DORA metrics. For example, if code reviews are taking too long, implement an automation that nudges reviewers after a specific period. Or if your MTTR is higher than you’d like, centralize all your incident data in one place, or automate your incident response to streamline resolution efforts and reduce recovery times.
Adopting DORA metrics is about giving your team a clearer view of how they’re doing and where they can improve. By using a developer portal to track these metrics, you’re not only centralizing data from various tools but also making it accessible and actionable for everyone. With these insights at your fingertips, it becomes easier to spot trends, set realistic goals and identify areas for improvement.
Check out Port’s live demo to see what you can do with DORA metrics yourself, and visit the docs guide for DORA metrics on this page.
