Getting Buy-In for DORA Metrics

EverQuote prides itself on having a data-driven culture. But even with this organization-wide commitment, its engineering team initially struggled to embrace metrics that would help them understand and improve performance. Many leaders would give up after a failed attempt, but Virginia Toombs, VP of Engineering Operations, took a step back to understand what went wrong so they could try again — and succeed.

Along the way, the EverQuote team learned what to avoid when implementing engineering metrics and how to successfully roll them out. For them, it was all about empowering team members, collecting actionable data in the right platform, and correlating metrics for a more holistic view of what was happening across the organization.

Lessons Learned About Measuring Engineering Productivity

When EverQuote decided to measure engineering productivity a few years ago, it started by purchasing a tool, like many organizations commonly do. But it encountered problems as it hadn’t considered what to measure or how to leverage those insights to improve performance. Because EverQuote didn’t know which engineering metrics best suited its unique team and processes, it ended up with a tool that didn’t have mature throughput or flow metrics — two things it would learn were core to its success. The result? Virginia and her team saw detailed engineering metrics but lacked a comprehensive view of the organization’s performance.

This issue caused a domino effect. Measuring only granular metrics made team members feel that individual performance was being judged, rather than the process itself, and enthusiasm for the program came to a halt. That’s when the engineering operations team decided to rethink the approach and start from scratch.

Using Metrics to Improve Developer Experience

EverQuote’s engineering operations team is a central function within engineering whose main goal is to create an environment where engineers can thrive. This team optimizes processes, encourages collaboration, and coaches on agile techniques. For them, it’s essential to understand how engineering processes are performing so they can make data-driven decisions to improve. This team made two important decisions when rolling out engineering metrics for the second time.

First, they took the time to understand which engineering metrics applied to their organization. Rather than starting with granular metrics, they decided to lead with the big picture, adopting the four original DORA metrics: Deployment Frequency (DF), Mean Lead Time for Changes (MLTC), Mean Time to Recover (MTTR), and Change Failure Rate (CFR). From these high-level metrics, they would still be able to identify bottlenecks or issues and drill down into more granular metrics as needed.

To support DORA, and to provide visibility into its corresponding metrics, EverQuote adopted Code Climate Velocity. With the Velocity Software Engineering Intelligence platform, they could identify organizational trends, look at data by teams or applications, and dig into specific DORA metrics. For example, if they see that MLTC is high, they can click into it to see exactly where the holdup is — maybe a long Time to Open or Time to First Review is preventing the PRs from getting to production as expected. Starting at a high level helps them understand their systems holistically, and then they can drill down as needed, which is more efficient and saves team members from metric fatigue.

Second, they empowered teams to own their metrics by educating them in how to read and interpret the data, and creating processes to discuss performance at the end of a sprint. They held these conversations as a team, not during one-on-ones, and focused on how they could better collaborate to improve as a unit. This strategy exemplifies one of EverQuote’s core principles: If you work as a team, you succeed as a team.

Successfully Implementing DORA DevOps Metrics

The EverQuote journey to measurement has come full circle. Now, engineers embrace engineering metrics as a tool for continuous improvement. After two iterations of implementing metrics, the team has learned three major lessons for successful adoption:

• Collect data you plan to act on. Although measuring and tracking every possible engineering metric may be tempting, it can prevent you from seeing the forest for the trees. Instead, be intentional about the metrics that your organization can derive insights from to take action.

• Correlate metrics and drill down as needed. Measuring DORA metrics gives EverQuote a full view of how engineering systems work at any given time. Being able to double-click into them in Velocity lets them quickly identify and resolve the root cause of an issue when it arises.

• Use consistent data. EverQuote has 17 engineering teams spread across multiple functions and locations. To maintain consistency, they align on how metrics will be defined and calculated. This process is essential to ensure they speak the same language and can benchmark against other teams and the industry.

Combining DORA DevOps metrics with other engineering metrics in Code Climate Velocity has helped EverQuote nurture its data-driven culture. To learn more about successfully rolling out engineering metrics within your organization, chat with a Velocity specialist.