Ways To Overcome DevOps Automation Challenges

Key takeaways

You’ve implemented new processes, adopted new tools, and shaped a culture that emphasizes cross-functional collaboration.
An on-call policy tailored to a team at New Relic may not apply to your business.
New Relic gives you the necessary visibility to monitor your DevOps efforts and continuously improve your results at every step.

Your team has embraced DevOps. You’ve implemented new processes, adopted new tools, and shaped a culture that emphasizes cross-functional collaboration. But you haven’t reached your maximum speed yet. As a result, you’re missing something that prevents your organization from becoming an actual, high-performing DevOps machine.

That missing piece is often data measurement. Although measurement is the most prominent pillar of the CALMS concept (culture, automation, lean, measure, share) created by DevOps expert Jez Humble, it is often overlooked by DevOps teams in their quest for incredible great speed and more autonomy. However, this can create significant issues, as data accuracy is critical to a well-functioning DevOps team, whether it’s an incident response or understanding the complexity of microservices.

However, it is also designed for those standing still without taking advantage of DevOps to fully succeed in their digital transformation.

By sharing real experiences, especially lessons learned from New Relic, we want to help you overcome the final hurdles to your DevOps success. To better understand how to set reliability goals and unravel your microservices approach’s unique threads of communication and development requirements, we’ve assembled proven best practices that show you how to get things done faster and more efficiently.

Chapter 1: Balancing SLOs with Rapid Application Delivery #

Your development cycles are faster, and you deploy code more often, but how reliably? Quality and reliability are two equally essential outcomes of a successful DevOps approach.

This is where the ERS in Site reliability engineering (SRE) is a cross-functional role, taking on traditionally specialized responsibilities independent of development, operations, and other IT groups. As the SRE relies on collaboration between development and operations, it goes hand in hand with the DevOps culture. DevOps and SRE have many standard features, but SRE is mainly dedicated to continuous improvement and managing measurable results, particularly using SLOs (Service Level Objectives).

Let’s start with some important definitions:

“Basically, [SRE] is what happens when you ask a software engineer to design a business function.”

—Ben Treynor Loss, VP Engineering, Google #

Define appropriate SLIs and SLOs #

Although SRE industry best practices dictate SLIs and SLOs for every service you provide, it can be complicated to define and deploy them if you have never done so. Here are seven steps we use at New Relic to determine SLOs and SLIs:

1. Identify system boundaries: The boundary of a system is defined by one or more components presenting one or more features to external customers. Internally, your platform may have multiple moving elements (service nodes, database, load balancer, etc.). Still, these individual elements are not considered system boundaries because they do not directly present functionality to clients. Instead, think of multiple features working together to demonstrate functionality. For example, a login service that offers an API capable of authenticating user login credentials is a logical group of components that work collectively as a system. Therefore, before defining your SLIs, start by grouping the elements of your platform into systems and characterizing their boundaries. Focus on this aspect in the following steps because SLIs and boundary SLOs are very useful.

2. Define the functionality presented by each system: Now, group the platform components into logical units (UI/API tier, connection service, data storage/query tier, legacy data tier, data ingestion, and routing). At New Relic, the limits of our system are aligned with the limits of our engineering team. Using these groupings, articulate the set of features exposed by each boundary in the system.

3. Create a clear definition of what is “available” for each feature: For example, “send messages to the right destination” is one way to describe the availability expectations of a data routing feature. Use precise language to explain availability rather than technical terms that not everyone knows.

4. Define the corresponding technical SLIs: It is now time to define one or more SLIs per feature using your definition of the availability of each segment. From our example above, a data routing feature SLI could be “time taken to deliver a message to the correct destination.”

5. Measure for a benchmark: Monitoring will let you know whether or not you are meeting your availability goals. Using your monitoring tool, collect baseline data for each SLI before setting your SLOs.

6. Apply SLO targets (by SLI/functionality): Once you have collected the data, before defining your SLOs, ask your customers questions that will help you identify their expectations and adapt your SLOs to meet them. Then choose SLO targets based on your credentials, customer opinion, what your team can commit to supporting, and what is achievable given your current technical means. Following our example of SLI for data routing, the SLO could be “99.5% of messages transmitted in less than 5 seconds”.

7. Iterate and adjust: Don’t use a set-your-own approach to your SLOs and SLIs. Instead, it would help if you considered that they will (and should) evolve as your services and customer needs evolve.

Additional Guidance for SLOs and SLIs #

Ensure that each logical instance of a system has its SLO: for example, for shared systems (as opposed to horizontal systems), measure SLIs and SLOs separately for each shard.

Distinguish between SLIs and alerts: the SRE process does not replace a natural alert system.

Use compound SLOs where appropriate: You can create a compound SLI to capture multiple SLI conditions and make it easier for customers.

Create customer-specific SLOs where appropriate: It is not uncommon for large customers to receive SLAs that provide them with better service availability than those offered to other customers.

Chapter 2: Creating a fair and effective policy #

The following measure to improving reliability while accelerating deployments is ensuring that your organization can quickly and efficiently deal with any software issues (day or night) promptly and efficiently. To do this, you need an on-call policy.

But that’s because most organizations don’t fully understand the concept of rotation. So this error not only causes stress and regret due to your failed SLAs with customers and sets up an unproductive and unpleasant mood for the exhausted and frustrated engineering team.

Start with the basics. A practical and fair on-call policy requires two crucial prerequisites:

Structured organization and system: Solving problems effectively is much easier when your systems (services or applications) and product teams are all well organized and structured into logical units. For example, our 57 engineering teams at New Relic support 200 individual services. Each team acts autonomously to manage at least three services during the lifecycle, from design to deployment to maintenance.

A culture of responsibility: With DevOps, each team is responsible for the code it deploys in production. Teams naturally make decisions about changes and deployments when they’re in charge and on duty, compared to traditional environments where someone else is in order of the code once it’s in production.

1. Structure your team and your organization fairly #

Each engineer and team leader in the product organization takes responsibility for the team’s services on a rotational basis. Consider the size of your organization’s entire engineering entity and individual teams before choosing an on-call rotation approach. For example, if the team includes six engineers, each engineer could be the primary on-call contact every six weeks.

2. Be flexible and creative when implementing rotations #

You could let each team design and implement its on-call rotation policy. For example, a team uses a script that randomly rotates the secondary person’s on-call order.

3. Track Metrics and Monitor Incidents #

To ensure that the on-call rotation is fair and effective, monitoring it and tracking incident metrics is imperative. At New Relic, we follow the number of calls, the number of call hours, and the number of out-of-hours calls.

Tracking metrics helps us draw attention to teams facing unmanageable call loads (if a team averages more than one out-of-hours call per week, that team is considered to have a cumbersome load with high strain). By mastering these metrics, we can shift priorities by reducing the technical burden on a team or providing more support to improve services.

4. Adapt the policy to align with your business situation #

An on-call policy tailored to a team at New Relic may not apply to your business. To create a fair and effective on-call rotation, consider the following:

Growth: How fast is your business and engineering group growing? Do you have a high staff turnover?

Location: Is your engineering organization centralized or spread across multiple sites? Do you have adequate resources to ensure continuous rotations?

How involved are the dependencies between the services? #

Tools: Do you have issue resolution tools that send automated, actionable issue notifications to engineers?

Culture: Is on-call part of your engineering culture? Do you have a non-blaming culture focused on finding causes and fixing them rather than finding the culprits?

Chapter 3: Responding Effectively to Incidents #

On-call rotations go hand in hand with incident management. What is an Incident? An incident occurs when a system behaves unexpectedly, potentially impacting customers (or partners and employees).

Skill at the heart of the “you develop it, and you own it” DevOps approach, incident management is often devalued, with teams turning away from it once the problem is solved. Organizations lacking effective incident management frequently improvise on the fly, using whatever organization, methods, and communications are available. When there’s a big problem, everyone scrambles to devise a plan to solve it.

There is a much better approach, reducing the duration and frequency of failures and giving responsible engineers the support they need to respond effectively.

Creating an effective incident management process

Define severity levels: Severity levels determine the level of support needed and the potential impact on customers. For example, New Relic uses a severity scale of 1-5.

Level 5: no impact on customers, can be used to raise awareness of a problem.

Level 4: Minor bugs or low data latency impact but do not inconvenience customers.

Level 3: High data latency or unavailable features.

Levels 2 and 1: serious incidents result in breakdowns.

Strengthen your departments: Every department should have a monitoring and alerting system for proactive incident reporting so you can respond to and resolve them as quickly as possible.
Define Response Roles: At New Relic, engineering and support team members play the following roles during an incident: Incident Manager (leads resolutions), Technical Lead (diagnoses and fixes), communications (notifies everyone), communications manager (coordinates emergency communications strategy), incident interface (collaborates with technical support and management for severity level 1), emergency manager (optional for level 1) and engineering manager (manages the post-incident process).
Create an action plan: This is the series of tasks per role, which covers everything that happens in the lifecycle of an incident, including reporting it, defining its severity, choosing appropriate technical leads to contact, debugging and resolving the issue, managing the flow of communications, handing over responsibilities, closing the incident, and conducting post-mortem analysis.
Implement tools and automation to support the lifecycle: from monitoring to alerts, dashboards, and incident tracking; automating the process is essential to keep employees involved and ensure the plan’s effective execution.
Conduct post-incident analysis: After the incident, ask your teams to conduct post-incident research within two days. Emphasize that the purpose of the study is not to find culprits but to understand the natural causes of the problem.
Implement a no-repeat incident policy: If a service issue impacts your customers, it’s time to identify and pay the debt. A no repeat incident policy means that your team will cease all activity on this service until the primary source of the problem has been corrected or contained.

Chapter 4: Overcoming the Complexity of Microservices #

Microservices and DevOps work better together. Companies have realized that breaking monolithic applications into separate services can bring dramatic gains in productivity, speed, agility, scalability, and reliability.

But while teams recognize the changes needed to develop, test, and deploy microservices, they often overestimate the substantial changes required in the areas of collaboration and communications.

Communication best practices for a microservices environment:

Make upstream and downstream dependencies aware of major changes: Before deploying significant changes to your microservices, notify the teams that depend on it upstream and downstream so that, if something goes wrong, they Don’t waste time looking for the root cause.

Communicate quickly and often: This is especially important for version management, deprecations, and temporarily providing backward compatibility.

Treat internal APIs like external APIs: Make your API easier for developers to understand with documentation, detailed error messages, and a process for submitting test data.

Treat downstream teams, i.e., customers: Create a README file with an infrastructure diagram, description and instructions for local execution, and information on how to participate.

Create an announcements-only channel: It’s essential to have a single credible source for important announcements rather than relying on teams to glean important information from multiple boards, chats, and emails.

Using data to understand better how microservices work #

Breaking down your monolithic applications into microservices is very difficult. You must first understand the whole of a system before partitioning it into service boundaries. Then, it can be complicated to partition it precisely to create real microservices (each with a single function, detailed and loosely associated with applications and other services).

Here are some measurement and monitoring tips you can apply to spot microservices in your environment that may have too many interdependencies with other services and applications:

Deployments: Does your team synchronize deployments with upstream and downstream units? You haven’t decoupled your services if you see deployment markers in your monitoring solution synced across multiple services.
Communications: A microservices should require little communication with other services to perform its function. If you see assistance with many round-trip requests to the same downstream services, that’s a clear sign that it’s not decoupled. Throughput is another marker to watch. If the calls per minute for a specific microservices are much higher than the application’s, the service is not decoupled.
Datastores: Each microservices should have its datastore to avoid deployment, database downtime, and schema changes. Good monitoring can show you if each microservices uses its data store.
Scalability: In a proper microservices environment, the peaks of services placed on hosts should match the throughput peaks of individual services. This indicates dynamic extension, one of the essential benefits of microservices. On the other hand, if you see matching spikes across all services and hosts, there’s good reason to believe your services aren’t decoupled.
Developers per application: If you effectively communicate with your microservices teams, you won’t need “architectural gurus” who care for all the microservices to keep them running smoothly. Suppose you have 100 engineers for ten services, and two of your engineers are developing on all ten services. In that case, this is probably a sign that they are not decoupled and that the development of all your services relies on too much knowledge—exclusive and communication skills of these two developers.

Chapter 5: Using Data to Achieve Digital Velocity #

The fundamental element of DevOps is speed: faster software delivery, more immediate problem solving, and more rapid innovation. You’ve achieved digital velocity when you’ve achieved the rate your business needs to seize new market opportunities, beat the competition in innovation, and delight your customers.

Measure and track DevOps performance #

Provide instant feedback that allows everyone to focus on their tasks.
Optimize software delivery, performance, and business results

Here are some fundamentals that should guide your overall DevOps experience:

1. Eliminate data silos #

Even though metrics are at the heart of DevOps doctrine, many teams don’t understand that data silos mean that everyone sees noon on the performance front. With performance data silos, there is no standard, consistent language between applications, infrastructure, and what users experience. Deploying instruments across all your systems to present everything on a single platform puts everyone on the same page, with shared data and metrics to bring different functions together as one team.

At New Relic, our engineering teams struggled with a proliferation of tools and conflicting priorities. Teams started pulling in the same direction by getting all stakeholders to agree on the most crucial SLOs and then putting those SLOs into shared dashboards.

2. Eliminate complexity #

While modern application architectures can simplify and accelerate development in a variety of ways, the dynamic quality of today’s modular architecture creates a new kind of complexity, with many individual components forming a cohesive whole.. Benefiting from a complete vision of your architecture, even if it is ephemeral, allows you to deal with this complexity. In addition, having the correct data helps you understand what’s working and point your team in the right direction.

3. Understand the impact of changes #

With DevOps, deployments and code changes are more frequent; your team needs to master them to avoid potential problems. Ensure you have a reporting feature showing your recent implementations and their before/after impact on application performance and customer experience, including any errors. So you can quickly correlate changes and their potential consequences and enable your team to respond soon, recall a release, or provide a quick resolution to any issues that may arise.

Conclusion #

The popularity of DevOps has spread widely, benefiting companies from many different industries engaged in their pursuit of speed, productivity, quality, and innovation through the adoption of DevOps principles. By achieving maximum digital velocity, high-performing DevOps organizations successfully deploy more, with much faster failure recovery time than their lower-performing competitors.

Data is the fuel of a successful DevOps engine. Best practices, like the ones described in this article, are all about using data well to fuel success. New Relic gives you the necessary visibility to monitor your DevOps efforts and continuously improve your results at every step.

Related reading