Are you drowning in dashboards? Do you have thousands of metrics but no clear understanding of whether your service is actually reliable? If so, you’re not alone. In the world of microservices, it’s easy to collect data. The real challenge is turning that data into a meaningful measure of service health.

This is where Service Level Indicators (SLIs) and Service Level Objectives (SLOs) come in. They are the cornerstone of Site Reliability Engineering (SRE) and provide a powerful framework for defining, measuring, and communicating the reliability of your services.

For too long, engineering teams have relied on low-level system metrics like CPU utilization, memory usage, and disk I/O. While these are useful for debugging, they don’t tell you what your users are experiencing. Your CPU could be at 10%, but if every request is timing out, your service is failing.

SLIs and SLOs shift the focus from system health to user happiness. They provide a shared language for product, engineering, and business teams to agree on what reliability means and how much of it is “good enough.”

SLI, SLO, SLA: What’s the Difference?

Let’s clarify the terminology:

• SLI (Service Level Indicator): A quantitative measure of some aspect of the level of service that is provided. It’s a metric that matters.
• SLO (Service Level Objective): A target value or range of values for an SLI. This is the goal you are trying to meet.
• SLA (Service Level Agreement): An explicit or implicit contract with your users that includes consequences for meeting or failing to meet your SLOs.

Key takeaway: You measure SLIs to check if you’re meeting your SLOs. If you don’t, you might violate your SLA. This guide focuses on SLIs and SLOs, the internal tools for building reliable systems.

Choosing the Right SLIs: What to Measure?

The most effective SLIs are tied directly to user-facing journeys. For microservices, you can categorize them based on the service’s function. The Google SRE book suggests four golden signals, which are a great starting point:

• Latency: The time it takes to service a request.
• Traffic: The amount of demand being placed on your system.
• Errors: The rate of requests that fail.
• Saturation: How “full” your service is.

Here’s how to apply these to different types of microservices:

1. User-Facing Request/Response Services (e.g., API Gateway, Frontend Service)

• Availability SLI: The proportion of successful requests. (successful_requests / total_requests)
• Latency SLI: The proportion of requests served faster than a threshold. (fast_requests / total_requests)

Example:

• SLI: Percentage of HTTP GET requests to /api/v1/users/{id} that complete successfully in under 300ms.
• SLO: 99.9% over a rolling 28-day window.

2. Asynchronous Processing Services (e.g., Message Queue Worker)

• Freshness SLI: The age of the oldest message in the queue. max(time.now() - message.timestamp)
• Throughput SLI: The rate at which messages are being processed. (processed_messages / time_period)
• Execution Errors SLI: The proportion of messages that fail processing. (failed_messages / total_messages)

Example:

• SLI: Percentage of messages in the payment-processing queue that are processed within 60 seconds of being enqueued.
• SLO: 99.5% over a rolling 28-day window.

3. Data Processing Pipelines (e.g., ETL Jobs)

• Freshness SLI: The time elapsed since the last successful data update. time.now() - last_successful_run_timestamp
• Correctness SLI: The percentage of data that is free of defects. This often requires a separate validation process. (valid_records / total_records)
• Throughput SLI: The amount of data processed per unit of time. (bytes_processed / second)

Example:

• SLI: The percentage of daily data aggregation jobs that complete successfully within the 2-hour batch window.
• SLO: 99% over a 90-day window.

Defining SLOs: How Good is Good Enough?

An SLO is a statement of intent. It’s a reliability target that balances customer expectations with the cost and complexity of achieving it. A 100% SLO is almost always the wrong target, it’s impossibly expensive and leaves no room for innovation or failure.

The Error Budget: Your Secret Weapon

The difference between your SLO and 100% is your error budget.

Error Budget = 100% - SLO

For a 99.9% SLO, your error budget is 0.1%. This is the amount of unreliability you are allowed to have over the SLO window.

The error budget is the most powerful tool that SRE provides. It transforms conversations about reliability from emotional debates into data-driven decisions.

• If you have error budget remaining: You are free to ship new features, take risks, and perform maintenance that might cause some instability.
• If you have exhausted your error budget: All development on new features stops. The team’s entire focus shifts to reliability improvements, bug fixes, and stability work until the service is back within its SLO.

This creates a self-regulating system that perfectly balances innovation and reliability.

A Practical Implementation with Prometheus

Let’s define an availability and latency SLI for a hypothetical user-service.

Step 1: Instrument Your Application

Your service needs to export metrics. Using a Prometheus client library, expose the total number of HTTP requests and their latencies, partitioned by status code and path.

# Example using Flask and prometheus_client
from flask import Flask
from prometheus_client import Counter, Histogram, make_wsgi_app
from werkzeug.middleware.dispatcher import DispatcherMiddleware

app = Flask(__name__)

# Metrics for SLIs
REQUEST_LATENCY = Histogram('http_requests_latency_seconds', 'Request latency', ['method', 'endpoint'])
REQUEST_COUNT = Counter('http_requests_total', 'Total requests', ['method', 'endpoint', 'status_code'])

@app.route('/users/<user_id>')
@REQUEST_LATENCY.labels(method='GET', endpoint='/users/<user_id>').time()
def get_user(user_id):
    # Your logic here
    status_code = 200
    # ...
    REQUEST_COUNT.labels(method='GET', endpoint='/users/<user_id>', status_code=status_code).inc()
    return {"user_id": user_id, "name": "John Doe"}

# Add Prometheus wsgi middleware to route /metrics requests
app.wsgi_app = DispatcherMiddleware(app.wsgi_app, {
    '/metrics': make_wsgi_app()
})

Step 2: Define SLIs in PromQL

Now, we can query these metrics in Prometheus to calculate our SLIs.

Availability SLI (requests that are not 5xx errors):

# SLI: Proportion of successful requests to the user service
sum(rate(http_requests_total{job="user-service", status_code!~"5.."}[5m]))
/
sum(rate(http_requests_total{job="user-service"}[5m]))

Latency SLI (requests faster than 300ms):

# SLI: Proportion of fast requests to the user service
sum(rate(http_requests_latency_seconds_bucket{job="user-service", le="0.3"}[5m]))
/
sum(rate(http_requests_latency_seconds_count{job="user-service"}[5m]))

Step 3: Set Up SLOs and Error Budget Alerts

With our SLIs defined, we can now set up alerting rules in Prometheus to notify us when we are burning through our error budget too quickly.

Let’s say our availability SLO is 99.9% over 28 days. Our error budget is 0.1%.

groups:
- name: slo_alerts
  rules:
  - alert: HighErrorBudgetBurn
    expr: |
      # Availability SLO: 99.9% over 28 days
      # Error budget: 0.1%
      # Alert if we burn through 2% of the monthly budget in 1 hour
      sum(rate(http_requests_total{job="user-service", status_code=~"5.."}[1h]))
      /
      sum(rate(http_requests_total{job="user-service"}[1h]))
      > (0.001 * 28 * 24 * 0.02) # 0.1% budget * 28 days * 24 hours * 2% burn rate
    for: 5m
    labels:
      severity: page
    annotations:
      summary: "High error budget burn for user-service"
      description: "The user-service has burned through 2% of its 28-day error budget in the last hour."

This alert is incredibly powerful. It doesn’t just fire when the system is down; it fires when the rate of failure is high enough to jeopardize the monthly SLO. This gives you an early warning to fix issues before they become SLO-violating events.

Common Pitfalls to Avoid

1. Too Many SLIs: Start with 1-3 critical, user-centric SLIs. Don’t try to measure everything.
2. Ignoring the User Journey: Don’t define SLIs in a vacuum. Talk to your product managers and users to understand what they care about.
3. Setting Unrealistic SLOs: Don’t aim for 100%. The cost of “five nines” is astronomical. Is it worth it for your service?
4. No Buy-In: SLOs and error budgets are a cultural tool. If your leadership and product teams don’t agree to the rules (e.g., freezing features when the budget is spent), the system will fail.
5. Analysis Paralysis: Don’t spend months perfecting your SLIs. Start with something simple, measure it, and iterate. “Good enough” is better than nothing.

Conclusion: A New Conversation About Reliability

SLOs and error budgets are not just metrics; they are a framework for making objective, data-driven decisions about reliability. They align incentives across teams and empower engineers to take ownership of their service’s stability.

By moving the conversation from “the site is slow” to “we’ve consumed 75% of our monthly latency error budget,” you transform a subjective complaint into an objective, actionable signal.

Your action plan:

1. Pick one critical, user-facing service.
2. Define one availability and one latency SLI for it.
3. Negotiate a “good enough” SLO with your product team.
4. Implement the SLI tracking and set up an error budget burn alert.
5. Start having data-driven conversations about reliability.

Stop chasing perfection and start managing reliability. Your engineers, your product managers, and most importantly, your users will thank you.