Site Reliability Engineer Interview Questions

I'd start by checking our error budget to understand the severity and timeline. First, I'd examine monitoring dashboards to identify patterns - are errors correlated with traffic spikes, specific endpoints, or recent deployments? I'd analyze application logs and metrics using tools like Prometheus and Grafana to isolate the root cause. Next, I'd check infrastructure metrics - CPU, memory, disk I/O, and network latency across all service dependencies. I'd review recent deployments using our CI/CD pipeline logs and consider rolling back if a recent change correlates with the issue. For immediate mitigation, I'd implement circuit breakers or rate limiting to protect downstream services. I'd also examine database performance, connection pool exhaustion, and third-party service dependencies. Once stabilized, I'd conduct a thorough post-mortem to identify systemic improvements like auto-scaling policies, better alerting thresholds, or architectural changes to prevent recurrence.

I'd implement the three pillars of observability: metrics, logs, and traces. For metrics, I'd use Prometheus to collect RED metrics (Rate, Errors, Duration) and USE metrics (Utilization, Saturation, Errors) for each service. Key metrics include request rate, error rate, P95/P99 latency, CPU/memory usage, and business-specific KPIs. For distributed tracing, I'd implement Jaeger to track requests across services, enabling root cause analysis of performance issues. Centralized logging would use ELK stack or similar, with structured JSON logs and correlation IDs. For alerting, I'd create tiered alerts: P0 (immediate paging) for SLO breaches affecting customers, P1 (business hours) for degraded performance, and P2 (email) for early warnings. To prevent alert fatigue, I'd implement alert aggregation, suppress duplicate alerts, use escalation policies, and regularly review/tune alert thresholds. I'd create runbooks for common alerts and use tools like PagerDuty for intelligent routing. Dashboard-wise, I'd build service-level dashboards showing health, dependencies, and SLIs, plus a high-level overview dashboard for executives showing business impact.

```python import os import shutil import logging import subprocess from datetime import datetime, timedelta logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s') logger = logging.getLogger(__name__) def get_disk_usage(path): try: total, used, free = shutil.disk_usage(path) return (used / total) * 100 except Exception as e: logger.error(f"Error getting disk usage for {path}: {e}") return None def cleanup_old_logs(log_dir, days=30): cutoff_date = datetime.now() - timedelta(days=days) cleaned_files = [] try: for root, dirs, files in os.walk(log_dir): for file in files: if file.endswith('.log'): file_path = os.path.join(root, file) file_time = datetime.fromtimestamp(os.path.getmtime(file_path)) if file_time < cutoff_date: os.remove(file_path) cleaned_files.append(file_path) logger.info(f"Removed old log file: {file_path}") return cleaned_files except Exception as e: logger.error(f"Error cleaning logs in {log_dir}: {e}") return [] servers = ['/var/log', '/opt/app/logs'] for server_path in servers: usage = get_disk_usage(server_path) if usage and usage > 80: logger.warning(f"Disk usage {usage:.1f}% exceeds threshold for {server_path}") cleanup_old_logs(server_path) ```

During my time at a fintech company, our payment processing service went down during Black Friday, affecting 40% of transactions. I was the on-call engineer and received automated alerts at 2 AM showing 500 errors spiking. I immediately followed our incident response process: first, I acknowledged the alert and declared a Severity 1 incident, automatically creating a war room Slack channel and conference bridge. I quickly assessed the impact using our monitoring dashboards and determined it was a database connection pool exhaustion issue caused by a recent deployment. For immediate mitigation, I rolled back the deployment and restarted the affected services, restoring service within 20 minutes. Throughout, I provided regular updates every 10 minutes to stakeholders, including executives and customer support. I maintained clear, factual communication about our progress and estimated resolution time. The next day, I facilitated a blameless post-mortem where we identified that our load testing hadn't caught the connection pool configuration error. We implemented action items including better pre-production testing, improved monitoring for database connections, and enhanced deployment safeguards. This experience reinforced the importance of thorough testing and clear communication during incidents.

At my previous company, the product team wanted to launch a new feature before a major industry conference, but our error budget was already 80% consumed due to recent infrastructure changes. The product manager argued that missing the conference would cost significant competitive advantage and revenue. Instead of simply saying 'no,' I proposed a data-driven approach using our SLO framework. I presented the current reliability metrics and calculated the risk of further error budget consumption, showing potential customer impact in concrete terms - estimated 2 hours of additional downtime per month affecting 15% of users. I then worked with the product team to identify the feature's core components and suggested a phased rollout approach. We agreed to launch a basic version with feature flags, allowing us to quickly disable it if reliability suffered. I also negotiated for additional engineering resources to address the underlying infrastructure issues in parallel. We implemented enhanced monitoring specifically for the new feature and established clear rollback criteria. The feature launched successfully with no reliability impact, and we restored our error budget within two weeks through infrastructure improvements. This experience taught me that SRE isn't about blocking features, but enabling sustainable growth through data-driven risk management and creative solutions.

In my previous role, our team spent 6 hours weekly manually provisioning development environments for new developers, often causing 2-3 day delays for onboarding. The process involved creating AWS resources, configuring databases, setting up monitoring, and deploying applications - all done through manual console clicks and custom scripts. I identified this as high-value automation opportunity and proposed building a self-service portal. I spent two weeks developing a Python Flask application that integrated with our existing tools: Terraform for infrastructure provisioning, Ansible for configuration management, and our CI/CD pipeline for deployments. The portal allowed developers to request environments through a simple web form, which would trigger automated workflows. I included approval workflows for cost management and integrated Slack notifications for status updates. Before rollout, I measured baseline metrics: 6 hours average setup time, 3 people involved per request, and 15 support tickets monthly related to environment issues. After implementation, environment provisioning was reduced to 20 minutes of self-service work, zero support tickets for standard requests, and our team reclaimed 24 hours monthly to focus on reliability improvements. The automation also reduced configuration drift and improved consistency across environments. Six months later, developer satisfaction scores for onboarding increased by 40%, and we extended the system to support staging environments.

First, I'd acknowledge the alert and set an initial timeline for assessment. I'd check recent deployments, database connections, and external dependencies since CPU/memory look normal. I'd examine error logs to identify patterns and affected endpoints. If the issue impacts more than our error budget or affects critical user flows, I'd page the on-call engineer and create an incident channel. I'd communicate status updates every 15 minutes to stakeholders. While investigating, I'd prepare rollback options if a recent deployment is suspected. If the error rate stabilizes above 1% after 30 minutes or shows signs of cascading failures, I'd escalate to senior engineers regardless of timezone. Throughout, I'd document findings in our incident tracking system and avoid making changes without peer review during high-stress situations.

I'd schedule an immediate meeting with the development team and product stakeholders to discuss launch readiness. I'd explain that deploying without observability creates blind spots during incidents and could impact our SLA commitments. I'd propose a compromise: implement basic structured logging and health check endpoints as minimum viable monitoring, deploy with feature flags to limit blast radius, and set up basic Prometheus metrics for request rate, latency, and error rate. For circuit breakers, I'd suggest using our service mesh configuration as a temporary solution. I'd offer to pair with developers to implement these quickly, potentially using our existing monitoring templates. If business pressure remains, I'd document the risks formally and propose a phased rollout starting with 5% traffic, expanding only after observing stable metrics. This balances delivery needs while maintaining our reliability standards.

I start by understanding user expectations and business requirements, then analyze similar services in our ecosystem for baseline performance. I examine our infrastructure capabilities, dependency chains, and historical data to set realistic targets. For new services, I typically start with 99.5% availability to allow for learning and iteration, then adjust based on actual performance. When product teams push for unrealistic targets like 99.99%, I explain the engineering cost: going from 99.5% to 99.9% might require 2-3x more engineering effort for redundancy, monitoring, and operational procedures. I use error budgets to show that 99.99% allows only 4.3 minutes of downtime monthly versus 36 minutes for 99.5%. I propose starting conservative and improving gradually based on user feedback and business impact data. I emphasize that missed SLOs damage user trust more than slightly conservative initial targets.

I'd implement a follow-the-sun model with three 8-hour shifts covering primary time zones, using pairs for coverage during peak hours. Each engineer would do one week primary on-call per month, with built-in escalation to secondary on-call after 30 minutes. I'd establish clear on-call expectations: acknowledge alerts within 5 minutes, provide updates every 30 minutes during active incidents, and hand off context during shift changes. To prevent fatigue, I'd implement a maximum of 3 alerts per night policy - beyond that, we escalate to secondary and investigate systemic issues the next day. I'd track on-call metrics like alert frequency, time to resolution, and after-hours incidents per person monthly. For fair distribution, I'd rotate schedules quarterly and compensate after-hours work with time off or adjusted schedules. Most importantly, I'd hold monthly retrospectives to improve runbooks, reduce alert noise, and address automation opportunities. If someone consistently gets more complex incidents, I'd analyze patterns and potentially adjust service ownership.

At my previous company, senior leadership decided to postpone our database migration from legacy infrastructure to save costs, despite increasing outages and performance degradation. I prepared a detailed analysis showing our current system was operating at 85% error budget consumption with incident frequency doubling monthly. I calculated that delaying the migration would cost us approximately $200K in engineering time for firefighting and potential revenue impact from outages. I requested a 30-minute meeting with the VP of Engineering and presented three options: full migration, partial migration of critical services only, or enhanced monitoring with dedicated support for legacy systems. I emphasized how the delay conflicted with our reliability commitments to customers and team morale. The VP appreciated the data-driven approach and business impact framing. We agreed on a hybrid approach - migrating the two most critical services immediately while creating a more comprehensive plan for the rest. This reduced our incident volume by 60% within two months.

I maintain a learning routine through multiple channels: I follow SRE-focused newsletters like SRE Weekly and DevOps Weekly, participate in local SRE meetups, and attend conferences like SREcon annually. I dedicate Friday afternoons to exploring new tools and reading postmortems from other companies. For evaluation, I use a three-tier approach: proof of concept in development environments, pilot with non-critical services, then gradual production rollout. Before adopting new technology, I assess whether it solves a real pain point, has strong community support, and aligns with our team's skill set. For example, when evaluating Prometheus versus our existing monitoring solution, I ran a six-week parallel deployment to compare alert accuracy and operational overhead. I involve the team in decision-making through monthly 'tech radar' sessions where we discuss emerging tools. I prioritize stability over novelty - we only adopt new solutions when they significantly improve reliability, reduce toil, or solve problems our current tools cannot address effectively.