DevOps Engineer Interview Questions

I'd start with `kubectl describe pod <pod-name>` to check events and current status. Then examine logs with `kubectl logs <pod-name> --previous` to see what caused the crash. I'd verify resource limits and requests in the pod spec, check if the container image exists and is accessible, and validate environment variables and config maps. Next, I'd inspect the liveness and readiness probes - incorrect probe configurations often cause crash loops. I'd also check if required secrets or persistent volumes are mounted correctly. If the issue persists, I'd examine the deployment or replica set configuration, verify network policies, and check cluster resource availability. Finally, I'd review recent changes in the CI/CD pipeline or configuration management that might have introduced the issue.

I'd implement a pipeline using GitLab CI/Jenkins with these stages: source control trigger, automated testing (unit, integration, security scans), Docker image building with semantic versioning, and deployment to staging for automated smoke tests. For production, I'd use blue-green deployment with Kubernetes and Istio service mesh for traffic management. The pipeline would deploy to the 'green' environment while 'blue' serves traffic, run health checks and automated tests, then gradually shift traffic using weighted routing. For 2-minute rollbacks, I'd maintain previous versions with automated monitoring triggers that detect anomalies in error rates, response times, or business metrics. ArgoCD would manage GitOps-based deployments with Helm charts, while Prometheus and Grafana provide real-time monitoring with alert rules that can trigger automatic rollbacks via webhook integrations.

I'd use Terraform with a modular approach, creating reusable modules for common infrastructure components like VPCs, ECS clusters, and RDS instances. The directory structure would separate environments with shared modules: `/modules/vpc`, `/modules/database`, `/environments/dev`, `/environments/staging`, `/environments/prod`. Each environment would have its own terraform.tfvars file with environment-specific values like instance sizes, scaling parameters, and network configurations. I'd implement Terraform workspaces or separate state files for environment isolation. For secrets management, I'd use AWS Systems Manager Parameter Store or AWS Secrets Manager with different paths per environment. The pipeline would use GitOps with environment-specific branches, automated terraform plan on PRs, and require approvals for production changes. Version pinning of modules and providers ensures consistency, while Terragrunt could manage remote state and reduce code duplication across environments.

When I joined my previous company, developers were manually deploying to staging using SSH and custom scripts, which led to inconsistent environments and frequent deployment failures. I proposed implementing GitOps with automated CI/CD pipelines, but the team was initially resistant due to concerns about losing control and the learning curve. I started by documenting the current pain points - deployment failures cost us 2-3 hours weekly, and environment inconsistencies caused 20% of our bug reports. I created a pilot pipeline for one microservice, demonstrating 80% faster deployments and zero environment drift. I organized lunch-and-learn sessions to teach the concepts and provided hands-on workshops. Most importantly, I made the transition gradual - developers could still use the old method while the new pipeline ran in parallel. After three weeks of parallel running with better reliability metrics, the team voluntarily switched. The key was showing value through metrics rather than mandating change.

During Black Friday, our e-commerce platform experienced a 300% traffic spike that overwhelmed our payment processing service, causing checkout failures and potential revenue loss of $50K per hour. I immediately initiated our incident response protocol, creating a Slack war room and scheduling a bridge call. I assigned roles: frontend team to implement a graceful error message, backend team to investigate database bottlenecks, and infrastructure team to scale payment service replicas. I coordinated with the business team for customer communications and provided hourly updates to executives with clear ETAs. The key was maintaining a single source of truth in our incident tracker, ensuring all teams knew their specific tasks, and preventing duplicate efforts. I also made the decision to temporarily route 30% of traffic to our backup payment processor while we scaled the primary service. We restored full functionality in 90 minutes. Post-incident, I led a blameless post-mortem that identified auto-scaling improvements and load testing gaps.

Our company decided to migrate from Jenkins to GitLab CI/CD within six weeks due to licensing costs and maintenance overhead, but I had zero experience with GitLab CI. The migration affected 45 applications and couldn't be delayed due to budget constraints. I started by identifying the core concepts I needed: YAML pipeline syntax, GitLab runners, and artifact management. I spent the first weekend going through GitLab's documentation and building simple pipelines for personal projects to understand the fundamentals. I joined GitLab community forums and found similar migration case studies. Most importantly, I collaborated with a colleague who had GitLab experience at their previous company - we scheduled daily 30-minute knowledge-sharing sessions. I created a migration checklist and converted one simple application pipeline first, documenting lessons learned. This became my template for the remaining applications. I also set up a staging GitLab instance to test complex scenarios without affecting production. By week four, I had successfully migrated 80% of pipelines and was training other team members.

First, I'd acknowledge the incident and establish communication channels with stakeholders. I'd start by checking monitoring dashboards (Grafana, CloudWatch) to identify when the degradation began and correlate it with the deployment timeline. Next, I'd examine application logs, infrastructure metrics, and database performance to isolate the issue. If the root cause isn't immediately clear, I'd implement a quick rollback to the previous stable version to restore service. Once service is restored, I'd conduct a thorough post-mortem analysis, document lessons learned, and implement preventive measures like improved health checks and automated rollback triggers. Throughout the process, I'd maintain clear communication with affected teams and provide regular status updates to stakeholders.

I'd start by understanding the team's concerns through one-on-one conversations to identify specific pain points about IaC adoption. Then I'd create a phased migration plan starting with non-critical environments to demonstrate value and build confidence. I'd organize hands-on workshops showing how Terraform can solve their current problems like configuration drift and environment inconsistencies. To address the learning curve, I'd pair experienced team members with those new to IaC and create internal documentation with real examples from our infrastructure. I'd implement quick wins by automating repetitive tasks they currently do manually, showing immediate time savings. Throughout the process, I'd track metrics like deployment frequency and mean time to recovery to demonstrate improved outcomes, sharing success stories in team meetings and celebrating early adopters to build momentum.

I'd implement a three-tier monitoring strategy using Prometheus, Grafana, and AlertManager. For infrastructure, I'd track CPU, memory, disk, and network metrics per node, plus Kubernetes-specific metrics like pod restart counts and resource quotas. For applications, I'd monitor the four golden signals: latency (p95, p99 response times), traffic (RPS per service), errors (4xx/5xx rates), and saturation (queue depths, connection pools). I'd implement distributed tracing with Jaeger to track requests across services. To prevent alert fatigue, I'd use severity-based routing: P0 alerts for service outages go to PagerDuty with immediate escalation, P1 for performance degradation go to Slack, and P2 for capacity planning go to email. I'd implement alert aggregation and suppression rules, use dynamic thresholds based on historical data, and require all alerts to have clear runbooks with remediation steps.

I'd implement a blue-green deployment strategy using a load balancer (ALB or HAProxy) to route traffic between environments. First, I'd containerize the application using Docker and set up identical blue and green environments. The deployment process would involve: building the new version in the inactive environment, running automated health checks and smoke tests, gradually shifting traffic using weighted routing (10%, 25%, 50%, 100%), and monitoring key metrics during each phase. For database changes, I'd use backward-compatible migrations deployed separately from application code, ensuring the old version can still operate during the transition. I'd implement comprehensive health checks that verify database connectivity, external service availability, and core functionality. The load balancer would automatically route traffic away from unhealthy instances. If issues are detected, I can instantly roll back by switching traffic back to the previous environment. This approach eliminates maintenance windows while providing safe rollback capabilities.

I maintain a structured approach to continuous learning through multiple channels. I follow key DevOps communities on Reddit, attend virtual meetups and conferences like KubeCon, and subscribe to newsletters from companies like HashiCorp and Docker. I dedicate 2-3 hours weekly to hands-on experimentation with new tools in my personal lab environment. When evaluating new technologies, I use a framework considering factors like community adoption, vendor stability, integration complexity, and business value. I never introduce new tools directly to production - instead, I start with proof-of-concepts in development environments, gather team feedback, and assess operational overhead. For example, when evaluating service mesh technologies, I tested Istio and Linkerd in isolated environments before recommending Linkerd for its simplicity and operational maturity. I believe in being an early adopter of proven technologies rather than bleeding-edge experimenter, prioritizing system reliability while staying informed about emerging trends that could provide future value.

I encountered this exact situation when developers wanted daily releases but security required 2-day manual reviews for each deployment. I organized a joint meeting to understand both teams' core concerns - developers needed faster feedback cycles to meet sprint goals, while security needed to prevent vulnerable code from reaching production. I proposed implementing 'shift-left' security practices by integrating automated security scanning into the CI/CD pipeline using tools like SonarQube and Snyk. This provided immediate feedback to developers while maintaining security standards. We established risk-based review processes: low-risk changes (documentation, UI updates) went through automated checks only, medium-risk changes required lightweight security review, and high-risk changes (authentication, data handling) kept the full review process. I also set up security champions within development teams to provide first-line security guidance. The result was a 70% reduction in review time while actually improving security posture through earlier detection. Both teams felt heard and the solution addressed their underlying needs rather than just their stated positions.