Amazon DevOps Engineer Interview Questions & Answers

I would correlate the same failing requests across ALB logs, target health, connection signals, optional NLB evidence, application telemetry, traces, and deployment versions. I would first separate ALB-returned 502s from target-returned 502s, identify the first failing hop, correct that boundary, and verify the 502 rate returns to baseline.

The system passes its simple health test, but one out of every five real requests still fails. The goal is to find the first place where those failing requests stop working correctly. I would compare evidence from each step of the request journey during the same time period. I would also check whether failures appeared after a release, settings change, or scaling event. Passing a simple health test only proves that one configured test request works. It does not prove that normal customer requests, heavier traffic, connected services, or every running application version work correctly.

1. Are clients receiving the 502 directly from the ALB, and is there an intermediate NLB anywhere in the affected request path?
2. Are ALB access logs available, and can requests be correlated with timestamps, request IDs, or trace IDs?
3. Did the 20% failure rate begin after a deployment, configuration change, scaling event, or target replacement?
4. Are failures concentrated on specific targets, Availability Zones, deployment versions, routes, or request types?
5. What protocol, port, path, thresholds, and accepted response codes are configured for the target health check?

I would treat this as a fault-boundary isolation problem. First I define one incident window and preserve evidence before changing code or configuration. Then I correlate the same request population across ALB access logs, target health, connection metrics, application telemetry, traces, optional NLB observations, and deployment versions.

I start with the ALB access log because it tells me what the ALB returned and what it observed from the target. I compare elb_status_code, target_status_code, target:port, request_processing_time, target_processing_time, response_processing_time, trace_id, error_reason, classification, and classification_reason when populated. If target_status_code is 502, the target produced an HTTP 502, so I move into application logs and traces. If the ALB returned 502 while the target status is missing and the ALB has connection, protocol, or response-classification evidence, I investigate the ALB-to-target boundary.

For that boundary, I look for target connection failures, resets or early closes, malformed HTTP responses, incorrect target port or protocol, TLS problems, security-group or network-ACL issues, and network-path problems. Connection metrics such as TargetConnectionErrorCount, connection counts, and reset-related evidence help show whether failures cluster around transport behavior. I correlate metrics by the same incident window because aggregate metrics do not identify an individual request by themselves.

Next I examine target health. Target-health reason codes such as Target.ResponseCodeMismatch, Target.Timeout, Target.FailedHealthChecks, and Elb.InternalError describe why a target may not be considered healthy. UnHealthyHostCount is different: it is a CloudWatch health metric, not a reason code. A passing health check proves that the load balancer can reach the target using the configured health-check protocol and port and that the configured health-check path returns an accepted result. It does not prove that real request paths, authentication, headers, payloads, dependencies, capacity under load, every business route, or the intended deployment version work correctly.

If the target returned 502, I correlate the request or trace identifier with application logs and distributed traces. I look for the first failing span or application event, including application-generated 5xx responses, malformed HTTP responses, crashes or restarts, connection-pool exhaustion, downstream database, cache, or API failures, and other resource pressure. Traces may be sampled, so the absence of a trace is not proof that a request did not occur. Logs and metrics still need to be correlated.

If an NLB is actually present in the architecture, I inspect it as an additional transport hop. I look for TCP resets, connection failures, rejected flow evidence, Availability Zone patterns, and capacity indicators. An NLB works at the transport layer and does not itself create an HTTP 502 response, but a reset or failed connection at that hop can cause an upstream HTTP component to fail the request.

I also treat timeout evidence separately. Target.Timeout, high target response time, slow application work, saturated connection pools, database or cache waits, and CPU pressure can identify a latency problem. However, an ALB request timeout normally appears as HTTP 504, so I do not label every timeout as a 502. I use timing evidence to distinguish timeout failures from 502 cases caused by resets, connection failures, early closes, or malformed responses.

Finally, I overlay deployment versions, feature flags, configuration changes, infrastructure changes, and scaling events on the failure timeline. A 20% failure rate can mean that only a subset of targets or one application version is bad. Once the evidence identifies the first failing boundary, I make the smallest safe correction supported by that evidence, such as correcting the target port or protocol, TLS configuration, network policy, malformed response, application defect, dependency issue, or faulty deployment.

For verification, I use the same production signals. The 502 rate should return to its normal baseline, the associated connection or application errors should disappear, target health should remain stable, and key latency and error indicators should improve. For prevention, I define a user-facing availability SLI, such as successful request ratio, and an appropriate SLO before choosing alert thresholds. I alert on sustained 5xx symptoms and error-budget impact with ownership, severity, runbook context, and noise controls. Dashboards should correlate ALB 5xx, target 5xx, target health, latency, application telemetry, and deployment versions. I also control trace sampling, retention, ingestion cost, high-cardinality attributes, clock skew, and missing telemetry, and I redact credentials, tokens, personal information, and sensitive payloads.

1. Define the incident time window and preserve evidence before making changes.
2. Correlate the same failing request population using timestamps, target address, request IDs, or trace IDs.
3. Inspect ALB access logs and compare elb_status_code with target_status_code plus ALB error or classification evidence.
4. If the target returned 502, investigate application logs, traces, dependencies, crashes, malformed responses, and resource pressure.
5. If the ALB returned 502 without a valid target response, inspect target connection failures, resets or early closes, port and protocol configuration, TLS, security groups, network ACLs, and network-path evidence.
6. Review target-health reason codes separately from CloudWatch health metrics such as UnHealthyHostCount.
7. Inspect connection and reset metrics for patterns aligned with the same failure window.
8. If an intermediate NLB exists, inspect its TCP reset, connection, flow, Availability Zone, and capacity evidence as an additional transport hop.
9. Treat timeout evidence as a separate branch because an ALB request timeout normally surfaces as 504 rather than 502.
10. Correlate failures with deployment versions, feature flags, configuration changes, infrastructure changes, and scaling events.
11. Apply only the smallest correction supported by the evidence.
12. Verify that the 502 rate returns to baseline and that health, connection, application, and latency signals improve, then add SLO-based alerts, dashboards, and a runbook.

The investigation becomes more expensive as traffic volume, target count, telemetry volume, and retention increase. ALB access logs can generate large data sets, while distributed traces consume more storage as sampling increases. High-cardinality values such as unrestricted request IDs should be used for logs and traces rather than careless metric labels because they can make metric systems expensive. Longer retention increases storage and query cost. Sampling lowers trace cost but can miss rare failures, so traces should be supported by logs and metrics. Operators must also account for missing telemetry, different system clocks, ingestion delays, maintenance effort, and privacy controls.

Common mistakes are assuming that healthy targets prove real requests are healthy, treating every ALB 502 as an application-generated 502, or relying on one metric without correlating the same request population. Another mistake is treating UnHealthyHostCount as a target-health reason code. Do not assume an NLB exists in every architecture or claim that an NLB itself generates HTTP 502. Do not classify every timeout as 502 because an ALB request timeout normally appears as 504. Other mistakes include changing configuration before preserving evidence, ignoring deployment versions, treating missing sampled traces as proof, mixing unrelated time windows, and logging credentials or sensitive payloads.

Explain the investigation as a request-by-request fault-boundary exercise. Start with ALB versus target status evidence, then walk through target health, connection behavior, application telemetry, optional NLB evidence, and deployment versions. Explicitly state what a passing health check proves and what it cannot prove about real traffic.

I would preserve the timeline, slice the payment funnel, compare the affected path with healthy cohorts, and correlate client, API, trace, dependency, authorization, retry, and business evidence. Green infrastructure narrows the search but does not prove payment success. I would confirm the boundary, fix it safely, and verify recovery.

This question asks how I would find why more shoppers stop buying when they use one way to pay in one place, even though the normal health screens still look fine. I would first save what happened and when. Then I would compare each step of the buying journey with similar shoppers who are not having the problem. I would follow the affected customers from the first payment action to the final result. The goal is to prove where the problem begins, make the smallest safe change, and confirm that buying works normally again.

1. At which checkout step does the first clear drop appear: payment initiation, authorization, or order completion?
2. Do we have a correlation ID or session ID that links client events, API logs, traces, and dependency calls?
3. Should the healthy comparison include both the same Region with other payment methods and the same payment method in other Regions?
4. Were there deployments, configuration changes, provider events, or mitigation actions near the first customer impact?

I would start with the user-visible objective: restore successful checkout for the affected payment method and Region. The smallest useful signal set is funnel conversion, payment authorization success, order-completion success, response-code distribution, latency percentiles, retries, plus supporting client, API, dependency, and business evidence.

I would define SLIs before alert thresholds. An SLI, or service level indicator, is the behavior I measure. Useful SLIs here are payment-authorization success rate, order-completion success rate, payment-related error or timeout rate, and payment latency percentiles, broken down by Region and payment method. An SLO, or service level objective, is the acceptable target for an SLI. I would base symptom alerts on those objectives or a justified healthy baseline, not on CPU alone.

My investigation follows the same seven-step flow as the diagram:

1. SYMPTOM: Confirm that abandonment is higher for one payment method in one Region while broad infrastructure health remains green.
2. PRESERVE + SCOPE: Preserve the incident timeline, including first impact, deployments, configuration changes, provider events, and mitigations.
3. SLICE + COMPARE: Slice by Region, payment method, application version, response code, and latency percentile. Compare the same Region with other payment methods, the same payment method in other Regions, and an equivalent healthy time window.
4. CORRELATE EVIDENCE: Correlate client events, API logs, traces, dependency calls, authorization outcomes, retries, and business success metrics by a consistent session or correlation ID.
5. TEST HYPOTHESIS: Test whether the difference is caused by a client or application regression, API behavior, a dependency or payment provider, authorization behavior, or configuration or data. Reject hypotheses contradicted by the healthy cohort and correlated evidence.
6. ROOT CAUSE + FIX: Confirm the failing boundary before changing anything. Apply the smallest safe correction supported by evidence. If the risk is high, contain or roll back first.
7. VERIFY + MONITOR: Verify funnel recovery, authorization and order-completion recovery, normalized response codes and latency, and no regression in the comparison cohort. Continue monitoring afterward.

Green infrastructure dashboards are useful, but limited. Healthy CPU, memory, Pod status, and broad capacity signals reduce suspicion of a widespread compute or orchestration problem. They cannot prove that a payment request was authorized correctly, that a dependency succeeded, that payment-specific configuration is correct, or that customers completed checkout.

The telemetry lifecycle also matters. Client instrumentation, APIs, services, dependencies, and business logic create the evidence. Instrumentation or a collector can enrich it with consistent attributes such as Region, payment method, application version, response code, service identity, and correlation ID. High-cardinality values such as raw user IDs should not become unrestricted metric labels. Traces may be sampled to control cost, while important business counters and error signals should remain complete enough to show the real symptom. Telemetry is transported through the observability pipeline, stored in the appropriate log, trace, and metric backends, queried using shared attributes, and retained long enough to cover the incident and comparison window.

There are important tradeoffs. Sampling lowers trace ingestion cost but can miss rare failures or bias the evidence if the sampling policy favors successful traffic. Aggregation makes dashboards cheaper and easier to read but can hide a payment-specific problem if Region or payment method is aggregated away. High-cardinality dimensions improve slicing but can increase storage and query cost. Missing client telemetry reduces confidence about browser or application behavior. Clock skew between clients, services, and dependencies can distort the incident timeline, so timestamps should be synchronized and correlation should not rely on time alone. Retaining more detailed telemetry helps later investigation but increases storage cost. Sensitive payloads, credentials, tokens, and personal data must be redacted or excluded.

For alerting, I would use customer-symptom alerts based on the SLIs and SLOs above. Each alert should have an owner, severity, affected Region and payment method, runbook context, and noise controls such as a minimum traffic level and a sustained evaluation window. I would test the dashboards and alerts with known failure scenarios or synthetic payment checks and confirm that a real payment-path problem changes the business SLIs even when CPU, memory, and Pod health remain normal.

1) Confirm the customer-visible symptom and preserve the incident timeline. 2) Slice the funnel by Region, payment method, application version, response code, and latency percentile. 3) Build healthy comparison cohorts from the same time window. 4) Correlate client events, API logs, traces, dependency calls, authorization outcomes, retries, and business metrics with consistent identifiers and attributes. 5) Test client, API, dependency, authorization, configuration, and data hypotheses; reject those contradicted by evidence. 6) Confirm the failing boundary and apply the smallest safe correction, containment, or rollback. 7) Verify funnel, authorization, order-completion, response-code, and latency recovery, then monitor for regression.

The main cost is observability data, not algorithm speed. More detailed logs, traces, and dimensions make diagnosis easier but increase ingestion, storage, query, and retention costs. High-cardinality labels such as unique users or sessions can make metric systems expensive, so they should be controlled and used mainly where appropriate for logs or traces. Sampling reduces trace cost but may hide rare failures. More dashboards and alerts also require maintenance, testing, ownership, and tuning so that operators are not overwhelmed by noise.

Common mistakes are treating green CPU, memory, and Pod dashboards as proof that checkout is healthy; changing code before preserving evidence; looking only at infrastructure instead of the user funnel; failing to compare with a healthy cohort; aggregating away Region or payment method; treating one trace or log as proof of root cause; ignoring client events, authorization outcomes, retries, or business success metrics; using uncontrolled high-cardinality metric labels; overlooking sampling bias or clock skew; and verifying only infrastructure after the change instead of confirming customer and business recovery.

Explain the sequence in order: preserve, slice, compare, correlate, test, confirm, fix, verify. State explicitly that green infrastructure narrows the fault boundary but does not prove payment success. That distinction is the key judgment this question is testing.

I would validate the APPROVED_JSON file first, then use the default AWS credential chain to discover every enabled Region. Before querying any instances, I would verify that every enabled Region has one non-empty approved AMI. Then I would query each Region with pagination, finite timeouts, and at most three total attempts for retryable AWS calls. I would collect non-terminated instances using the wrong AMI, sort them by Region and instance ID, print TSV output, and return exit 0, 1, or 2.

The script receives one JSON file that maps each enabled AWS Region to its approved AMI. It first checks the argument and the JSON file. Next, it asks AWS which Regions are enabled. Before looking at any EC2 instances, it verifies that every enabled Region has a non-empty AMI mapping. Then it checks every instance except terminated instances. It records only instances whose current AMI differs from the approved AMI. The script is read-only. It prints deterministic tab-separated output and handles failures without stopping healthy Regions.

1. Should a missing AMI mapping for any enabled Region stop the script before all instance queries? Yes. The required behavior is exit 2 before any DescribeInstances call.
2. Should stopped and shutting-down instances also be checked? Yes. Every state except terminated is included.
3. Should one failed Region stop processing the remaining Regions? No. The failure should be logged, the other Regions should continue, and the final status should be 1.

The program requires exactly one argument. That argument is the APPROVED_JSON file path. The file must be readable. Its top-level value must be a JSON object. Every Region key and every AMI value must be a non-empty string. If these checks fail, the script writes an error to stderr and exits with status 2. No instance query has happened yet.

The script calls EC2 DescribeRegions using the normal AWS default credential chain. It keeps Regions whose OptInStatus is opt-in-not-required or opted-in. The AWS CLI call uses finite connect and read timeouts. AWS standard retry behavior is limited to three total attempts. If Region discovery cannot succeed, the script logs the failure and exits with status 1 because this is an AWS operational failure rather than invalid input.

After Region discovery succeeds, the script checks every enabled Region against APPROVED_JSON. This happens before the first DescribeInstances call. If any enabled Region is missing or maps to an empty value, the script logs the configuration problem and exits with status 2. The key invariant is simple: instance scanning starts only after every enabled Region has a valid approved AMI.

For each enabled Region, the script reads that Region's approved AMI and calls DescribeInstances. AWS CLI pagination remains enabled, so all pages are processed. Each returned instance provides InstanceId, ImageId, and State. The script ignores only instances whose state is terminated. Pending, running, shutting-down, stopping, and stopped instances are eligible. If current ImageId is different from the approved AMI, the script records region, instance ID, current AMI, and approved AMI.

A non-retryable DescribeInstances failure makes that Region fail immediately. A retryable failure can be retried by the AWS CLI, but there are at most three total attempts. If a Region still cannot be queried, the program writes the error to stderr, increments its Region-failure count, and continues with the remaining Regions. This preserves useful results from healthy Regions. At the end, any such Region failure makes the final exit status 1.

APPROVED_JSON contains {"us-east-1":"ami-new"}. In us-east-1, running instance i-1 uses ami-old and running instance i-2 uses ami-new. Both instances are eligible because neither is terminated. For i-1, ami-old differs from ami-new, so the script records a mismatch. For i-2, the values are equal, so it records nothing. The only stdout record is the four tab-separated fields us-east-1, i-1, ami-old, and ami-new.

After the Region loop, the script sorts mismatch records first by Region and then by instance ID. It prints region, instance_id, current_ami, and approved_ami as tab-separated fields. It logs Region count, scanned instance count, and mismatch count to stderr. With no operational Region failures it exits

1. If at least one Region failed, it exits
2. Invalid arguments, invalid JSON, or an incomplete Region mapping use exit 2 before instance queries.

The main idea is to separate configuration validation from EC2 scanning. The central invariant is that no DescribeInstances call begins until every enabled Region has one non-empty approved AMI. Once that invariant is true, each Region can be processed independently. The script compares every non-terminated instance ImageId with the approved AMI for that Region and stores only mismatches. AWS failures are isolated per Region so healthy Regions still produce results. Finally, sorting by Region and instance ID gives deterministic output. All AWS operations are reads, so an unchanged account produces the same result without modifying resources.

The Bash program first validates that exactly one APPROVED_JSON file was supplied. jq confirms that the file contains a JSON object and that every key and value is a non-empty string. A small AWS wrapper applies finite connect and read timeouts and configures the AWS CLI standard retry mode for at most three total attempts. DescribeRegions discovers the enabled Regions through the default credential chain. The program then validates every enabled Region mapping before it makes any DescribeInstances call. During the Region loop, DescribeInstances automatically paginates all results. jq produces InstanceId, ImageId, and State. The loop skips only terminated instances, counts the other instances, and appends ImageId mismatches to a temporary TSV file. If one Region cannot be queried, the script logs that failure and continues. At the end, sort orders the mismatch rows by Region and instance ID. The script prints them to stdout, writes counts to stderr, exits 1 after any Region failure, otherwise exits 0. Invalid input exits 2 before instance queries.

A common mistake is starting DescribeInstances before proving that every enabled Region has a valid approved AMI. That breaks the required exit-2 behavior. Another mistake is filtering only running instances instead of checking every state except terminated. Candidates may also disable pagination, forget finite AWS CLI timeouts, or allow more than three total attempts. A Region failure should not stop healthy Regions. Other common errors are mixing operational failures with invalid-input exit 2, printing unsorted rows, or using spaces instead of the required four tab-separated fields.

Explain the pre-flight safety rule first: discover enabled Regions and validate the complete Region-to-AMI mapping before any DescribeInstances call. Then explain the read-only per-Region scan, bounded AWS retries, failure isolation, and final deterministic TSV sort.