Microsoft DevOps Engineer Interview Questions & Answers

I would time every deployment stage under one run ID, compare each duration with a known-good baseline, and investigate the largest differences. Then I would use timestamps, retries, cache behavior, external latency, resource signals, and telemetry gaps to prove the bottleneck before changing it.

A deployment taking more than two hours only tells me that the whole process is slow. It does not tell me which part is responsible. I would divide the process into smaller steps and measure how long each one takes. Then I would compare those times with a normal deployment and focus on the biggest difference. I would check whether the delay is expected, whether work is still moving slowly, or whether a step has stopped or keeps trying again. I would change only the proven problem, repeat the run, and watch later runs.

1. Do we already record start time, end time, status, and retries for every deployment stage?
2. Is there a known-good deployment or historical baseline I can compare with?
3. Are approval waits or other deliberate pauses included in the two-hour duration?
4. Can one deployment run ID be correlated across stage records, logs, metrics, traces, and approvals?
5. Have agent capacity, caching, retries, or external services recently changed?

I would use an evidence-first flow: measure every stage, correlate one run, compare with normal behavior, classify the delay, change only the proven bottleneck, and verify with the same measurements.

The symptom is simple: the deployment takes more than two hours. I would not assume which stage is slow from the total duration alone. The total should be treated as the sum of identifiable stages.

I would capture start time, end time, duration, wait time, status, and retry count for the major stages:

• Queue wait
• Agent provisioning
• Checkout
• Dependency restore
• Build
• Tests
• Scans
• Artifact transfer
• Approval wait
• Deployment
• Rollout readiness
• Post-deployment verification

This shows where the time is actually spent instead of treating the whole pipeline as one number.

I would use one run ID for the deployment and correlate that same run across stage records, structured logs, metrics where appropriate, traces when available, deployment events, and approval records.

The run ID prevents evidence from different deployments from being mixed together. I would also keep timestamps consistent and account for clock skew because incorrect clocks can make stage boundaries look wrong.

For metrics, I would avoid putting a unique run ID on every metric series because that creates high cardinality and unnecessary ingestion and storage cost. Run-level detail is usually better correlated through logs, traces, events, or carefully designed links from aggregate metrics.

I would compare every current stage duration with a known-good deployment or historical normal range. I would inspect the largest positive duration differences first.

If queue time increased sharply, I would investigate capacity or scheduling. If dependency restore became slower, I would investigate cache effectiveness and package-feed latency. If approval wait explains the extra time and the approval is intentional, that is deliberate waiting rather than a technical hang.

A historical median and higher percentiles are more useful than relying on one previous run when enough history exists because normal duration can vary from run to run.

For the stages with the largest differences, I would collect evidence such as:

• Stage timestamps and progress events
• Retry and backoff history
• Cache hits, misses, and reuse behavior
• External-service latency and errors
• CPU, memory, disk, and network saturation
• Structured logs and deployment events
• Traces across instrumented boundaries
• Missing telemetry or gaps in expected events

Each signal answers a different question. Metrics show trends and saturation. Logs show detailed events and errors. Traces show latency across instrumented boundaries. None of them alone automatically proves the root cause.

I would classify the abnormal time into one of three practical states.

Expected wait: The deployment is intentionally waiting, such as for an approval or scheduled gate. Records should explain why the stage is paused.

Slow but progressing: The stage continues making forward progress, but it is slower than the baseline. Possible areas to investigate include limited resources, serialized work, weak cache reuse, large artifacts, or slow external services.

Hung or retrying: The stage shows no meaningful forward progress, repeatedly fails, repeatedly retries, spends time in backoff, or misses expected progress events. I would inspect retries, timeouts, dependency health, and stage-specific evidence.

The distinction matters because the correction depends on the evidence. More capacity will not fix a deliberate approval wait, and increasing a timeout will not correct an operation that repeatedly fails.

After narrowing the slow stage, I would test one hypothesis at a time.

For resource saturation, I would correlate the slow interval with CPU, memory, disk, or network pressure. For retries, I would correlate retry timestamps and backoff time with the stage duration. For caching, I would inspect hits, misses, and whether reusable inputs changed. For an external dependency, I would compare its latency and error behavior with normal runs.

If the evidence disproves a hypothesis, I would reject it and move to the next supported explanation rather than continuing as though it were confirmed.

I would change only the proven bottleneck. The exact correction depends on the confirmed cause. It might involve restoring effective caching, removing unnecessary serialization, correcting a retry condition, providing appropriate capacity to a saturated stage, or addressing a slow external dependency.

I would avoid changing several unrelated things at the same time because that makes it difficult to know which change helped and increases deployment risk.

I would rerun the deployment with the same stage measurements. I would compare the affected stage duration with the known-good baseline and confirm that the total duration improved for the expected reason.

I would also check that retries, failures, or other health signals did not become worse. Finally, I would watch several later runs for regression instead of declaring success after one unusually fast deployment.

The primary service-level indicator, or SLI, is deployment duration. I would also track stage-duration SLIs because the total duration alone does not identify the bottleneck. A service-level objective, or SLO, defines the acceptable deployment-performance target over time. Alert thresholds should follow that objective rather than being arbitrary.

A useful dashboard would show total deployment duration, stage durations, queue time, retry counts, success or failure state, cache effectiveness where available, and relevant resource or external-service signals. Operators should be able to compare a current run with normal historical behavior.

Alerts should be symptom based and actionable. A deployment-duration SLO breach should identify the affected pipeline or environment, ownership, severity, and the runbook or investigation view. Noise controls should prevent repeated alerts for the same condition.

Telemetry has operational limits. Sampling can hide some traces. Short retention can remove evidence before an investigation begins. Missing instrumentation creates blind spots. High-cardinality labels increase metric cost. Clock skew can distort duration calculations. Logs and traces can increase ingestion and storage costs. Credentials, tokens, personal information, and sensitive payloads must be redacted before telemetry is stored.

The key takeaway is: total deployment time equals the sum of its stages. Measure before optimizing. Correlate one run, compare with a baseline, prove the bottleneck, make the smallest safe correction, and verify it.

1. Record the deployment symptom and one run ID.
2. Split the run into queue, agent, checkout, restore, build, tests, scans, artifact, approval, deployment, readiness, and verification stages.
3. Capture start time, end time, duration, wait time, status, and retries for each stage.
4. Compare every stage with a known-good baseline.
5. Investigate the largest duration differences first.
6. Correlate timestamps, retries, cache behavior, external latency, resource saturation, logs, metrics, traces, approvals, and telemetry gaps using the same run ID.
7. Classify the delay as expected waiting, slow but progressing, or hung/retrying.
8. Test the strongest evidence-supported hypothesis and reject disproved hypotheses.
9. Apply only the smallest correction supported by confirmed evidence.
10. Rerun with the same measurements, compare with the baseline, and monitor later runs for regression.

Stage timing is inexpensive because it needs only a small amount of data for each deployment stage. Logs and traces can create much more stored data, especially with long retention or high collection rates. Metrics are efficient for trends, but high-cardinality labels can become expensive. Additional instrumentation also creates maintenance work. The goal is to collect enough evidence to explain the delay without creating unnecessary data volume, cost, privacy risk, or operational complexity.

Common mistakes are looking only at the total two-hour duration, guessing which stage is slow, optimizing before measuring, comparing unrelated runs, treating an intentional approval wait as a technical failure, treating a slow but progressing stage as hung, ignoring retries and backoff time, ignoring cache misses or external-service latency, assuming one signal such as CPU usage proves the cause, using unique run IDs indiscriminately as metric labels, overlooking telemetry gaps or clock skew, changing several things at once, and declaring success after one faster run without monitoring later deployments.

Present the answer as a clear diagnostic sequence: decompose the two hours into stage durations, correlate one run ID, compare each stage with a known-good baseline, investigate the largest delta, classify the delay as expected waiting, slow progress, or hung/retrying behavior, change only the proven bottleneck, and verify using the same measurements.

I would propagate W3C trace context everywhere, create spans with OpenTelemetry, buffer and export them through scalable collectors, apply redaction, deduplication and head or tail sampling, and store retained traces in indexed tiers. I would track freshness, completeness and sampling bias so operators know how trustworthy the evidence is.

The goal is to follow one user request as it moves through many parts of a system, even when some work happens later or on another machine. Operators should be able to see where time was spent, where work failed, and how the pieces are connected. The design must keep working during temporary failures, avoid collecting private information, control storage cost, and keep different customers separated. It must also show when information is late, missing, or selectively kept, because an incomplete picture can lead to the wrong conclusion.

1. What request and span volume should the tracing system support?
2. Which synchronous protocols, message systems, and background-job mechanisms are in scope?
3. What trace-search freshness and retention requirements matter most?
4. Are there tenant-isolation, regional, privacy, or compliance constraints?
5. Which investigations must remain possible when aggressive sampling is enabled?

I would design the system as one end-to-end flow: context propagation, span creation, collection, processing, storage, query, and trace-quality reporting.

For synchronous HTTP or gRPC calls, I would use W3C Trace Context. The traceparent header carries the trace ID, parent span ID, trace flags, and version information. tracestate can carry vendor-specific state.

For asynchronous messaging, the producer injects trace context into message headers and the consumer extracts it before creating its span. For scheduled or background work, I would continue the parent relationship when that causality is valid, or create a new root span with an explicit span link when the work is delayed, fan-out, fan-in, or otherwise not well represented by a simple parent-child relationship.

A trace ID identifies the whole distributed operation. A span ID identifies one operation inside that trace. Parent span IDs describe the main causal chain. Span links represent additional causal relationships, which is important for asynchronous and batch processing.

I would not rely on wall-clock timestamps alone for ordering because hosts can have clock skew. Causality should come primarily from parent-child relationships and span links, while timestamps and collector arrival times provide supporting timing evidence. The system should expose observed clock-skew or timing anomalies when they affect interpretation.

Instrumented services create spans through OpenTelemetry SDKs or supported auto-instrumentation. Each span should contain stable operation names, start and end timestamps, status, span kind, and carefully selected attributes.

Resource attributes describe where telemetry originated, such as service name, service instance, deployment environment, cloud, region, and relevant runtime identity. I would follow OpenTelemetry semantic conventions so common operations are described consistently across services.

For services that are not yet instrumented, I would use supported auto-instrumentation or observe boundaries through an ingress proxy, service mesh, or eBPF-based tooling where appropriate. These methods can reveal request boundaries and latency, but they may not expose every internal application operation or business attribute. That instrumentation gap must be visible rather than treating partial coverage as complete application tracing.

Applications export spans with OTLP to a local or nearby OpenTelemetry agent or collector. The collector batches telemetry before forwarding it. A bounded memory buffer absorbs short bursts, and an optional disk spool can absorb longer downstream interruptions.

Exporters use bounded retry with backoff for retryable failures. I would not allow telemetry queues to grow without limit because observability must not exhaust application or host resources. When downstream capacity is insufficient, an explicit backpressure policy decides whether to use disk buffering, reduce sampling, or shed telemetry. Queue saturation, export failures, dropped spans, and spool utilization should themselves be observable.

Retries can create duplicate deliveries, so ingestion should support idempotent processing or deduplication based on stable span identity such as trace ID plus span ID. The implementation must avoid treating separate legitimate spans as duplicates.

Regional collectors should be stateless or horizontally scalable where practical. They authenticate ingestion, use TLS, enforce tenant boundaries, validate schema, transform or redact sensitive fields, perform sampling where configured, and forward accepted telemetry to storage.

Head sampling makes an early keep-or-drop decision. It is inexpensive and protects applications, collectors, network bandwidth, and storage from excessive volume. Its limitation is that the decision happens before the final trace outcome is known.

Tail sampling delays the decision until enough of a trace has arrived. It can preferentially retain errors, unusually slow traces, important operations, or traces selected by tenant or business policy. The tradeoff is that it requires buffering partial traces, additional memory, consistent routing, and a bounded decision timeout.

Late spans may arrive after the tail-sampling decision. The system therefore needs an explicit late-arrival policy and must expose late-span counts or incomplete-trace indicators rather than implying that every retained trace is complete.

Dynamic policies can combine head sampling, tail sampling, per-service budgets, and per-tenant quotas. The effective sampling rate and sampling policy should be visible because retained traces are not automatically representative of all traffic.

Every span must be associated with the correct tenant before it enters shared processing. I would authenticate telemetry ingestion, enforce tenant-aware quotas, isolate tenant data logically or physically as required, and enforce authorization again on the query path.

Credentials, authentication tokens, personal information, secrets, and sensitive payloads should not be collected by default. I would use attribute allow-lists where practical and perform redaction or transformation before durable storage. Encryption protects telemetry in transit and at rest, and access to trace data should be audited.

Attribute cardinality also needs limits. Unbounded values such as arbitrary IDs or payload fields can make indexing and storage very expensive, so only useful and controlled attributes should be indexed.

The trace store can use immutable, compressed chunks partitioned by time and commonly by tenant and service. A separate index should cover fields operators search most often, such as trace ID, service, operation, status, tenant, time range, and carefully selected tags.

A trace-ID lookup maps to the stored trace data. Searches by service, operation, status, tenant, or time use the index to find matching traces and then load their spans. I would avoid indexing every arbitrary attribute because high-cardinality indexes become expensive.

Consistent with the diagram, I would use tiered retention: approximately 7-15 days in a high-performance hot tier, 30-90 days in a cost-optimized warm tier, and roughly 6-24 months in cold archive when business or compliance requirements justify it. These are design examples, not universal requirements, so I would tune them to the actual retention, compliance, and cost constraints.

Operators need trace-ID search, service and operation search, latency views, service-dependency views, critical-path or waterfall analysis, and span drill-down. Trace views should correlate with logs, metrics, profiles, events, and other diagnostic signals when common resource attributes and correlation identifiers exist.

I would not claim that a trace alone proves root cause. A slow span shows where elapsed time was observed, but determining why it was slow might require database telemetry, logs, profiles, infrastructure metrics, events, or other evidence.

The tracing platform needs its own SLIs. I would track ingestion success rate, end-to-end ingestion latency, collector queue saturation, exporter failures, dropped-span rate, sampling rate, tail-sampling decisions, late spans, and collector health.

I would define SLOs with the interviewer around required freshness and acceptable telemetry loss rather than inventing arbitrary thresholds. Alerts should represent conditions that materially reduce investigation capability, such as sustained ingestion delay, excessive drops, or queues approaching their safe capacity. Alerts should have an owner, severity, runbook context, and noise controls.

I would expose three data-quality dimensions beside trace queries and supporting dashboards.

Freshness describes whether the evidence is recent enough for the investigation. Useful signals include ingestion latency, last successful ingest time, last-seen timestamps, and collector health.

Completeness describes how much of the real execution was captured. Useful signals include ingestion success, dropped spans, missing context, known instrumentation gaps, effective sampling rate, and expected-versus-observed spans where that expectation can be measured.

Bias describes whether retained traces represent the wider request population. Useful signals include head-sampling rates, tail-sampling criteria, policy differences by service or tenant, population coverage, and known blind spots.

A fresh trace can still be incomplete. A complete retained trace can describe that individual execution accurately while still being statistically biased if sampling preferentially keeps errors or slow requests. Operators should see these limitations before generalizing from trace evidence.

Collector failures should result in bounded local buffering, retry with backoff, disk spooling where appropriate, and failover to healthy collectors when available. Collectors should be horizontally scalable and avoid unnecessary local state. If downstream capacity remains unavailable and buffers reach their limits, telemetry should degrade according to an explicit policy instead of consuming resources without bound.

Cost is controlled through head sampling, tail sampling, per-tenant quotas, batching, compression, attribute-cardinality controls, selective indexing, and tiered retention. High-value traces can be retained longer than routine traces when that tradeoff is justified.

Finally, I would test the design with known synchronous calls, asynchronous messages, background jobs, retry scenarios, missing instrumentation, duplicate delivery, clock skew, collector outages, backpressure, sampling decisions, tenant isolation, redaction, query paths, and retention transitions. I would verify that freshness, completeness, and bias indicators change correctly during those tests. That verifies not only that traces are collected, but also that operators understand when the evidence is trustworthy.

1. Define the tracing objective, trace-quality SLIs, required SLOs, retention needs, privacy requirements, tenant boundaries, and cost limits.
2. Propagate W3C trace context through synchronous calls, message headers, and background work; use span links when a simple parent-child relationship is not correct.
3. Instrument applications with OpenTelemetry and use supported boundary instrumentation for services that are not yet instrumented.
4. Export with OTLP through nearby collectors using batching, bounded memory, optional disk spooling, retry with backoff, and explicit backpressure behavior.
5. Authenticate telemetry, isolate tenants, validate schema, redact sensitive data, control cardinality, and deduplicate retry-created copies.
6. Apply head sampling for early volume control and tail sampling for outcome-aware retention.
7. Store compressed trace chunks in hot, warm, and cold tiers while maintaining selective indexes for trace ID and common query dimensions.
8. Provide trace search, latency analysis, service dependencies, critical-path analysis, span drill-down, and correlation with other telemetry.
9. Expose freshness, completeness, and bias indicators with query results.
10. Test synchronous, asynchronous, background, retry, duplicate, clock-skew, sampling, backpressure, privacy, collector-failure, and retention scenarios.

The main cost grows with the number of spans created. Without sampling, network traffic, collector processing, storage, and indexing all increase roughly with span volume. Tail sampling uses extra memory because collectors must hold partial traces until a decision is made. Disk buffering adds storage and I/O during outages. Indexing every attribute is expensive, so only useful fields with controlled cardinality should be indexed. Longer retention increases storage cost, which is why older traces move from hot to warm and cold tiers. Ongoing maintenance includes collector scaling, sampling rules, schema governance, tenant quotas, redaction policies, indexes, retention rules, and monitoring the tracing system itself.

Common mistakes are propagating context across HTTP but losing it in messages or jobs; treating wall-clock timestamps as perfect execution order despite clock skew; representing every asynchronous relationship as a parent-child span instead of using links when appropriate; using only head sampling and missing important failures; using tail sampling without sufficient buffering, routing, or a late-span policy; allowing collector queues to grow without bounds; retrying exports without handling duplicates; indexing uncontrolled high-cardinality attributes; storing secrets, tokens, or personal data in spans; treating proxy, auto-instrumentation, service-mesh, or eBPF observations as complete application instrumentation; failing to isolate tenants at ingestion and query time; keeping unnecessary data in expensive hot storage; and showing retained traces without exposing missing spans, instrumentation gaps, freshness, or sampling bias.

Explain the design as one flow: propagate context, create spans, collect safely, sample deliberately, protect data, store and index efficiently, query the traces, and measure the quality of the evidence. Spend extra time on asynchronous propagation, tail sampling, collector failure, backpressure, tenant and privacy controls, and freshness-completeness-bias because those details distinguish a production design from basic tracing.

I would model the dependencies as a directed graph and use Kahn’s topological sort with a min-heap. Each prerequisite points to the server that depends on it. I track each server’s in-degree, meaning its unfinished prerequisite count. Servers with in-degree zero enter the heap, so I always choose the lexicographically smallest eligible server. If I cannot process every server, the graph contains a cycle. The expected time is O(V + E + V log V), with O(V + E) auxiliary space.

We have a list of servers and dependency pairs. A pair [server, prerequisite] means the prerequisite must finish before that server can be patched. We must return every server exactly once. If several servers are ready at the same time, we choose the smallest identifier in lexicographic order. Bad identifiers, unknown servers, duplicate dependencies, self-dependencies, and cycles must raise ValueError. The solution uses a directed graph, an unfinished-prerequisite count for each server, and a min-heap containing the servers that are currently safe to patch.

1. Should each dependency always use the exact format [server, prerequisite]?
2. Should every invalid input, including a cycle, raise ValueError without returning a partial result?
3. Is lexicographic ordering required only among servers that are currently eligible?

The input contains 1 through 100,000 unique, non-empty ASCII server identifiers and dependency pairs. A dependency [server, prerequisite] means prerequisite must be patched and verified before server. The result must contain every server exactly once. When several servers are currently eligible, the lexicographically smallest one must be chosen. The function must not mutate the inputs. It must not write to stdout or stderr and must not perform file, network, retry, or timeout behavior.

I create an adjacency list with edges from each prerequisite to the servers that depend on it. I also store an in-degree for every server. In-degree means the number of prerequisites that are still unfinished. For the diagram example, db points to api, cache points to api, and api points to web. The initial in-degrees are api = 2, web = 1, db = 0, and cache = 0.

A server is eligible when its in-degree is zero. The example initially has cache and db eligible. Both go into a min-heap. Python heapq removes the lexicographically smallest string, so cache is selected before db.

Start with eligible heap [cache, db] and an empty result. Pop cache and append it. The result becomes [cache]. Processing cache satisfies one prerequisite of api, so api's in-degree changes from 2 to 1. Api is not ready yet. Next pop db. The result becomes [cache, db]. Api changes from 1 to 0, so api enters the heap. Pop api next. The result becomes [cache, db, api]. Web changes from 1 to 0, so web enters the heap. Finally pop web. The result becomes [cache, db, api, web]. All four servers have now been processed.

A server enters the heap only when its in-degree becomes zero. Therefore, every prerequisite for that server has already been processed. Every server we pop is safe to patch. The min-heap always exposes the lexicographically smallest currently eligible server, which gives the required deterministic tie-break. If fewer than V servers are processed, some servers could never reach in-degree zero. That means a directed cycle exists, so the function raises ValueError instead of returning a partial order.

The code first validates the server list, server count, identifiers, and uniqueness. It then validates each dependency before adding it to the graph. It rejects malformed dependency pairs, invalid identifiers, unknown servers, self-dependencies, and duplicate dependency pairs. It builds prerequisite-to-dependent adjacency information and in-degree counts. Next it heapifies all zero-in-degree servers. Kahn's loop repeatedly pops the smallest eligible server, appends it to the result, decreases the in-degree of each dependent, and pushes a dependent when its in-degree becomes zero. The final length check detects a cycle.

Let V be the number of servers and E be the number of dependencies. Building and validating the graph uses O(V + E) expected time under normal Python dictionary and set hashing behavior. Each edge is processed once. Each server enters and leaves the min-heap at most once, adding O(V log V). Total expected time is O(V + E + V log V). Auxiliary space is O(V + E). Important failure cases are an invalid server count, invalid identifier, unknown server reference, self-dependency, duplicate dependency pair, and cycle.

Use Kahn’s topological sort with a min-heap. Represent every dependency [server, prerequisite] as a directed edge prerequisite -> server. The in-degree of a server is its number of unfinished prerequisites. The central invariant is that every server in the heap has in-degree zero, so all of its prerequisites have already been processed. The heap exposes the lexicographically smallest eligible server. After processing a server, decrease the in-degree of each dependent. Push a dependent when its count reaches zero. If all V servers are processed, the result is a valid required order. If not, a cycle exists.

The function first checks that servers is a list containing from 1 through 100,000 entries. It validates every server as a non-empty ASCII string and rejects duplicate server identifiers. It creates an adjacency set for each server, an in-degree count initialized to zero, and a set used to detect duplicate dependency pairs. For each dependency, it checks that the value is exactly a two-element list, validates both identifiers, confirms both servers are known, rejects self-dependencies and duplicates, then adds the prerequisite -> server edge and increments the dependent server's in-degree. Next it places every zero-in-degree server in a min-heap. The loop repeatedly pops the lexicographically smallest eligible server, appends it to the result, and decreases the in-degree of each dependent. A dependent enters the heap when its in-degree reaches zero. If the final result does not contain every server, a cycle exists and ValueError is raised. Otherwise the complete deterministic patch order is returned. The runner uses the exact diagram example and checks it silently with an assertion.

One common mistake is reversing the graph edge and storing server -> prerequisite instead of prerequisite -> server. That gives the wrong in-degree meaning. Another mistake is using a plain queue, which can produce a valid topological order but does not guarantee the required lexicographic tie-break. Candidates may also forget to reject self-dependencies, duplicate dependency pairs, or unknown servers. Another serious error is returning the partial order when a cycle exists. Finally, claiming O(V + E) total time is incorrect for this implementation because the min-heap contributes O(V log V).

State the invariant before showing the loop: a server enters the min-heap only when its in-degree is zero, so all of its prerequisites are already complete. Then explain that heapq chooses the lexicographically smallest eligible server. This one explanation connects dependency safety, correctness, and deterministic tie-breaking.