189 DevOps Engineer Interview Questions & Answers

CI/CD helps teams integrate code often, test changes quickly, and deliver software through a controlled and repeatable process. Continuous Integration builds and tests small changes so problems are found early. After the checks pass, the pipeline creates one versioned and traceable artifact and promotes that same artifact through staging and quality gates. Continuous Delivery keeps the software ready for production but can require a human release decision. Continuous Deployment goes further and automatically releases a change to production after all required gates pass.

CI/CD gives a team a safer and more repeatable way to move a software change from a developer to users. Instead of waiting for many large changes and combining them near release time, developers send small changes often. Automatic checks quickly show whether a change works. After the checks pass, the team keeps one identified package and moves that same package through later checks. This reduces late surprises, repeated manual work, and uncertainty about what was tested or released. It also makes the release process easier to repeat and understand.

1. Does your team require a human approval before production releases?
2. Does your pipeline promote the same built artifact through staging and production?
3. Which automated checks must pass before a change can reach production?

Continuous Integration means developers integrate small changes frequently. A commit or pull request triggers the pipeline. The pipeline builds the application and runs automated checks such as unit tests, integration tests, static analysis, and security scans. If a required test or check fails, the pipeline stops that change and sends fast feedback to the developer. The failed change does not continue toward release.

When the required CI checks pass, the pipeline creates one versioned, immutable, and traceable artifact. That artifact is tied to the commit or build identity. The important production practice is to promote the same artifact instead of rebuilding different packages for different environments. This makes it easier to know that the software tested in staging is the same software being considered for production.

The artifact is deployed to staging. Additional quality gates can run there, such as smoke tests, end to end tests, performance tests, and approval checks. Controlled promotion means the artifact moves forward only after the required conditions pass.

Continuous Delivery keeps the software in a releasable state. After the required gates pass, a human can approve the production release. Continuous Deployment removes that manual production decision. A change that passes the required gates is released to production automatically.

CI/CD solves two major delivery problems shown in the workflow. First, large late integrations create conflicts and slow feedback. Small frequent integration finds problems earlier. Second, manual and inconsistent release steps are slow and error prone. Automated checks, traceable artifacts, controlled promotion, and deployment automation make the process more repeatable. The main limitation is that automation is only as useful as its checks. Weak tests or poor gates can still allow bad changes through, so production teams also need monitoring and a recovery plan.

A common mistake is saying CI only means running tests. CI also includes integrating small changes frequently and receiving automated build and test feedback. Another mistake is rebuilding the application separately for each environment instead of promoting the same tested artifact. Teams also confuse Continuous Delivery with Continuous Deployment. Continuous Delivery keeps software ready to release while allowing a human production decision. Continuous Deployment automatically releases changes that pass the required gates. Another mistake is allowing failed checks to continue through the pipeline instead of stopping the change and returning fast feedback to the developer.

Explain CI/CD as one flow from a small code change to production. Start with frequent integration and fast automated feedback. Then explain the versioned traceable artifact and why the same artifact is promoted through staging and quality gates. Finish with the key distinction: Continuous Delivery keeps software ready for production with a human release decision, while Continuous Deployment automatically releases changes that pass the required gates.

Continuous integration automatically integrates code changes and validates them with builds and tests. Continuous delivery extends that process so the application is always ready to release, but production deployment can still require a human approval or manual action. Continuous deployment goes one step further and automatically sends every change that successfully passes the required pipeline checks to production. The main difference is therefore how far automation continues toward production.

The easiest way to compare these practices is to ask where automation stops. Continuous integration focuses on combining code changes and checking that they work. Continuous delivery continues until the software is prepared and ready for production, but a person can still decide when the production release happens. Continuous deployment continues through the production step automatically. All three can share early pipeline steps such as merging code, building it, testing it, and creating a versioned artifact. The important difference is the final release boundary and whether a human decision remains.

1. Are you asking about the normal conceptual difference or a specific pipeline implementation?
2. Does your delivery process require human approval before production?

I would describe the three practices as increasing levels of automation.

With continuous integration, developers frequently merge changes into the main code line. The pipeline automatically builds the code and runs automated tests such as unit and integration tests. It may also create an artifact. The main goal is fast feedback so integration problems are found early. Production deployment is outside the main goal of continuous integration.

Continuous delivery extends that workflow. After the required checks pass, the pipeline creates a versioned package or artifact and can validate it in a staging or preproduction environment. The software stays ready for release. A production deployment may still wait for a human approval. In the diagram example, the production environment has an approval gate before the deployment job runs.

Continuous deployment removes that normal manual gate. If a change passes every required automated check, the pipeline automatically deploys it to production.

The main tradeoff is control versus speed. Continuous delivery keeps a deliberate production decision. Continuous deployment reduces waiting and manual work. In either model, a failed required check stops that change from progressing. If production has a problem, the team still needs monitoring and a safe recovery method such as rollback or roll forward. Automation does not remove operational controls.

A common mistake is treating continuous delivery and continuous deployment as the same thing. Delivery means the software is kept ready to release, while deployment means a passing change normally reaches production automatically. Another mistake is saying continuous integration automatically deploys to production. Its main purpose is integration and fast automated validation. Teams also sometimes assume continuous deployment means there are no controls. In practice, automated tests, security checks, deployment validation, monitoring, and recovery controls become even more important because there is no normal manual production gate.

Explain the practices as one pipeline with three stopping points. Say that continuous integration focuses on integration and validation, continuous delivery keeps the software ready for production but can keep a human approval, and continuous deployment automatically continues through production after the required checks pass. This makes the difference easy to remember and directly answers where automation stops.

Version control is essential because it gives the CI/CD workflow one shared history and a specific commit to build, test, package, and deploy. Branches and pull requests support review before changes reach the main branch. A push or merge can trigger the pipeline for that exact commit. The commit identifier also links builds and deployments back to the source change, which improves traceability and lets the team redeploy a known good previous commit when rollback is needed.

Version control gives the delivery process one trusted history of the code. Team members can work on separate changes, review them before they become part of the main code, and identify every accepted change with a unique value. Automation can start when an accepted change reaches the main code. The same accepted version can then move through checking, building, testing, packaging, and deployment. This makes it easier to understand what was released, who changed it, and which earlier version should be restored if a release causes a problem.

1. Does the team trigger delivery when code is pushed to the main branch, when a pull request is merged, or through another release event?
2. Are build artifacts promoted between environments, or does each environment rebuild from source?
3. How does the team record which commit is running in each environment?

I would start with the shared repository. It gives the team one shared commit history instead of separate copies with no common record. Developers can create branches for isolated work and use pull requests for discussion, automated checks, and code review before merging.

After a change is accepted, the commit gets a unique identifier. In the diagram, the accepted commit is c0d1e2f. A push or merge to the main branch can trigger the CI/CD pipeline. The pipeline checks out that exact commit, then builds it, runs automated tests, creates an artifact, and deploys it to an environment.

The commit identifier is important because it connects the source change to the pipeline run, artifact, and deployment. That gives the team traceability. They can answer what changed, when it changed, and which source version produced a release.

The same commit also gives the pipeline the same source code. That alone does not guarantee identical build output. Reproducible output also needs pinned dependencies and a controlled build environment.

For rollback, the team can redeploy a known good previous commit or, when available, the known good artifact produced from that commit. This is safer than manually guessing which source files should be changed during an incident.

The practical value is that version control connects review, automation, traceability, and recovery. It becomes the source reference that the rest of the release workflow can follow.

A common mistake is saying that version control alone makes a build reproducible. A commit pins the source, but dependencies and the build environment must also be controlled. Another mistake is building from a moving branch name without recording the actual commit identifier. Teams can also lose traceability if deployments are not linked back to commits. For rollback, another mistake is choosing an arbitrary older commit instead of a verified known good version.

Explain the release flow in order. Start with the shared repository, then branches and pull requests, then the commit identifier, then the pipeline trigger, and finally build, test, packaging, deployment, traceability, and rollback. Mention that a commit pins the source but does not by itself guarantee identical build output.

I would use six main stages: source, build, automated test, artifact, deployment, and verification. Each stage should have a clear exit condition. Work moves forward only when the current stage completes successfully and its required checks pass. If a gate fails, I stop the pipeline and notify the team instead of automatically moving forward. I would also store a versioned artifact so the same tested output can be deployed rather than rebuilding it for each environment.

A basic delivery process should move a code change through a clear series of checks before people depend on it. First, the change is saved in the shared code repository. Next, the system creates the software, checks that it works, stores the exact result, places that result in the chosen environment, and finally confirms that the release is healthy. Each step has a condition that must succeed before the next step starts. If a required check fails, the process stops and the team is notified instead of continuing with a bad release.

1. Does the team deploy automatically to production, or is approval required before production?
2. Which automated checks are required before a release can move forward?
3. Does the same stored artifact move through staging and production?

I would describe the pipeline as six ordered stages with a gate after each stage.

1. Source. A developer commits and pushes code to the Git repository. A repository event can trigger the pipeline. The source gate confirms that the expected change is available and repository rules are satisfied.

2. Build. The pipeline restores dependencies and compiles or packages the application. The build gate passes only when the build finishes successfully and required build quality checks pass.

3. Automated test. The pipeline runs appropriate unit tests, integration tests, linting, static analysis, and security checks. The gate passes only when the required tests and quality policies succeed.

4. Artifact. The successful build output is packaged and published to an artifact repository. Examples include a JAR file, package, or container image. The stored artifact should be versioned and traceable to the source change. I prefer promoting the same tested artifact instead of rebuilding it for each environment.

5. Deployment. Automation deploys that artifact to the target environment such as staging or production. Infrastructure configuration, application configuration, and secrets are handled by their appropriate systems. The deployment gate confirms that the deployment completed successfully.

6. Verification. The pipeline runs smoke tests, health checks, and other release checks. Logs, metrics, traces, and alerts help operators observe the deployed system. Verification proves that deployment completion alone is not enough. The service must also meet its expected health criteria.

If any required gate fails, I stop automatic promotion, notify the team, correct the problem, and run the appropriate pipeline work again. Production pipelines may also include approvals, progressive delivery, or rollback rules, but those are policy choices around the same core flow.

Common mistakes include treating deployment as the final proof of success, rebuilding the application separately for every environment, allowing failed tests or scans to be ignored, and storing artifacts without a version or source reference. Another mistake is mixing responsibilities. Git stores source history, the CI system coordinates pipeline work, the artifact repository stores build outputs, deployment automation changes the target environment, and observability tools report runtime signals. A good pipeline keeps these responsibilities clear and blocks automatic promotion when a required gate fails.

Explain the stages in order and give one sentence for the purpose of each stage. Then emphasize the gate rule: the next stage starts only after the current stage and its required checks succeed. Mention that a failed gate stops automatic promotion and that the same tested artifact should normally be promoted rather than rebuilt. Finish by saying that verification checks the real deployed service, not just whether the deployment command completed.

I choose rolling when I want low extra capacity and the change is small enough to tolerate old and new versions running together. I choose blue green when very fast rollback matters and I can afford two complete environments. I choose canary when change risk is higher and I have strong production signals plus fine grained traffic control. Database changes must remain compatible with every application version that can still serve traffic or be restored during rollback.

Rolling, blue green, and canary are three ways to move a stateless service from an old version to a new version. The choice depends on how much spare capacity is available, how carefully traffic must be controlled, how quickly the team must return to the old version, whether both application versions can safely use the database, and how much trusted production feedback is available. Rolling changes replicas gradually. Blue green keeps two complete environments and switches traffic. Canary exposes a small amount of real traffic first, then increases it only when results remain healthy.

1. How much spare capacity can the service use during a release?
2. Do we have traffic controls that can route a percentage or user segment?
3. How fast must rollback be if the new version is unhealthy?
4. Will the release include database schema changes?
5. Which production signals are trusted enough to decide whether exposure should increase?

For a stateless service, rolling deployment is usually the simplest choice when capacity is limited and the change has low or moderate risk. The orchestrator gradually replaces old replicas with new replicas. This normally needs only a small amount of extra capacity. Traffic control is limited because users can reach both versions while the rollout is in progress. Rollback is slower than a traffic switch because replicas must be changed back. Database changes must stay compatible while both application versions are active.

Blue green deployment keeps the current version in one full environment and the new version in another. After the new environment is validated, traffic moves to it at once. The main benefit is very fast application rollback because traffic can be sent back to the old environment. The main cost is high capacity because both environments exist at the same time. Database changes still need careful sequencing because the old environment may need to work again after rollback.

Canary deployment sends a small share of traffic to the new version, such as one percent, then increases exposure in stages. It is useful when the change is risky or uncertain. It needs fine grained routing and strong production signals such as errors, latency, service objectives, business indicators, logs, alerts, and anomaly detection. If those signals degrade, the release should stop increasing exposure and the canary can be reduced to zero. This limits blast radius but requires stronger observability and more operational discipline.

Common mistakes are treating rolling as an instant rollback method, assuming blue green removes the need for database compatibility planning, and using canary without reliable traffic control or production signals. Another mistake is increasing canary exposure only because basic health checks pass. Health checks show availability, but a canary decision should also consider user facing errors, latency, service objectives, business indicators, logs, and alerts. It is also a mistake to assume that switching application traffic automatically reverses a database change.

Compare the three strategies using the same five points: capacity, traffic control, rollback speed, database compatibility, and production signals. Then give a practical choice. Rolling favors low overhead, blue green favors very fast application rollback, and canary favors controlled risk with strong observability.

I would prepare rollback before deployment. Every release would have an immutable record containing the exact code artifact, image digest, configuration version, database migration identifier, feature flags, compatibility information, status, and previous known good release. I would deploy progressively, watch error rate, latency, health checks, saturation, business metrics, and database errors, and stop promotion when defined conditions remain unhealthy. The rollback process would select the most recent successful release with verified artifacts and a compatible database schema. I would restore code and configuration, change the database only when a tested reversal is safe and lossless, restore traffic, and verify service and data health before closing the incident.

A reliable rollback plan means deciding how to return to a working version before a new release begins. I would keep an exact record of each release and keep the previous working version available. I would first send only a small amount of traffic to the new version and watch whether the service remains healthy. If important checks stay bad, promotion stops. The system then selects a known working version, restores traffic to it, and confirms that normal service and data behavior have returned before the incident is closed.

1. Do you use canary or another progressive release method?
2. Which health signals should trigger automatic rollback?
3. Must the previous application version work with the current database schema?
4. Are application and configuration artifacts stored as immutable versions?

I would store release metadata such as the release identifier, source commit, code version, image digest, configuration version, database migration identifier, feature flags, artifact references, compatibility information, deployment status, and rollback target. The artifact store must keep the exact application package, configuration bundle, and migration scripts used for that release.

I would deploy progressively. For example, I could start with a 5 percent canary and promote it only while health remains acceptable. I would watch request errors, p95 and p99 latency, request success, CPU and memory saturation, queue depth, business metrics, synthetic checks, health checks, and database errors. Automatic rollback should require a monitored failure condition to persist for a defined period. An operator can still trigger rollback immediately.

When rollback starts, I would stop further rollout, drain work when required, and select the most recent known good release with status SUCCESS. It must have passed required health checks, have all required artifacts available and verified, and remain compatible with the current database schema. I would prefer a target from the same major minor version line when appropriate.

For database safety, I would use expand and contract migrations. Expansion should use compatible changes such as nullable columns. Destructive cleanup happens only after every application instance uses the new version. A down migration runs only when it is tested, safe, and lossless. Otherwise the compatible expanded schema stays in place.

After restoring code, configuration, and traffic, I would verify health checks, error rate, latency, business metrics, database connectivity, synthetic tests, and important user flows. I would then invalidate caches or clear queues only when required, resume background jobs, mark the release as rolled back, record the timeline, and update the runbook.

Common mistakes include rebuilding an older version instead of using the exact stored artifact, restoring application code while leaving incompatible configuration behind, choosing a rollback target only by version number, and ignoring database compatibility. Another mistake is triggering rollback from a short metric spike instead of using a sustained failure condition. Teams may also assume every database migration can be reversed or restore a database snapshot even when that would discard valid production writes. Finally, a rollback is incomplete if the team restores traffic but skips health checks, business metrics, database validation, synthetic tests, user flow verification, incident records, or runbook updates.

Explain rollback as a prepared production workflow rather than a command that deploys an older version. Walk through release metadata, health signals, the trigger, target selection, database compatibility, execution, and verification in that order. Emphasize that code and configuration are normally restored from exact stored artifacts, while database recovery needs stricter safety checks. Finish by saying that rollback is complete only after service health, data behavior, and important user flows are verified.

I would build and test the application once, store that immutable artifact, and promote the same artifact through development, staging, and production. I would keep normal values such as endpoints, feature settings, log levels, and timeouts in external configuration such as a ConfigMap. I would keep passwords, API keys, and tokens in a separate secret system. At startup, the application loads both sources and validates every required value. If validation fails, it should fail fast instead of starting with incomplete or unsafe configuration.

I would create the application package only once and use that same tested package in development, staging, and production. Each environment would supply its own settings when the application starts. Normal settings and sensitive values would be stored separately. This means changing an endpoint, feature setting, password, or token does not require rebuilding the application package. Before the application starts serving traffic, it checks that every required value exists and is valid. If an important value is missing or incorrect, startup stops immediately so the problem is found during deployment instead of later while users are using the service.

1. Are the applications running in Kubernetes or another deployment platform?
2. What system is used to store and deliver secrets?
3. Do configuration changes require approval, version history, or an audit record?

I would use a build once and configure at runtime model. Source code goes through checkout, testing, build, and packaging in the CI pipeline. The result is one immutable application artifact, such as app.jar. That artifact is stored in an artifact repository and promoted unchanged to development, staging, and production.

For normal environment values, I would use external configuration. In Kubernetes, a ConfigMap is a common choice. Each environment can have its own API endpoint, feature setting, log level, timeout, or other safe value without changing the artifact.

Secrets are kept separately. Passwords, API keys, tokens, and similar values can be supplied through Kubernetes Secrets or a managed secret system. Secrets should not be committed to Git or packaged inside the application artifact. Access should follow least privilege.

When the application Pod starts, the application loads its ConfigMap values and secrets, then validates every required setting. If all required values are valid, initialization continues and the application starts. If a required value is missing or invalid, the application fails fast and does not begin serving traffic.

The main benefit is that the exact tested artifact moves through every environment. This makes promotion easier to trace and avoids environment specific rebuilds. The operational tradeoff is that configuration and secret changes become important deployment inputs, so they need controlled access, validation, and useful change history.

Common mistakes include putting production settings directly inside the application artifact, building a different artifact for each environment, storing secrets in Git, treating secrets like normal configuration, and starting the application before required values are validated. Another mistake is rebuilding during promotion, because production would then run a different artifact from the one that was already tested. Teams should also avoid giving broad access to secrets or changing configuration without validation and useful change history.

Start with the main rule: build once and promote the same immutable artifact. Then explain three boundaries clearly. Normal configuration stays outside the artifact, secrets are stored separately with restricted access, and startup validation stops the application when required values are missing or invalid.

I would detect which services and shared dependencies changed, calculate the affected services, and start CI only for those services. Each affected service gets its own pipeline for build, tests, security checks, contract tests, packaging, and publishing. I would publish immutable versioned images and metadata once, then promote those same artifacts through environments with GitOps. Independent services can deploy in parallel. When compatibility requires ordering, I would make an additive provider or schema change first, deploy consumers next, and remove the old contract later. Production uses canary or blue green delivery with health and service level gates. If a release fails, I would normally roll back only the failing service when contracts remain compatible, and coordinate dependent services only when required.

The main goal is to let every service move at its own speed without rebuilding or redeploying software that did not change. When code is pushed, the delivery system first finds which service folders, shared libraries, contracts, or database dependencies changed. It then starts work only for services affected by those changes. Each affected service is tested and packaged separately. The exact package that passes testing is stored and later promoted through development, staging, and production. Releases are introduced gradually and watched closely. A failure should normally affect only the service that caused it.

1. Is this one repository containing all services and shared libraries?
2. Which dependencies are shared between services?
3. Are API and database changes required to remain backward compatible?
4. Is production deployment automatic or approval based?
5. What service level signals should stop a progressive rollout?

I would start with change detection. A CI orchestrator such as GitHub Actions reads changed paths and combines that information with dependency sources such as package manifests, import relationships, API contracts, and database migration dependencies. The result is the set of affected services. A shared library change can therefore trigger its dependent services, while an isolated service change triggers only that service.

Each affected service runs its own CI pipeline. Typical stages are checkout, build, unit tests, image build, security scanning, contract testing, packaging, and publishing. Quality gates stop the pipeline when the build, tests, coverage, security checks, contract checks, performance budget, or policy checks fail.

Successful pipelines publish immutable versioned container images and other artifacts. I also publish provenance, an SBOM, and signatures. I build once and promote the same tested artifact instead of rebuilding it for each environment.

For delivery, GitOps with Argo CD can reconcile each service independently into Kubernetes. Development and staging can be automatic, while production uses progressive delivery. A canary can move through 5 percent, 25 percent, 50 percent, and then 100 percent traffic. Blue green delivery and feature flags are also useful controls.

Deployment ordering depends on compatibility. Independent services deploy in parallel. When ordering is required, I first introduce an additive provider or schema change, then deploy consumers, and remove the old contract only after consumers have moved.

Metrics, logs, traces, alerts, service level objectives, and error budgets provide feedback. If a release fails, I roll back the failing service to its prior immutable version when contracts remain compatible. I coordinate dependent services only when compatibility requires it. Database changes stay backward compatible, so database recovery normally uses a roll forward fix rather than reversing a destructive migration.

A common mistake is rebuilding every service after every repository change, which removes much of the value of independent deployment. Another mistake is detecting only changed folders and ignoring shared library, API contract, or database dependencies. Teams also make releases harder to reproduce when they rebuild an artifact for each environment instead of promoting one immutable artifact. Other mistakes include using mutable image tags, skipping contract tests, removing an old API contract before consumers have migrated, deploying incompatible provider and consumer changes in the wrong order, treating readiness checks as complete evidence of application health, and automatically reversing database migrations that are not safely reversible. Another mistake is rolling back several healthy services when only one service failed and contracts are still compatible.

Explain the design as one continuous flow: detect impact, test only affected services, build immutable artifacts once, verify compatibility, promote each service independently, release progressively, observe production, and recover with the smallest safe scope. Clearly call out that dependency changes can expand the affected service set and that compatibility determines deployment ordering and rollback scope.

I would build the application once in CI, publish the immutable container image to an OCI registry, and keep the desired deployment state in Git. I would maintain separate configuration for dev, staging, and production. Promotion would happen through reviewed pull requests that update the image reference. An in cluster GitOps agent such as Argo CD or Flux would read its environment repository, compare desired state with the Kubernetes cluster, and reconcile differences. Secrets would remain in an external secret store. CI credentials and runtime credentials would stay separate and use least privilege.

The main idea is to separate building the application from deciding what should run in each environment. CI creates the application package once and stores it in a registry. Git records which version should run in dev, staging, and production. Changes move forward only after checks, review, and approval. Each environment regularly compares the approved Git state with the running system. If someone changes the running system directly, the approved state can restore it. Sensitive values stay outside Git, and separate identities are used for build work and runtime access.

1. Do you want a separate repository for each environment, or separate environment paths in one repository?
2. Should production promotion require manual approval or a change window?
3. Which external secret store and workload identity system are available?
4. Should drift always be corrected automatically, or should some resources only create an alert?

I would separate CI from deployment control. CI handles linting, tests, build, security scanning, packaging, image publishing, and software bill of materials generation. It publishes an immutable OCI image to the registry. CI can update the image tag or digest in Git through a pull request, but it does not need Kubernetes deployment credentials.

Desired state lives in environment Git repositories for dev, staging, and production. Each environment contains application manifests, its environment overlay, and secret references. Promotion uses a controlled pull request flow. Automated policy, lint, and security checks run first. A human reviews the change, then the approved image reference moves from dev to staging and finally to production.

Each Kubernetes cluster runs its own GitOps agent, such as Argo CD or Flux. The agent reads its environment repository, compares desired state with live state, and synchronizes required changes through the Kubernetes API. If live state differs from Git, the agent can restore the approved state when automatic drift correction is allowed.

Secret values are not stored in Git. Git keeps references used by an external secrets integration. The runtime identity reads only the required secret values from a service such as AWS Secrets Manager, Google Cloud Secret Manager, or HashiCorp Vault.

For rollback, I would revert the Git commit or restore the previous image reference. The agent then reconciles the cluster back to that known good state. Readiness, liveness, and smoke checks help detect bad deployments. If reconciliation fails, the controller retries with backoff, exposes status and events, sends alerts, and allows controlled manual intervention. Prometheus metrics, logs, traces, alerts, dashboards, and Git history provide operational visibility and audit history.

Common mistakes include letting CI deploy directly to production while also claiming Git is the deployment source of truth, rebuilding the application during every promotion instead of promoting the same tested artifact, storing plaintext secrets in Git, giving the CI identity broad Kubernetes access, sharing one powerful credential between CI and runtime controllers, changing production manually without updating Git, treating a retry as a rollback, and assuming every reconciliation failure repairs itself automatically. Another mistake is rolling back only the live cluster while leaving Git on the bad version, because the controller can apply that bad desired state again.

Explain the design as one clear flow. Start with build once, then publish the artifact, then store desired state in Git, then promote through reviewed Git changes, then let one controller per cluster reconcile its environment. After that, cover drift correction, secret references, rollback, failure handling, credential separation, and observability. Make the boundary between CI credentials and runtime credentials explicit because it is an important security decision.