A failed CI/CD pipeline is a diagnostic problem, not a signal to rerun every job. This guide provides a stage-by-stage troubleshooting workflow for locating the first meaningful failure, separating infrastructure issues from code defects, checking logs and artifacts, handling deployment health problems, and deciding when a rollback is safer than a forward fix.
Overview
CI/CD pipeline troubleshooting becomes slower when teams treat the final red job as the root cause. A later step may fail only because an earlier step produced no artifact, installed the wrong dependency version, or never completed successfully. The reliable approach is to reconstruct what happened in order: identify the first failing stage, capture the relevant evidence, form one testable hypothesis, and rerun only the smallest useful unit.
The exact labels differ between GitHub Actions, GitLab CI, Jenkins, and cloud delivery systems, but most pipelines contain similar stages:
- Checkout: retrieve the intended commit, branch, tag, or pull request reference.
- Environment setup: select the runner image, language runtime, operating system, and required services.
- Dependency installation: resolve and install packages using the expected lockfile and registry credentials.
- Build and validation: compile, lint, format, scan, and run unit or integration tests.
- Artifact handling: package, upload, download, and verify the files needed by later jobs.
- Deployment: authenticate to the target system and apply the release.
- Health checks and release control: confirm that the new version is usable before promoting or completing the release.
Start by recording the commit or release identifier, pipeline run ID, failing job, runner type, environment, and time of failure. This prevents a later retry from obscuring the original evidence. Avoid changing several variables at once: if you update the runner, clear the cache, edit the test command, and rerun the pipeline together, you may get a green result without learning which change mattered.
What to track
Use the first-failure rule
Read the pipeline from the beginning and find the earliest step that did not complete as expected. Distinguish between an explicit error and a cascade. For example, a deployment job that reports “file not found” may simply be consuming an artifact that the build job never produced. The build failure is the primary investigation point.
Failure symptoms and likely causes
| Symptom | Likely causes to check | Useful next action |
|---|---|---|
| Checkout fails | Invalid ref, missing commit, inaccessible repository, submodule or shallow-clone issue | Confirm the exact revision and repository permissions; inspect submodule configuration. |
| Dependency installation fails | Lockfile drift, unavailable registry, expired token, incompatible runtime, network or proxy failure | Compare the runtime and lockfile with local or known-good runs; verify registry access without exposing credentials. |
| Build fails immediately | Missing environment variable, wrong toolchain, unsupported compiler, malformed configuration | Print safe environment metadata, not secret values, and confirm the expected tool versions. |
| Tests fail consistently | Real regression, changed fixture, schema mismatch, time-zone assumption, service or database setup error | Run the smallest failing test with the same inputs and dependencies used by CI. |
| Tests fail intermittently | Race condition, shared state, timing dependency, resource contention, external service instability | Record frequency, affected tests, runner details, and order dependence before retrying. |
| Artifact is missing or invalid | Wrong path, upload condition skipped, retention issue, naming mismatch, incomplete build | List the artifact contents and verify the producer and consumer use the same name and path. |
| Authentication or permission fails | Wrong identity, expired credential, missing scope, protected environment, branch restriction | Identify the principal and required permission; rotate or replace credentials through the approved secret store. |
| Deployment applies but health checks fail | Bad configuration, incompatible migration, readiness failure, image issue, traffic or dependency problem | Compare the new version with the last healthy version and inspect application and platform events. |
| Rollback is blocked | Irreversible migration, unavailable previous artifact, changed configuration, incomplete release metadata | Stop further promotion and use the documented recovery or forward-fix procedure. |
Inspect logs as evidence
Read a small window around the first error, then expand outward to find the command, inputs, and preceding warning that explain it. Capture:
- the full command or script that ran, including its working directory;
- runtime, operating system, runner image, and tool versions;
- the commit, dependency lockfile, and configuration revision;
- exit codes, timestamps, retry counts, and timeout values;
- service names, artifact paths, and deployment target identifiers;
- the distinction between a redacted secret reference and the secret value itself.
Do not paste tokens, private keys, signed URLs, customer data, or complete authorization headers into tickets or chat. If a credential may have been printed, treat exposure as an incident: preserve the log reference, revoke or rotate the credential according to your team’s process, and limit further distribution.
Separate code, environment, and platform causes
A useful triage question is whether the same revision fails in the same way on a controlled rerun. A deterministic failure usually points toward code, configuration, dependencies, or a reproducible environment mismatch. A failure that disappears without any code change may indicate cache corruption, a transient service problem, runner capacity, network instability, or a flaky test. That does not make it harmless; record it and look for recurrence.
For dependency problems, verify that the pipeline uses the intended lockfile and fails when the lockfile is inconsistent. For caches, temporarily bypass or invalidate the cache as a diagnostic test, then fix the cache key rather than leaving caching disabled indefinitely. A useful cache key should change when relevant dependency manifests, runtime versions, or build inputs change.
Check secrets, permissions, and artifacts deliberately
Authentication failures often appear as generic network or authorization errors. Identify which identity the job uses, what resource it is accessing, and whether the request comes from a permitted branch, environment, or deployment context. Check scopes and expiration without printing secret values. Prefer short-lived, narrowly scoped credentials where the platform and operating model support them.
For artifacts, inspect both sides of the handoff. The producer should create the expected file, upload it under a predictable name, and fail clearly when it is absent. The consumer should verify the download, inspect the file type and contents, and confirm that it corresponds to the intended commit. Packaging errors are easier to diagnose when artifact metadata includes a version or revision identifier.
Investigate deployments and decide on rollback
When a deployment fails, first determine whether the release was rejected before application, partially applied, or accepted but unhealthy. Check deployment events, application logs, readiness and liveness results, configuration changes, database migrations, and dependency connectivity. A successful command is not proof that the service is healthy.
Rollback is generally easier when releases are immutable, previous artifacts remain available, configuration is versioned, and database changes are backward-compatible. If a migration cannot safely be reversed, a forward fix or a controlled traffic reduction may be safer than restoring only the application binary. Use the release runbook rather than improvising under pressure.
Cadence and checkpoints
Troubleshooting should happen during the incident, but pipeline reliability also benefits from scheduled review. Use the following checkpoints as a lightweight recurring practice:
- Every failure: preserve the original run, identify the first failure, classify the cause, and link the fix or follow-up.
- Weekly or per release cycle: review repeated failures, manual reruns, timeout changes, flaky tests, and jobs that require special knowledge.
- Monthly: inspect runner image changes, dependency and toolchain updates, cache behavior, secret rotation paths, artifact retention, and deployment recovery steps.
- Quarterly: test rollback or recovery in an appropriate non-production environment, verify ownership, remove obsolete jobs, and review whether pipeline stages still match the delivery workflow.
- After a platform change: recheck authentication, permissions, runner behavior, artifact handling, webhook triggers, and deployment integrations before relying on the pipeline for a critical release.
Keep a simple pipeline reliability record with the run date, service, revision, stage, symptom, suspected cause, confirmed cause, action, owner, and follow-up date. Teams that track delivery performance can connect this record with their broader review of deployment frequency, lead time, recovery time, and change failure patterns; the DORA metrics implementation guide provides a framework for organizing that review.
Pipeline ownership template
Every production pipeline should have an explicit owner, even when several teams contribute to it. Record:
- Service and repository: the code and delivery target covered.
- Primary owner: the team responsible for maintaining the workflow.
- Backup owner: the escalation contact when the primary team is unavailable.
- Build and deployment dependencies: runners, registries, cloud accounts, clusters, databases, and external services.
- Recovery options: retry conditions, rollback procedure, feature-flag mitigation, and forward-fix path.
- Evidence locations: logs, artifacts, dashboards, runbooks, and incident records.
- Review date: the next scheduled check and the events that require an earlier review.
For Kubernetes-based delivery, document how manifests are generated and promoted rather than treating generated YAML as the only source of truth. The comparison of Helm, Kustomize, Jsonnet, and raw YAML can help teams clarify that responsibility.
How to interpret changes
A single failure deserves a focused investigation; a pattern deserves a system change. Group failures by stage, service, runner, branch, dependency, and deployment target. Then ask whether the frequency changed after a specific event, such as a runner image update, dependency refresh, cache change, permissions edit, or new test suite.
Interpret the signals carefully:
- More checkout failures may indicate trigger, repository, or access changes rather than application defects.
- More installation failures may point to lockfile drift, registry availability, runtime changes, or credential problems.
- More test failures after a code change may be a real regression, but a simultaneous runner or service change should remain in scope.
- More retries with eventual success can hide flaky tests and infrastructure instability; measure the retry rate instead of counting only successful pipelines.
- More deployment health failures may reflect configuration or capacity changes even when the build remains green.
- Longer duration without more failures still affects delivery flow and may reveal cache misses, serial jobs, oversized artifacts, or inefficient test setup.
When a fix is proposed, state the hypothesis and the evidence that would confirm it. For example: “The dependency job fails because the runner uses a runtime older than the lockfile requires; confirmation is a matching failure on the affected image and a successful run on the supported image.” This approach makes reruns informative rather than habitual.
For recurring test instability, assign an owner and a deadline. Quarantine can be appropriate when it prevents a known flaky test from blocking unrelated delivery, but it should preserve visibility and include a removal condition. Do not convert a real product failure into a tolerated retry simply because the pipeline is under release pressure.
Incident handoff workflow
- Stabilize: pause promotion if the release state is uncertain and identify whether production impact exists.
- Summarize: share the revision, first failing stage, exact error, timeline, affected environment, and actions already attempted.
- Assign: name one incident owner and separate technical investigators from the person coordinating communication.
- Choose a path: retry a proven transient step, fix forward, roll back, or apply a documented mitigation.
- Close the loop: record the confirmed cause, recovery result, prevention task, owner, and due date.
When to revisit
Return to this guide whenever a build fails, a runner image or runtime changes, a dependency registry or authentication method changes, a deployment is blocked, a flaky job recurs, or a release retrospective identifies unexplained manual work. Revisit it on a monthly or quarterly cadence even when pipelines are green: green runs can conceal retries, skipped checks, stale artifacts, or an owner who has moved teams.
Update the team’s runbook after every confirmed failure that required non-obvious investigation. Refresh command examples for the CI/CD systems and cloud platforms you actually use, and verify that log locations, permission models, secret-handling instructions, and rollback steps still match the current environment. Review security guidance whenever credential storage, token lifetimes, or platform authentication practices change.
For the next pipeline failure, use this compact checklist:
- Save the original run and record the revision, environment, and failing job.
- Find the first meaningful failure, not merely the last red step.
- Inspect the command, inputs, versions, exit code, and surrounding log lines.
- Classify the issue as code, dependency, cache, runner, permission, artifact, deployment, or flaky behavior.
- Test one hypothesis with the smallest safe rerun.
- Protect secrets and preserve only the evidence needed for diagnosis.
- Decide whether to retry, fix forward, mitigate, or roll back using the release runbook.
- Record the confirmed cause, owner, prevention task, and review date.
Use this workflow as a living part of delivery operations, not as a document consulted only during emergencies. Clear evidence, explicit ownership, and scheduled review turn failed builds into actionable feedback about the health of the software engineering system.