Building an ECS Blue/Green Deployment Pipeline

A container starting successfully is not the same thing as a deployment succeeding. That distinction shaped my ECS blue/green project: a URL shortener running on AWS, surrounded by the infrastructure, pipeline and operational controls needed to release changes more safely.

The application gave me something concrete to deploy. The engineering work was in what happened around it: provisioning the environment with Terraform, building and scanning images, authenticating GitHub Actions with AWS, shifting traffic and deciding what should happen when a new version behaved badly.

The project became less about getting containers to run and more about understanding the systems responsible for delivering changes safely.

Why an ordinary container update wasn't enough

The simplest deployment flow builds a new Docker image, pushes it to a registry and updates the running ECS service. ECS starts tasks using the replacement image and manages their lifecycle.

That is useful automation, but it does not tell you whether the new release is safe for real traffic.

An application can pass a health check while its core business operation is broken. For a URL shortener, a basic HTTP endpoint might respond normally while link creation or redirect resolution fails.

Health checks, application tests and production metrics answer different questions. None should automatically be treated as a substitute for the others.

I wanted the release process to recognise that difference. Instead of equating task readiness with success, the design needed a way to expose the replacement version to traffic, observe it and retain a path back to the previous version.

The architecture behind the release

Amazon ECS Fargate provides the container runtime. Rather than managing EC2 worker instances directly, the platform defines the container requirements and lets Fargate provide the underlying compute.

An Application Load Balancer provides the entry point and traffic-routing layer. Terraform defines the supporting AWS resources, while GitHub Actions connects source-code changes to the build and deployment workflow.

Amazon ECR stores the container images produced by the pipeline. IAM roles control which components can interact with AWS services.

GitHub repository
       |
       v
GitHub Actions ---- OIDC ---- AWS IAM / STS
       |
       v
Build and scan container image
       |
       v
Amazon ECR
       |
       v
ECS Fargate deployment
       |
       v
Application Load Balancer
       |
       +-- Existing application version
       |
       +-- Replacement application version

The repository contains the fuller infrastructure implementation and architecture diagram.

The important architectural separation is between provisioning the platform, producing a deployable artefact and controlling how that artefact reaches users.

Each component has a different responsibility. Understanding those boundaries becomes particularly important when a deployment fails.

Making infrastructure reproducible with Terraform

Using Terraform turned AWS configuration into something I could review and reproduce. Networking, IAM and ECS-related resources could be described in code instead of being assembled through a sequence of console clicks.

That matters for troubleshooting because a proposed infrastructure change becomes visible before it reaches AWS.

terraform fmt -check

terraform validate

terraform plan

These commands serve different purposes. Formatting checks consistency, validation checks the structure of the configuration, and planning evaluates the changes Terraform expects to make.

None of these checks proves that the application will work after deployment.

A reliable delivery process needs to separate infrastructure correctness from application behaviour. Terraform can show that an ECS service or load balancer will be modified, but it cannot guarantee that the replacement application will handle requests correctly.

Infrastructure as code also made teardown part of the development process.

Fargate, load balancing, networking and other AWS services can generate costs even when a portfolio application receives little traffic. Repeatability is valuable partly because it allows the environment to be recreated without keeping every resource running indefinitely.

Making the container artefact traceable

A deployment pipeline should make it possible to answer a deceptively simple question: exactly which image is running?

A mutable tag such as latest is convenient, but it is not a reliable historical record of a release. The same tag can point to different image contents over time.

Image digests provide content-addressed references. Commit-specific identifiers can also help connect source revisions to build outputs.

This becomes important during rollback. Reverting to a tag that has since moved may not restore the version you intended.

docker image inspect \
  --format='{{index .RepoDigests 0}}' \
  <image-reference>

The command illustrates how to inspect a local image's repository digest when one is available.

The broader principle is to preserve traceability between the source revision, container image and deployed workload.

A pipeline that reports success without making the deployed artefact identifiable creates unnecessary uncertainty when investigating incidents.

GitHub Actions without long-lived AWS keys

The CI/CD pipeline needed permission to interact with AWS. A traditional approach would involve storing an IAM user's access key ID and secret access key in GitHub repository secrets.

Instead, I used GitHub Actions with OIDC authentication.

This allows the workflow to request an identity token from GitHub and exchange it for temporary AWS credentials through Security Token Service, provided the IAM role's trust conditions are satisfied.

permissions:
  contents: read
  id-token: write

steps:
  - name: Configure AWS credentials
    uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
      aws-region: eu-west-2

This is an illustrative workflow fragment rather than a claim that the repository currently contains these exact lines.

The workflow permission allows GitHub to issue an OIDC token. AWS access still depends on the IAM trust policy and the permissions associated with the role.

That distinction became particularly important when I started debugging identity claims.

The OIDC failure that forced me to look deeper

During the project, I encountered problems while investigating GitHub's OIDC identity claims.

The tempting response to an authentication failure is to keep changing IAM permissions or workflow configuration until the request succeeds.

But an OIDC failure can occur before AWS even evaluates the permissions the assumed role would grant.

The important detail is the identity GitHub presents.

An OIDC token contains claims describing its issuer, audience and execution context. AWS evaluates relevant claims against conditions in the IAM role's trust policy.

If the subject does not match the trusted repository, branch or environment context, STS can deny role assumption even when the workflow has permission to request an identity token.

repo:example-org/example-repo:ref:refs/heads/main

repo:example-org/example-repo:environment:production

The subject can change depending on the workflow context and GitHub's OIDC configuration.

Investigating the claims helped me understand that authentication is not a single checkbox.

GitHub must issue the expected identity. AWS must trust that identity. The resulting role must then have the appropriate permissions for the requested operation.

Introducing traffic gradually

The deployment design used blue/green concepts with canary traffic shifting.

Rather than immediately directing every request to the replacement application version, a canary configuration can introduce the new version to a smaller percentage of traffic.

In the project design, the initial traffic shift was 10%, followed by a five-minute monitoring interval before continuing the rollout.

This provides an opportunity to observe the new version while the existing application continues serving most requests.

However, a monitoring window is only useful when meaningful signals are available.

Five minutes of observation might detect immediate startup failures or elevated HTTP error rates. It may not detect infrequently used application paths, gradual resource exhaustion or problems that only appear under sustained load.

The exact traffic-shifting and rollback behaviour depends on the ECS deployment controller and its configuration. Percentages alone do not guarantee a safe rollout.

Testing rollback by deliberately breaking the application

I wanted to verify that the deployment system could recover from a failed release, rather than assume the rollback configuration would work.

To test this, I deliberately introduced a failure into the application's /healthz endpoint and deployed the broken revision through the pipeline.

The deployment configuration used a 10% canary traffic shift and a five-minute bake period. The Application Load Balancer's health checks detected the unhealthy application targets, causing the associated CloudWatch alarm to enter the ALARM state.

ECS responded by automatically rolling back the deployment to the previous healthy revision. This demonstrated that the deployment's failure-detection and recovery mechanisms were connected and functioning.

Deploy revision with broken /healthz
                |
                v
ECS begins canary deployment
                |
                v
ALB detects unhealthy targets
                |
                v
CloudWatch alarm enters ALARM
                |
                v
ECS triggers automatic rollback
                |
                v
Previous healthy revision restored

The experiment validated an important failure scenario: a replacement revision that could not satisfy the configured health checks. It did not prove that every application regression would be detected, particularly failures that leave /healthz responding successfully.

Rollback needs meaningful signals

Rollback is often described as a feature that can simply be enabled. The more difficult engineering decision is determining when it should happen.

A deployment system needs a definition of unacceptable behaviour.

That might include unhealthy load-balancer targets, elevated server error rates, failed application requests or another metric closely connected to application correctness.

CloudWatch alarms can participate in automated deployment decisions when integrated with the relevant deployment mechanism.

But the alarm configuration matters.

An alarm that is too sensitive can interrupt healthy deployments. An alarm that only detects complete service failure may miss serious application regressions.

There is also a distinction between reverting application traffic and reversing every consequence of a release.

Rolling back an application version does not automatically reverse database schema changes or undo data written by the new version.

As a platform becomes more stateful, compatibility between application versions and persistent data becomes part of deployment reliability.

When container security checks complicate the pipeline

The project also involved investigating container vulnerabilities and security-scanning failures.

A Docker image can build successfully while containing vulnerable operating-system packages or application dependencies.

I worked through base-image updates and vulnerability-scanning issues while improving the pipeline.

Some of the debugging involved scanner handling of software bill of materials data embedded in image layers.

That required distinguishing between genuine vulnerability findings, dependency problems and the behaviour of the scanning tool itself.

Treating every failed security check as the same problem would have made troubleshooting less effective.

Updating a base image can address vulnerabilities, but it can also introduce compatibility changes. A security improvement still needs to be tested against the application's runtime behaviour.

The wider lesson was that build success, security assessment and runtime correctness are separate properties.

The cost of production-style infrastructure

A platform can be technically sophisticated while still being unnecessarily expensive for its intended workload.

Fargate tasks, load balancers, networking and managed AWS services all introduce operational costs.

For a portfolio environment, running a complete production-style architecture continuously may provide little additional value.

Infrastructure teardown therefore becomes part of the development workflow.

Terraform helps make resource creation and removal repeatable, but it is still important to inspect the AWS account for resources that may remain after a destroy operation.

The broader lesson is that architecture always involves trade-offs.

Reliability, operational complexity and cost need to be considered together. Production-style engineering does not mean using every available cloud service. It means understanding why each component exists and what it costs to operate.

What I took away

The most valuable part of building this platform was not learning how to launch containers on ECS.

It was learning how the surrounding systems influence whether a deployment is safe, reproducible and maintainable.

Terraform made infrastructure changes reviewable. GitHub Actions connected source control to deployment automation. OIDC introduced a more deliberate authentication boundary. ECS provided the runtime, while traffic shifting and monitoring shaped how releases could be evaluated.

The debugging work was particularly valuable because it forced me to understand why the system behaved a certain way rather than simply follow a configuration that happened to work.