In an earlier article Building Docker Containers and Pushing them to Docker Hub with Azure DevOps Pipelines, we looked at how Azure DevOps Pipelines could be used to facilitate the container building process. This article extends on that process and covers deploying containers to the Amazon Elastic Container Service (ECS) and triggering ECS to redeploy the containers when the pipeline is run automatically.

At a high level, the steps taken are:

• Create a new secret in the AWS Secret Manager for authentication with Docker Hub
• Create a new AWS IAM policy and role for ECS tasks and ECS task execution
• Create a new AWS IAM policy, group, and user for Azure DevOps integration
• Install the AWS extension and configure a new service connection in Azure DevOps
• Create an AWS ECS Task
• Create an AWS ECS Cluster
• Create an AWS ECS Service
• Modify an existing Azure DevOps Pipeline to install the AWS CLI and force a redeployment of the ECS Task

AWS Secrets Manager

For ECS to pull images from a private Docker Hub repository, a username and password are required. To facilitate this, a secret will be added to the AWS Secrets Manager.

Log into the AWS Console and navigate to Secrets Manager which is in the Security, Identity, & Compliance group of services.

Click Store a new secret, then select Other type of secrets. Configure the following secret key, value pairs

• username - your docker hub username
• password - your docker hub password

Configure a secret name etc. as shown below.

Leave the secret rotation settings at their defaults (disabled).

Click Store after reviewing the configuration.

AWS IAM

ECS - Secrets Manager Access

This policy provides ECS with access to the Docker Hub key-value pair created earlier in the Secrets Manager.

Create a new policy in IAM as per the below. The resource is the ARN for the secret created above for the private Docker Hub repository.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt",
                "secretsmanager:GetSecretValue"
            ],
            "Resource": [
                "arn:aws:secretsmanager:ap-southeast-2:xxxx:secret:prod/DockerHub/xxxx-5Izl3a"
            ]
        }
    ]
}

ECS - Task Role

Create a new role in IAM for the ECS task role. This role will have the policy created above attached so ECS can access the Docker repository username and password from Secrets Manager.

In the new role creation screen select the Elastic Container Service and then select the Elastic Container Service Task.

Attach the policy created previously for the Secrets Manager ARN.

Add any tags if desired, and then give the role a name and click create role.

ECS - Task Execution Role

Create another role for ECS task execution. As per the previous section, in the new role creation screen select the Elastic Container Service and then select the Elastic Container Service Task.

Click next on the remaining steps and finally give the role a name and click create role.

ECS - Container Instance Role

Create another role for ECS. As per the previous section, in the new role creation screen select the Elastic Container Service and then select the EC2 Role for Elastic Container Service.

Click next on the remaining steps and finally give the role a name and click create role.

For more information on this role, check the Amazon ECS Container Instance IAM Role article.

Azure DevOps Integration

To facilitate the Azure DevOps pipeline integrating with the AWS ECS we need an account with sufficient privileges. For an example policy from AWS check out Amazon ECS IAM Policy Examples.

Create a new policy

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecs:ListServices",
                "ecs:UpdateService",
                "ecs:ListTasks",
                "ecs:ListTaskDefinitions",
                "ecs:ListClusters"
            ],
            "Resource": "*"
        }
    ]
}

Create a new group and attach the policy created above and then create a new programmatic access user and assign it to the group created above.

Azure DevOps Service Connection

A Service connection definition is required within the Azure DevOps Project settings to facilitate the pipeline tasks working with AWS ECS.

Open the Project, then click Project settings, then click Service connections.

Click + New service connection and select aws. If you don’t see AWS in the list navigate to https://marketplace.visualstudio.com/items?itemName=AmazonWebServices.aws-vsts-tools and click Get it free and follow the prompts to install the extension.

Complete the form using the details from the AWS IAM account you created earlier, see the below example.

Amazon Elastic Container Service (ECS)

ECS - Task Definition

Navigate to ECS from the AWS Console screen and click Task Definitions > Create new Task Definition.

Select EC2 if you are following along.

Configure the Task Definition Name, the Task Role, and Task execution role, as configured previously. Configure the memory and CPU as desired (an EC2 instance large enough to cater for what is configured is required).

Note, during testing, I found it was impossible to launch a single container on a t2.micro with minimal memory and CPU assigned.

Then click Add Container. In this example, an NGINX container is being defined.

For ECS to pull the Docker image from the private repository check Private repository authentication and paste in the Docker Hub Secrets Manager ARN. The Image field should be in the format <dockerhubusername>/<repository>:<imagename>.

Complete the rest of the container definition as required. For example set the port mappings, hostname, and links etc. Click Add. Repeat the process until all the desired containers are defined and complete the task creation.

ECS - Cluster Creation

From the ECS navigation go to Clusters. At this point, we need to define a cluster to run the task on. Click Create Cluster.

Select the EC2 Linux + Networking and click Next step.

Configure the cluster with the required values, selecting the container instance IAM role created earlier.

Click Create.

ECS - Cluster Service Create

From the ECS Cluster screen, select the newly created cluster. On the Services tab click Create.

Configure the service as required selecting the Launch type as EC2 and selecting the Task Definition created previously. Refer to the AWS Service documentation for more details.

Click Next step.

Configure the network as required. In this case, no load balancer or service discovery is used — Click Next step.

Configure Service Auto Scaling if required. Click Next step.

Review the configuration and click Create Service.

The service should deploy the task definition to the EC2 instance/s created by ECS at this point, and we can move on to adjusting the Azure DevOps pipeline.

Azure DevOps Pipeline

Now that everything is set up and working from an AWS and Docker Hub point of view we can update the Azure DevOps pipeline to force the ECS Service to redeploy the task. When the task is redeployed, it will automatically include pulling update images from Docker Hub. Referring to the NGINX pipeline created in a previous post we will add two additional tasks.

• Install the AWS CLI into the build host
• Perform an AWS ECS Service Deployment

To install the AWS CLI requires a command line task. Add the command line task to the pipeline and configure as shown below. It requires two entries in the script textbox.

sudo python3 -m pip install setuptools wheel
sudo python3 -m pip install awscli

Next, add an AWS CLI task to the pipeline and configure as shown.

• AWS Credential should be the service connection credential added previously
• AWS Region should be the same as the region in which ECS was configured
• In the options and parameters
  • The cluster ARN is required. To find this use the AWS CLI command below
  • The service is defined by name

Get the ECS Cluster ARN:

aws ecs list-clusters

Now, once the Pipeline is triggered either manually or automatically it will a) build the Docker container images, b) push those images to the private Docker Hub repository, and c) instruct ECS to perform a new deployment and thus redeploy the containers using the new images on Docker Hub.