Lab 4: AWS Tools for VSTS

Author: Kirk Davis (4/22/20)

Create Azure CI/CD Pipeline for ASP.NET Core to AWS Fargate

In this lab, you’ll create an Azure DevOps build pipeline to build a Docker container image containing an ASP.NET Core MVC application and push the image to Amazon ECR (Elastic Container Registry). Then build a release pipeline to trigger Amazon ECS (Elastic Container Service) to update the running container with the latest version.

Azure DevOps to AWS Pipeline Diagram

Step 1: Install the AWS Tools for Azure DevOps

Set up an Azure DevOps Account

To use Visual Studio Team Services (VSTS) you will first need to sign up for a Visual Studio Team Services Account

Install the AWS Tools for Azure DevOps Extension

Go to the relevant Visual Studio Marketplace and search for AWS in the search box. (alternatively, the following URL is a direct link to the AWS Tools for DevOps: https://marketplace.visualstudio.com/items?itemName=AmazonWebServices.aws-vsts-tools )

Choose Get it free and sign in to your VSTS account if prompted. Then choose Install to install into your VSTS account.

 Download Team Services Extension

Please note that some corporate environments lockdown the permissions available in Azure DevOps so that you may not have the rights to perform the operations in these labs. For this reason, we recommend creating a new Azure DevOps account which Microsoft currently provides for free using a personal email address so that you have the rights to perform the operations in this lab.

Step 2: Create an Azure DevOps Project and add sample ASP.NET solution

  1. In Azure DevOps, create a new project. You can name it whatever you like. We suggest you keep the visibility set to private.

  2. After the project is done being created, select Repos in the left menu. You should now see options for adding code to your project.

Import sample ASP.NET solution from GitHub

  1. Under Import a repository, click the Import button.
  2. In the import panel on the right, ensure Repository type is set to Git, and paste the following GitHub URL into the ‘Clone URL’ field:

    https://github.com/Kirkaiya/FargateFromAzureDevOps.git
    
  3. Click the Import button at the bottom of the panel to begin importing code. This process will take up to 2 minutes to complete.

Step 3: Configure the AWS Tools for Azure DevOps

Follow the steps below to create and configure an Azure DevOps Project Service Connection for your new project.

  1. Find the gear icon for your project, either in the lower, left-hand corner of the window or by hovering over the project’s name.

  2. Choose the icon to open the Project settings for your project.

  3. Choose Service connections.

  4. Under + New service connection, select the AWS endpoint type. Press Next. This opens the Add AWS service connection form.

VSTS enter keys

Provide the following parameters, and then click Save:

  • Connection name (e.g. AWS-connection)

  • Access key ID (e.g. from credentials.csv that you downloaded)

  • Secret access key (e.g. from credentials.csv that you downloaded)

The connection name is used to refer to these credentials when you are configuring tasks that access this set of AWS credentials in your build and release definitions.

Step 4: Create the build pipeline

  1. From the left-menu (in your Azure DevOps project), look for the Repos item. Select Files. Then click the Set up build button (on the right).

  2. Under Configure your pipeline, select the Starter pipeline option (see screenshot below). Azure Pipeline Template list

  3. Under Review your pipeline YAML, edit the pipeline YAML file, removing the hello world step, and changing the multi-line script to do the docker build. Your file should look like this after editing:

    # Starter pipeline
    # Start with a minimal pipeline that you can customize to build and deploy your code.
    # Add steps that build, run tests, deploy, and more:
    # https://aka.ms/yaml
    
    trigger:
    - master
    
    pool:
    vmImage: 'ubuntu-latest'
    
    steps:
    - script: |
    echo Starting docker build
    docker build -t hellofargate .
    displayName: 'Run docker build'
  4. Next, near the top-right of the YAML editor, click the Show assistant link to show the list of tasks.

  5. Type ‘AWS ecr’ in the ‘Search tasks’ box, the select Amazon ECR Push in the list. This will open that task in the task pane.

  6. For AWS Credentials, click the drop-down and select the connection you created earlier.

  7. Select the AWS region you are working in today (e.g. US East (N. Virginia))

  8. See the screenshot below for the rest of the values to enter. .

    • Set Source Image Name: hellofargate
    • Set Target Repository Name: hellofargaterepo
    • Ensure you check the box for Create repository if it does not exist AWS Push to ECR Task
  9. Click Add to add the generated YAML to your pipeline’s YAML file.

  10. Your YAML file should look similar to the example below.

    Sample build pipeline YAML file:

    # Starter pipeline
    # Start with a minimal pipeline that you can customize to build and deploy your code.
    # Add steps that build, run tests, deploy, and more:
    # https://aka.ms/yaml
    
    trigger:
    - master
    
    pool:
    vmImage: 'ubuntu-latest'
    
    steps:
    - script: |
    echo Starting docker build
    docker build -t hellofargate .
    displayName: 'Run docker build'
    
    - task: ECRPushImage@1
    inputs:
    awsCredentials: 'YourAwsServiceConnectionName'
    regionName: 'us-west-2'
    imageSource: 'imagename'
    sourceImageName: 'hellofargate'
    repositoryName: 'hellofargaterepo'
    autoCreateRepository: true
  11. Click Save and run to save your pipeline and kick off a build (which will also push the generated container image to Amazon ECR). May need to press Save and run* again. This make take several minutes to complete.

  12. After the pipeline has successfully completed, log into the AWS console,

    • Navigate to the Elastic Container Service (ECS) console.
    • Click Repositories on the left menu, and verify that there is a new repository named ‘hellofargaterepo’ present.

Step 5: Create ECS Fargate Service

In this section, you will create an ECS Task definition, an ECS cluster, and then a service on that cluster to run a task based on the task defintion.

A cluster can be of one of two main types: EC2 based, or Fargate based. With EC2-based clusters, you manage EC2 instances (virtual machines) on which Docker containers will be launched. With Fargate clusters, there are no EC2 instances to manage, and Docker containers are launched on AWS-managed infrastructure. For this lab, we will create a Fargate cluster.

A task defines one or more containers to launch together, which share storage and networking.

Step 5.1: Create ECS Task Definition

  1. In the AWS Management console, navigate to the ECS console.

  2. Under ‘Amazon ECR’ in the left menu, select Repositories.

  3. In the Repositories list, find and click on the ‘hellofargaterepo’ repository (click on the name, don’t select the radio-button).

  4. In the Images list, click the ‘copy’ icon to the left of the Image URI for the newest (top) image. It should end in ‘:latest’. See screenshot below. ECR select repo view

  5. Paste the repo image URI into notepad, or somewhere else you can retrieve it in the next section.

  6. Under ‘Amazon ECS’ in the left menu, select ‘Task Definitions’.

NOTE: IF ECS Cluster creation fails due to missing ECS Service linked role, see this doc page:

Create ECS Service Linked Role docs

  1. Click Create new Task Definition.

  2. Under Select launch type compatibility, select FARGATE, then click Next step.

  3. Under Configure task and container definitions, enter HelloFargateTask (no quotes) for the task definition name.

  4. Under Task execution IAM role (not ‘Task Role’), in the ‘Task execution role’ dropdown, ensure ‘ecsTaskExecutionRole’ is selected. If this role is not available in the drop-down, select ‘Create new role’ to generate a new one.

  5. Select 0.5GB for the Task memory, and 0.25 vCPU for Task CPU, in their respective dropdowns.

  6. Click Add container to show the ‘Add container’ pane.

  7. For ‘Container name’, enter hellofargatecontainer (no quotes).

  8. In the ‘Image*’ box, paste in the hellofargaterepo image URI you copied earlier. It should look similar to:

    12345678901.dkr.ecr.us-west-2.amazonaws.com/hellofargaterepo:latest

  9. Under Port mappings, for ‘Container port’, enter 80 (for HTTP traffic).

  10. Click the ‘Add’ button at the bottom right. You do not need to change any other settings. The container is added, and the ‘Add container’ pane will close.

  11. Click the Create button at the bottom of page.

  12. After the task is created, click the ‘View task definition’ button to review the task (if desired), and to re-show the left menu.

Step 5.2: Create ECS Cluster

  1. Under ‘Amazon ECS’ on the left menu, select Clusters.

    *Note: Ensure you click ‘Clusters’ under the ‘Amazon ECS’ menu, and not under ‘Amazon EKS’. EKS is a different cluster-orchestration service.

  2. Click Create Cluster.

  3. Select the Networking only cluster that is ‘Powered by AWS Fargate’, then click Next step. See screenshot below. Cluster template selection UI

  4. Give your cluster a name (e.g. LabFargate). Make a note of the cluster name, as you’ll need it in the next section (to create the Azure release pipeline).

  5. Leave the other settings to their default values, then click Create to create the cluster. If you get an error stating that a role hasn’t been created, exit the wizard and try it again. The ECS creates an IAM role and if it hasn’t been completed then wait a couple minutes and retry it.

Step 5.3: Create ECS Service

An ECS Service is a long running task.

  1. Under Amazon ECS on the left menu, select Clusters.

    *Note: Ensure you click ‘Clusters’ under the ‘Amazon ECS’ menu, and not under ‘Amazon EKS’. EKS is a different cluster-orchestration service.

  2. Find and select (click on the name of) the cluster you created earlier.

  3. On the cluster details view, on the Services tab, click the Create button to start creating a service.

    Note: a service is just a long-running task, with options for scaling, service-discovery, load balancing, and more.

  4. Under Configure service, set the launch type to FARGATE.

  5. For ‘Task Definition’, select HelloFargateTask from the drop-down, and ensure the ‘Revision’ drop-down is set to ‘1 (latest)’ (or whatever revision is marked latest).

  6. For ‘Service name’, enter HelloFargateService.

  7. Set the number of tasks to 1. In a production application, you would typically use two or more tasks behind a load balancer, but for this lab, a single task will work fine.

  8. Leave the rest of the settings as is, and click ‘Next step’ at the bottom of the page.

  9. Under Configure network, for ‘Cluster VPC’, select the VPC from the drop-down list with the IP range starting with 172.31.0.0

  10. For subnet, choose any subnet. The specific subnet is not important for this lab.

  11. Ensure ‘Auto-assign public IP’ is set to ENABLED.

  12. Under Service discovery (optional), uncheck the ‘Enable service discovery integration’ checkbox. We won’t be using service discovery for this lab.

  13. Click Next step at the bottom of the page.

  14. Under Set Auto Scaling (optional), don’t change the setting, and just click Next step.

  15. Review your settings on the review page, then click Create service. After the service is done creating, click View service to view the service details page. ECS will now provision (start) one copy of the HelloFargateTask, which has one container, which contains a skeleton ASP.NET Core MVC app.

  16. If you want to view the ASP.NET Core MVC app, on the Services tab, click on the HelloFargateService service, then click the Tasks tab. click the only task ID (under ‘Task’ column at bottom), and look under Network for the task’s public IP address. You can paste that into your browser to see the website you deployed.

Step 6: Create the release pipeline

  1. In Azure DevOps, while still in the context of the project, click Pipelines on the left menu, then select the Releases sub-menu.

  2. Click the New pipeline button to begin creating a release pipeline.

  3. Under Select a template, click the Empty job link. See screenshot below. Screenshot of release pipeline template chooser

  4. Enter Prod (no quotes) for the stage name.

  5. Change the name of the pipeline to Release to prod (no quotes), see screenshot below. Screenshot of edit pipeline name

  6. Hover your mouse over the “1 job, 0 task” link in the Prod stage in the UI (see screenshot below) then click the link to show the (currently empty) tasks list. Screenshot of stage task add hover

  7. Change the display name of the agent job to something useful, like ECS Release Agent job, then click the “+” sign on the right side of the agent job bar to add a task to the job. See screenshot below. Screenshot of agent job add task link

  8. Type command in the task search box, then select the Command line task, and click Add.

  9. Name the task, ‘Install AWS CLI’ (no quotes), and in the command text field, replace the default echo commands with the following:

    echo Installing AWS CLI using pip
    pip install awscli
  10. Click the “+” sign on the right side of the agent job bar to add another task to the job. This time, type aws into the task search box, and select the AWS CLI task (click Add to add it to the job). See screenshot below. Screenshot of add AWS CLI task

  11. Select the new AWS CLI task in the list to show the details pane. Change the display name to something useful, like Update ECS Service.

  12. Select the AWS Connection you created earlier for the AWS Credentials, and choose the region (e.g. US East) you are working in today for the AWS Region drop-down.

  13. In the Command textbox, enter ecs

  14. For the subcommand, enter

    update-service
    
  15. For the ‘Options and parameters’ field, enter the following. Please note that if you named the ECS Cluster something different than LabFargate, then you will need to change the name in the command below to whatever you named your cluster. See screenshot below for an example:

    --cluster LabFargate --service HelloFargateService --force-new-deployment
    

    Screenshot of AWS CLI task for ECS

  16. Click the Save icon to save your changes. Enter a comment if desired, and leave the folder set to default.

Step 6.1: Create a manual release

  1. Click Releases on the left menu (under Pipelines), then click Create a release. See screenshot below. Screenshot of create release UI

  2. Under Create a new release, under ‘Pipeline’, select Prod (it has a lightening-bolt symbol to its left) to change the Prod stage trigger from automated to manual. Enter a description if desired, then click Create.

  3. Next, select ‘Release-1’ from the releases list to display the stages UI.

  4. Hover your mouse over the ‘Prod’ stage to show the ‘Deploy’ and ‘Logs’ buttons, then click Deploy. See screenshot below. Screenshot of release deploy button

  5. In the deploy pane that appears on the right, enter a comment if desired, then click the blue Deploy button.

  6. Once the release begins, you can click on the Prod stage rectangle to view its progress. When it is complete, you should see a view similar to the one below: Screenshot of completed ECS release agent job

Step 6.2: Verify release in AWS ECS console

  1. In the AWS console, navigate to the ECS console (if you’re not already there).

  2. Select Clusters from the left menu, then click on the LabFargate cluster.

  3. On the Services tab, click on the HelloFargateService service.

  4. Click on the ‘Events’ tab of the service to view events. You should see new events showing a new task starting. Click the Deployments tab, and there should be a new deployment row (scroll down if needed to see) with an ‘Updated time’ of the past few minutes.

Step 6.3: Congratulations! You have completed this lab

You created a build pipeline that builds the Docker image for the ASP.NET Core MVC app and pushes the image to Amazon ECR. You then created a release pipeline that triggers an ECS service to update itself. These two together will build and release changes to your Fargate-based web application when changes are pushed into the Git repository in Azure DevOps.