SSM Build Automation

Deploy SSM Build Automation

After completing the Builder Pipeline in the previous section, we will proceed to the automated deployment phase using the Systems Manager service. With the goal of deploying the new AMI that has completed OS Patching to the current application infrastructure, we will use an Automation Document.

With the Automation Document, the following activities will be executed sequentially:

• Automatically trigger the Builder Pipeline.
• Wait for the new AMI with completed OS Patching to be created.
• Update the CloudFormation Stack of the application with the new AMI value.
• Automatically trigger the AutoScalingReplacingUpdate policy to deploy the Blue/Green Deployment method for the Auto Scaling Group.

Furthermore, the key to this automation process is the AutoScalingReplacingUpdate policy to simplify the complexity of implementing the Blue/Green Deployment method manually. To easily visualize, let’s consider the following process:

• First, the AmazonMachineImage parameter of the pattern3-app Stack is updated with the new AMI ID, prompting CloudFormation to create a new version of the Launch Template.
• Since the Auto Scaling Group references the LatestVersionNumber of the Launch Template, CloudFormation recognizes this as a replacement change and proceeds to create a new Auto Scaling Group using the patched AMI.
• CloudFormation waits until the new EC2 instances send the cfn-signal and are evaluated by the Load Balancer with a Health Check of Healthy.
  • When the result is Healthy, CloudFormation shifts traffic to the new Auto Scaling Group and deletes the old Auto Scaling Group.
  • When the result is Unhealthy, CloudFormation proceeds to roll back the update process and retains the existing resources.

CloudFormation Stack

To proceed with infrastructure deployment, we will use the AWS CloudFormation service via the AWS Console or AWS CLI.

Please download the template here:

AWS CLI

Here are the initialization steps via AWS CLI:

1. Create CloudFormation Stack

   aws cloudformation create-stack --stack-name pattern3-automate --template-body file://pattern3-automate.yml --parameters ParameterKey=ApplicationStack,ParameterValue=pattern3-app ParameterKey=ImageBuilderPipelineStack,ParameterValue=pattern3-pipeline --capabilities CAPABILITY_IAM --region ap-southeast-2
   

2. Wait for the Stack to complete creation, this step only takes about 1 minute.

   aws cloudformation wait stack-create-complete --stack-name pattern3-automate --region ap-southeast-2
   

3. Verify that the CloudFormation Stack has been successfully created with the StackStatus as CREATE_COMPLETE.

   aws cloudformation describe-stacks --stack-name pattern3-automate --region ap-southeast-2 --query "Stacks[0].StackStatus" --output text
   

4. Note down the name of the Automation Document from the Outputs.

   aws cloudformation describe-stacks --stack-name pattern3-automate --region ap-southeast-2 --query "Stacks[0].Outputs" --output table
   

Prepare Automation Document

This section contains two parts. You only need to follow one of the two ways to obtain the Automation Document:

Manual Creation Steps

Only do this part if you did not deploy the pattern3-automate Stack in the previous section:

1. Access the Systems Manager service.
2. In the left navigation pane, under Change Management Tools, select Documents.

3. At the top right of the Documents pane, click the orange Create document button, then select Automation from the dropdown list.

4. The runbook design page opens with the default name NewRunbook. Click on the pencil icon next to that name and change it to pattern3-automate-CreateImage.

5. In the top bar, switch from Design mode to {} Code mode, and ensure the format selector on the right is set to YAML.

6. Delete the sample content in the editor pane, then paste the entire YAML specification found in the Specification Details section below.

7. Click the orange Create runbook button at the top right. The document will appear in the Owned by me tab on the Documents page.

aws imagebuilder list-image-pipelines --region ap-southeast-2 --query "imagePipelineList[?name=='pattern3-pipeline-ImagePipeline'].arn | [0]" --output text

aws ssm create-document --name pattern3-automate-CreateImage --document-type Automation --document-format YAML --content file://createimage.yml --region ap-southeast-2

aws ssm describe-document --name pattern3-automate-CreateImage --region ap-southeast-2 --query "Document.{Name:Name,Type:DocumentType,Status:Status}" --output table

Specification Details

This is the complete specification of the Automation Document, and it is the most crucial part of this chapter to read regardless of which method you chose:

• If creating manually, you paste the YAML snippet below into the Editor on the Console.
• If using CloudFormation, this YAML snippet is exactly what lies within the Content key of the AWS::SSM::Document resource in the pattern3-automate.yml file. In other words, CloudFormation is just a wrapper to register the document, while the actual logic resides here.

How it works:

1. First, we determine the schemaVersion and define the parameters.
   1. ImageBuilderPipeline: ARN of the Builder Pipeline we created in the previous section.
   2. ApplicationStack: Name of the CloudFormation Stack for the application - pattern3-app.
2. Next is the mainSteps specification. Using the values of the parameters, we create an action named ExecuteImageCreation via aws:executeAwsApi.
3. Then, we need to wait until the AMI finishes building using aws:waitForAwsResourceProperty. This is the longest step, taking about 20 to 30 minutes.
4. Once the AMI is in the AVAILABLE status, the GetBuiltImage action extracts the AMI ID and passes this value to the next step.
5. After obtaining the AMI ID, the UpdateCluster action proceeds to update the CloudFormation Stack of the application.
6. Finally, wait until the update process completes with the status UPDATE_COMPLETE.

description: >-
  Run the EC2 Image Builder pipeline to create a patched AMI, then update the
  Application Stack to deploy the new AMI via a Blue/Green methodology.
schemaVersion: '0.3'
parameters:
  ImageBuilderPipeline:
    description: (Required) ARN of the EC2 Image Builder Pipeline to execute.
    type: String
  ApplicationStack:
    description: (Required) Name of the Application Stack where the new AMI will be deployed.
    type: String
outputs:
  - GetBuiltImage.image
mainSteps:
  - name: ExecuteImageCreation
    action: aws:executeAwsApi
    maxAttempts: 3
    timeoutSeconds: 600
    onFailure: Abort
    inputs:
      Service: imagebuilder
      Api: StartImagePipelineExecution
      imagePipelineArn: '{{ ImageBuilderPipeline }}'
    outputs:
      - Name: imageBuildVersionArn
        Selector: $.imageBuildVersionArn
        Type: String

  - name: WaitImageComplete
    action: aws:waitForAwsResourceProperty
    maxAttempts: 3
    timeoutSeconds: 5400
    onFailure: Abort
    inputs:
      Service: imagebuilder
      Api: GetImage
      imageBuildVersionArn: '{{ ExecuteImageCreation.imageBuildVersionArn }}'
      PropertySelector: image.state.status
      DesiredValues:
        - AVAILABLE

  - name: GetBuiltImage
    action: aws:executeAwsApi
    maxAttempts: 3
    timeoutSeconds: 600
    onFailure: Abort
    inputs:
      Service: imagebuilder
      Api: GetImage
      imageBuildVersionArn: '{{ ExecuteImageCreation.imageBuildVersionArn }}'
    outputs:
      - Name: image
        Selector: $.image.outputResources.amis[0].image
        Type: String

  - name: UpdateCluster
    action: aws:executeAwsApi
    maxAttempts: 3
    timeoutSeconds: 600
    onFailure: Abort
    inputs:
      Service: cloudformation
      Api: UpdateStack
      StackName: '{{ ApplicationStack }}'
      UsePreviousTemplate: true
      Parameters:
        - ParameterKey: BaselineVpcStack
          UsePreviousValue: true
        - ParameterKey: NumberOfInstanceCluster
          UsePreviousValue: true
        - ParameterKey: MaxNumberOfInstanceCluster
          UsePreviousValue: true
        - ParameterKey: InstanceType
          UsePreviousValue: true
        - ParameterKey: LatestAmiId
          UsePreviousValue: true
        - ParameterKey: AmazonMachineImage
          ParameterValue: '{{ GetBuiltImage.image }}'
      Capabilities:
        - CAPABILITY_IAM

  - name: WaitDeploymentComplete
    action: aws:waitForAwsResourceProperty
    maxAttempts: 3
    timeoutSeconds: 3600
    onFailure: Abort
    inputs:
      Service: cloudformation
      Api: DescribeStacks
      StackName: '{{ ApplicationStack }}'
      PropertySelector: Stacks[0].StackStatus
      DesiredValues:
        - UPDATE_COMPLETE

Prepare Monitoring Script

We proceed to prepare a basic monitoring script that continuously sends requests to the URL of the Application Load Balancer before executing the Automation Document. By doing this, we can observe the entire process before, during, and after replacing the AMI.

#!/bin/bash
# Monitor application availability throughout the Blue/Green deployment process.
# Usage: ./watchscript.sh http://<ALB_DNS_NAME>

TARGET="$1"

if [ -z "$TARGET" ]; then
  echo "Usage: $0 http://<ALB_DNS_NAME>"
  exit 1
fi

while true; do
  RESULT=$(curl -s -o /dev/null -w "%{http_code} %{time_total}s" --max-time 5 "$TARGET")
  AMI=$(curl -s --max-time 5 "${TARGET}/details.php" | grep -o 'ami-[0-9a-f]\{8,\}' | head -n 1)
  echo "$(date '+%H:%M:%S')  StatusCode=${RESULT}  AMI=${AMI:-n/a}"
  sleep 2
done

chmod +x watchscript.sh

ALB=$(aws cloudformation describe-stacks --stack-name pattern3-app --region ap-southeast-2 --query "Stacks[0].Outputs[?OutputKey=='OutputPattern3ALBDNSName'].OutputValue" --output text)

./watchscript.sh "http://${ALB}"

# Monitor application availability throughout the Blue/Green deployment process.
# Usage: .\watchscript.ps1 -Target http://<ALB_DNS_NAME>

param([Parameter(Mandatory = $true)][string]$Target)

while ($true) {
    $time = Get-Date -Format 'HH:mm:ss'
    try {
        $sw = [System.Diagnostics.Stopwatch]::StartNew()
        $resp = Invoke-WebRequest -Uri $Target -UseBasicParsing -TimeoutSec 5
        $sw.Stop()
        $status = "$($resp.StatusCode) $([math]::Round($sw.Elapsed.TotalSeconds,3))s"
    } catch {
        $status = "ERROR $($_.Exception.Message)"
    }

    $ami = 'n/a'
    try {
        $details = Invoke-WebRequest -Uri "$Target/details.php" -UseBasicParsing -TimeoutSec 5
        $m = [regex]::Match($details.Content, 'ami-[0-9a-f]{8,}')
        if ($m.Success) { $ami = $m.Value }
    } catch { }

    Write-Output "$time  StatusCode=$status  AMI=$ami"
    Start-Sleep -Seconds 2
}

$ALB = aws cloudformation describe-stacks --stack-name pattern3-app --region ap-southeast-2 --query "Stacks[0].Outputs[?OutputKey=='OutputPattern3ALBDNSName'].OutputValue" --output text

.\watchscript.ps1 -Target "http://$ALB"

Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass

The script will continuously send requests and print the StatusCode along with the serving AMI ID. Thanks to this, you can see the exact moment the AMI is swapped, and more importantly, confirm whether the application experiences downtime or not.

Execute Automation Document

Once the Monitoring Script is running, we proceed to execute the Automation Document. You can choose one of the two methods below.

Method 1: via AWS Console

1. In the left navigation pane, under Change Management Tools, select Automation.
2. Click the Execute runbook button.
3. In the Owned by me tab, select pattern3-automate-CreateImage and click Next.
4. The Execute automation runbook page opens. For the execution mode, keep Simple execution.
5. In the Runbook details section, leave Runbook version as $DEFAULT.
6. In the Input parameters section, fill in:
   1. ImageBuilderPipeline: if this parameter has a default value, the field will be pre-filled with the pipeline ARN. If empty, paste your pipeline ARN into it.
   2. ApplicationStack: enter pattern3-app.
7. Click the Execute button at the bottom of the page.
8. The Console switches to the execution details page. Track the progress in the Executed steps section, where each step will sequentially transition from Pending to InProgress and then Success.

Method 2: via AWS CLI

# Get the name of the runbook owned by yourself. The lab only has one, so fetch the first item.
DOC_NAME=$(aws ssm list-documents --filters Key=Owner,Values=Self Key=DocumentType,Values=Automation --region ap-southeast-2 --query "DocumentIdentifiers[0].Name" --output text)
echo "Runbook: $DOC_NAME"

EXECUTION_ID=$(aws ssm start-automation-execution --document-name "$DOC_NAME" --parameters "ApplicationStack=pattern3-app" --region ap-southeast-2 --query "AutomationExecutionId" --output text)
echo "AutomationExecutionId: $EXECUTION_ID"

PIPELINE_ARN=$(aws imagebuilder list-image-pipelines --region ap-southeast-2 --query "imagePipelineList[?name=='pattern3-pipeline-ImagePipeline'].arn | [0]" --output text)

aws ssm start-automation-execution --document-name "$DOC_NAME" --parameters "ApplicationStack=pattern3-app,ImageBuilderPipeline=$PIPELINE_ARN" --region ap-southeast-2

# Get the name of the runbook owned by yourself. The lab only has one, so fetch the first item.
$DOC_NAME = aws ssm list-documents --filters Key=Owner,Values=Self Key=DocumentType,Values=Automation --region ap-southeast-2 --query "DocumentIdentifiers[0].Name" --output text
Write-Output "Runbook: $DOC_NAME"

$EXECUTION_ID = aws ssm start-automation-execution --document-name "$DOC_NAME" --parameters "ApplicationStack=pattern3-app" --region ap-southeast-2 --query "AutomationExecutionId" --output text
Write-Output "AutomationExecutionId: $EXECUTION_ID"

$PIPELINE_ARN = aws imagebuilder list-image-pipelines --region ap-southeast-2 --query "imagePipelineList[?name=='pattern3-pipeline-ImagePipeline'].arn | [0]" --output text

aws ssm start-automation-execution --document-name "$DOC_NAME" --parameters "ApplicationStack=pattern3-app,ImageBuilderPipeline=$PIPELINE_ARN" --region ap-southeast-2

2. Check the status of the Automation Document. The syntax is identical in both shells, except for how the $EXECUTION_ID variable was declared in the previous step.

   aws ssm get-automation-execution --automation-execution-id "$EXECUTION_ID" --region ap-southeast-2 --query "AutomationExecution.{Status:AutomationExecutionStatus,CurrentStep:CurrentStepName}" --output table
   

   Alternatively, view the list of executions:

   aws ssm describe-automation-executions --filters "Key=ExecutionId,Values=$EXECUTION_ID" --region ap-southeast-2 --query "AutomationExecutionMetadataList[0]" --output json
   

From the AWS Console, we can monitor each step and its status:

Verify AMI ID

We will proceed to verify whether the new AMI ID has been updated or not by accessing the DNS URL of the Application Load Balancer, and appending /details.php.

Cross-reference this with the values you saved in section 3:

• Amazon Image Id must be a different value, pointing to the AMI just built by Image Builder.
• Installed Packages will typically have a different count or newer version numbers, reflecting the newly installed patches.

Quick check via CLI:

# The AMI ID currently used by the Application Stack
aws cloudformation describe-stacks --stack-name pattern3-app --region ap-southeast-2 --query "Stacks[0].Outputs[?OutputKey=='OutputPattern3ActiveAmiId'].OutputValue" --output text

# The AMI ID returned by the Automation Document
aws ssm get-automation-execution --automation-execution-id "$EXECUTION_ID" --region ap-southeast-2 --query "AutomationExecution.Outputs" --output json

These two values must be identical.

At Step 1: ExecuteImageCreation, in the Outputs section, we can obtain the Pipeline Execution ARN.

Subsequently, we can cross-reference it with the value in the EC2 Image Builder service.

Finally, stop watchscript.sh by pressing Ctrl + C and move on to the Resource Cleanup section.