AMI Builder Pipeline

Deploy AMI Builder Pipeline

In this section, we will proceed to build an AMI Pipeline via EC2 Image Builder.

EC2 Image Builder is a service that simplifies the creation, maintenance, validation, and distribution of Linux/Windows Images.

• It can be used for EC2 AMIs, container images on Amazon ECR, as well as VM images for on-premises environments.
• Reduces time and effort spent on repetitive tasks.

Upon completing this section, we will be able to:

• Build an AMI Pipeline ourselves.
• Automatically create new AMIs containing the latest patches.
• Be ready to replace the old AMIs and deploy them to the current application environment.

CloudFormation Stack

To proceed with infrastructure deployment, we will use the AWS CloudFormation service via the AWS Console or AWS CLI.

Please download the template here:

The template takes the following parameters:

AWS CLI

Here are the initialization steps via AWS CLI:

1. Create CloudFormation Stack.

   aws cloudformation create-stack --stack-name pattern3-pipeline --template-body file://pattern3-pipeline.yml --parameters ParameterKey=BaselineVpcStack,ParameterValue=pattern3-base --capabilities CAPABILITY_IAM --region ap-southeast-2
   

2. Wait for the Stack to complete creation. This step is fairly quick, taking about 1 to 2 minutes, as we are just creating the definition of the pipeline and not building the AMI yet.

   aws cloudformation wait stack-create-complete --stack-name pattern3-pipeline --region ap-southeast-2
   

3. Verify that the CloudFormation Stack has been successfully created with the StackStatus as CREATE_COMPLETE.

   aws cloudformation describe-stacks --stack-name pattern3-pipeline --region ap-southeast-2 --query "Stacks[0].StackStatus" --output text
   

4. Note down the values in Outputs, especially Pattern3ImagePipeline (the pipeline ARN) and Pattern3ParentImageId.

   aws cloudformation describe-stacks --stack-name pattern3-pipeline --region ap-southeast-2 --query "Stacks[0].Outputs" --output table
   

aws ssm get-parameter --name /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 --region ap-southeast-2 --query "Parameter.Value" --output text

If you deployed using CloudFormation, you can skip the manual steps below and proceed directly to Execute Builder Pipeline.

Manual Build from Console

This section is for those who want to deeply understand each component of an AMI Pipeline. We will perform the following steps sequentially:

1. Create an S3 bucket for logging.
2. Create an IAM Role for EC2 Image Builder.
3. Create a Security Group for the build instance.
4. Create an Image Builder Component.
5. Create an Image Builder Recipe.
6. Create an Infrastructure Configuration.
7. Create an Image Builder Pipeline.

Create S3 Bucket

Due to the nature of the S3 service, when naming an S3 Bucket, we must follow certain rules specified at this URL.

1. Generate a unique UUID string with lowercase letters.

uuidgen | awk -F- '{print tolower($1$2$3)}'

([guid]::NewGuid().ToString('N')).Substring(0,12).ToLower()

2. Create the S3 Bucket using the prefix pattern3-logging combined with the generated UUID, and block all public access. This bucket only stores internal logs, so there is no reason to expose it to the Internet.

BUCKET="pattern3-logging-$(uuidgen | awk -F- '{print tolower($1$2$3)}')"
echo "Bucket: $BUCKET"

aws s3 mb "s3://${BUCKET}" --region ap-southeast-2

aws s3api put-public-access-block --bucket "$BUCKET" --public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true" --region ap-southeast-2

$BUCKET = "pattern3-logging-" + ([guid]::NewGuid().ToString('N')).Substring(0,12).ToLower()
Write-Output "Bucket: $BUCKET"

aws s3 mb "s3://$BUCKET" --region ap-southeast-2

aws s3api put-public-access-block --bucket $BUCKET --public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true" --region ap-southeast-2

Please note down the bucket name, you will need it in the IAM Role creation step and when configuring logging for the pipeline.

Create IAM Role

This IAM Role will be used for EC2 Image Builder acting as an instance profile for the temporarily created EC2 instance. This EC2 instance will perform the OS Patching task.

The following is the process to create an IAM Role:

1. Log in to the AWS Console and access the IAM service at - https://console.aws.amazon.com/iam/.
2. In the left navigation pane, select Roles and click the Create role button.

3. On the Select trusted entity screen, choose AWS service and set Use case to EC2, then click Next.

4. On the Add permissions screen, search for and select the following policies:
   1. EC2InstanceProfileForImageBuilder
   2. AmazonSSMManagedInstanceCore
5. Click Next to go to the Name, review, and create screen.
6. Enter the name pattern3-recipe-instance-role along with a detailed description.

7. Proceed to create it by clicking Create role.
8. Once successfully created, in the list of IAM roles, select pattern3-recipe-instance-role.
9. In the Permissions tab, click Add permissions and select Create inline policy.

10. Switch to JSON mode and use the following specification. Replace <S3_BUCKET_NAME> with the S3 Bucket name from the previous step.

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "WriteBuildLogs",
         "Effect": "Allow",
         "Action": "s3:PutObject",
         "Resource": "arn:aws:s3:::<S3_BUCKET_NAME>/*"
      },
      {
         "Sid": "DescribeBucket",
         "Effect": "Allow",
         "Action": [
            "s3:GetBucketLocation",
            "s3:ListBucket"
         ],
         "Resource": "arn:aws:s3:::<S3_BUCKET_NAME>"
      }
   ]
}

11. Click Next.
12. Enter the policy name and click Create policy to finish.

Create Security Group

We will need an EC2 Security Group for the temporary EC2 instance. The Security Group does not require any Inbound Rule; however, the default Outbound Rule is needed so the build instance can access the Internet via NAT Gateway to download patches.

1. Access the Amazon EC2 service.
2. In the left navigation pane, select Security Groups.

3. Click Create security group.
4. In the basic details section, enter the name pattern3-pipeline-instance-security-group and a description.
5. In the VPC section, select the VPC created by the pattern3-base Stack.
6. For Inbound Rules and Outbound Rules, keep the defaults and do not add any extra values.
7. Click Create security group to initialize.

Create a Builder Component

We will proceed to create a Component of the EC2 Image Builder service.

1. Access the EC2 Image Builder service.
2. In the left navigation pane, select Components.
3. Click the Create component button.
4. Proceed to choose the following values:
   1. Image Operating System (OS): Linux.
   2. Compatible OS versions: Amazon Linux 2023.
   3. Component name: pattern3-pipeline-ConfigureOSComponent.
   4. Component version: 1.0.0.
   5. Description: Component to update the OS with latest package versions.

5. In the Definition document section, select Define document content.
6. Copy and paste the following specification:

   name: ConfigureOS
   description: Update the operating system to the latest available packages
   schemaVersion: 1.0
   phases:
     - name: build
       steps:
         - name: UpdateOS
           action: UpdateOS
   

7. Click the Create component button to proceed with initialization.

Create a Builder Recipe

Before defining the Builder Pipeline, we must have a Builder Recipe.

1. Access the EC2 Image Builder service.
2. In the left navigation pane, select Image recipes.
3. Click the Create image recipe button.
4. In the Recipe details section, enter the following values:
   1. Name: pattern3-pipeline-ImageRecipe.
   2. Version: 1.0.0.
   3. Description: Pattern3 Configure OS Recipe.
5. In the Base image section, select Select managed images, then:
   1. Image operating system (OS): Amazon Linux.
   2. Image origin: Quick start (Amazon-managed).
   3. Image name: select the Amazon Linux 2023 x86 row.
   4. Auto-versioning options: select Use latest available OS version so each build fetches the latest AL2023 version.

6. In the Instance configuration section, leave the defaults. For Storage, it’s recommended to switch the Volume type to gp3 for lower cost and faster performance than gp2.
7. In the Components section, select the Component created in the previous step. You can quickly find it using the Owned by me filter.

8. Click the Create recipe button to proceed with initialization.

Create Infrastructure Configuration

The Infrastructure Configuration describes where the AMI is built: instance type, VPC, subnet, security group, and where to write logs. This is an independent resource that must be created before creating the pipeline.

1. Access the EC2 Image Builder service.
2. In the left navigation pane, select Infrastructure configuration, then click Create infrastructure configuration.
3. In the General section, enter:
   1. Name: pattern3-pipeline-InfraConfig.
   2. Description: Build infrastructure for pattern3 image pipeline.
   3. IAM role: select pattern3-recipe-instance-role created earlier.

4. In the AWS infrastructure section, enter:
   1. Instance type: t3.medium.
   2. SNS topic: leave blank.
   3. VPC, subnet, and security groups: select the VPC of the pattern3-base Stack, choose a Private Subnet, and select the pattern3-pipeline-instance-security-group security group.

5. In the Troubleshooting settings section:
   1. Keep Terminate instance on failure enabled.
   2. Key pair: leave blank, we do not use SSH.
   3. S3 location for logs: select the S3 bucket created in the previous step.

6. Click Create infrastructure configuration.

aws cloudformation describe-stacks --stack-name pattern3-base --region ap-southeast-2 --query "Stacks[0].Outputs" --output table

Create a Builder Pipeline

1. In the left navigation pane, select Image pipelines, then click Create pipeline.

2. General section:

   1. Pipeline name: pattern3-pipeline-ImagePipeline.
   2. Description: Pattern 3 pipeline to update OS.

3. Build schedule section, Schedule type: select Manual.

4. Recipe section:

   1. Recipe name: select pattern3-pipeline-ImageRecipe from the list.
   2. Versioning options: select Use latest version available so the pipeline automatically uses the latest recipe version. If you want to fix it, choose Specify recipe version then 1.0.0.

5. Workflow section: keep Use default workflows.

6. Infrastructure configuration section: select Select a configuration, then choose pattern3-pipeline-InfraConfig created in the previous step.

7. Distribution settings section: keep Create with default settings. The AMI will be created in the current Region ap-southeast-2, just as the Distribution details line shows.

8. Advanced settings section, expand if you wish to tweak. For the lab you can keep everything as default:

   1. Image tags and Tags: optional. If you want to easily find resources later, add the tag ResourceType = awsstudygroup-000099-imagebuilder.
   2. Auto-disable policy: keep the default 5 consecutive failures. This policy only applies to scheduled runs, not manual Run pipeline.
   3. Image log group and Pipeline log group: leave blank for Image Builder to use default CloudWatch log groups.
   4. Enhanced metadata (AMI only): defaults to Disabled, keeping it as is works fine. Choose Enabled if you want Image Builder to collect metadata via SSM Inventory to check component compatibility.
   5. EC2 image scanning: is grayed out and locked at Disabled. To enable vulnerability scanning, you must activate security scanning settings at the account level first, which utilizes Amazon Inspector and is out of the scope of this lab.
   6. ECR container scanning: ignore, we are not building container images.

9. Click Create pipeline at the bottom of the page.

Execute Builder Pipeline

Once the specification process is complete, we proceed to test the Builder Pipeline.

1. In the Image pipelines list, select the pipeline just created, click the Actions button and select Run pipeline.

2. Once execution is complete, we will check whether the new AMI has been initialized or not.

If using the AWS CLI, you can run the pipeline and monitor its status as follows:

# Get the pipeline ARN directly from Image Builder, works for both creation methods
PIPELINE_ARN=$(aws imagebuilder list-image-pipelines --region ap-southeast-2 --query "imagePipelineList[?name=='pattern3-pipeline-ImagePipeline'].arn | [0]" --output text)

BUILD_ARN=$(aws imagebuilder start-image-pipeline-execution --image-pipeline-arn "$PIPELINE_ARN" --region ap-southeast-2 --query "imageBuildVersionArn" --output text)

aws imagebuilder get-image --image-build-version-arn "$BUILD_ARN" --region ap-southeast-2 --query "image.state" --output json

aws imagebuilder get-image --image-build-version-arn "$BUILD_ARN" --region ap-southeast-2 --query "image.outputResources.amis[0].image" --output text

# Get the pipeline ARN directly from Image Builder, works for both creation methods
$PIPELINE_ARN = aws imagebuilder list-image-pipelines --region ap-southeast-2 --query "imagePipelineList[?name=='pattern3-pipeline-ImagePipeline'].arn | [0]" --output text

$BUILD_ARN = aws imagebuilder start-image-pipeline-execution --image-pipeline-arn "$PIPELINE_ARN" --region ap-southeast-2 --query "imageBuildVersionArn" --output text

aws imagebuilder get-image --image-build-version-arn "$BUILD_ARN" --region ap-southeast-2 --query "image.state" --output json

aws imagebuilder get-image --image-build-version-arn "$BUILD_ARN" --region ap-southeast-2 --query "image.outputResources.amis[0].image" --output text

Now, we will transition to the next section detailing the Build Automation deployment with the Systems Manager service.