Getting Started with EC2 Image Builder — Let’s Build an AMI Pipeline!

Computer screen with code and HTOP

Almost two years ago, I wrote my very first article on EC2 Image Builder. At the time, I was tasked with developing a build and management solution for AMI’s within the group I worked with. At the time, I intended to write more on this topic, but as with anything in life, “sometimes the best laid plans…” I’ve been thinking about this ever since, and this article is meant to serve as a first in a series on Image Builder. In this article we’ll cover the basics, and build a simple pipeline. As articles progress wI plan on diving into more advanced topics such as using CloudFormation to automate ImageBuilder management, building advanced components (than what I’m showing you today), and tracking down errors in our pipelines.

Disclaimer — since last writing about Image Builder, I have taken a job with AWS. This article is not written on behalf of Kevin the AWS employee, but rather Kevin on his free-time outside his employment.

What is EC2 Image Builder?

EC2 Image Builder is an AWS managed service that automates the creation, update and deployment of your Amazon Machine Images (AMIs). Whether you’re building bootstrapped images to aid in faster deployment, or have InfoSec requirements and need a solution to build repeatable and secure AMI’s, these are the types of use-cases Image Builder was built for. In my particular case, I was doing the latter. The team I worked on managed multiple AWS accounts across both the commercial and GovCloud regions with InfoSec requirements to provide “Golden AMI” images with a predefined configuration. We required an automated solution that would allow us to build once (okay, twice in our case) then distribute those images to all our managed accounts.

Building our First Job

In this demo, we will build a simple pipeline, including component and recipe to serve as an introduction to the service. I recognize this isn’t a sustainable approach as many if not all of you following along, manage multiple accounts across various AWS regions. Just as we have to crawl before we can run, I find that by taking this approach I learn more about what I’m working with than by jumping straight into the deep end. As this series progresses, and as I’ve already indicated we’ll get into more advanced topics.

Building a Component

A component defines the sequence of steps required to either customize an instance prior to image creation (a build component), or to test an instance that was launched from the created image (a test component).

The above is Amazon’s official definition of an Image Builder component. Components are created using either YAML or JSON to build a declarative document that describes the configuration for building, validating and/or testing and instance produced within your pipeline. It can also be called a component document. Components are made up of two types of runtime phases; build, and test.

Build stages consist of two phases; build, and validate. In the build phase, a base image is downloaded and the configuration specified for the phase is applied to build and launch an EC2 instance.

In the validation phase, Image Builder validates that customizations have been successfully applied to the instance. This is based on the specified configuration within the validation phase of the component document. If instance validation is successful, the instance is turned off, an image is created and Image Builder continues to the test stage if one has been defined.

The test stage has one phase…yep, you guessed it. Test. During this stage/phase a newly created image from the build stage is launched and test components are run on it to verify the instance is healthy, and running as expected.

To demonstrate creating a new component, we’re going to build a fairly straight-forward shell script that will install the Apache web-server and configure the index.html file on our AMI. If we’re successful, when we launch an EC2 instance from this AMI, we should see the message “Congrats, you built your first Image Builder component!” when we navigate to the public IP address of the instance.

# Install apache start and enable the service
sudo yum install -y httpd &&
sudo systemctl start httpd &&
sudo systemctl enable httpd
# Configure index.html
echo '<!doctype html><html><body><h1>Congrats, you built your first Image Builder component!</h1></body></html>' | sudo tee /var/www/html/index.html

In this demo we’ll be using Amazon Linux 2 to build a simple web-server. The script will install the Apache web-server (httpd), start and enable the service and configures the index.html file with the message, “Congrats, you built your first Image Builder component!” If the image pipeline runs properly, at the end of this tutorial we should see that very line in our web browser. We will store this script in an S3 bucket, and our component will pull it down to the instance from this bucket at runtime.

Let’s go ahead and log into the AWS console, once logged in, type in ‘EC2 Image Builder’ in the services search bar and click on the EC2 Image Builder service.

AWS console, service search-bar for EC2 Image Builder

If you haven’t created any Pipelines in Image Builder yet, you should be greeted by a screen that says ‘Getting Started’. On the left side of the screen you should see a menu, under ‘Saved configurations’, click ‘Components’.

Image Builder menu

In the upper right corner, we’ll click on the ‘Create component’ button to begin creating our component.

Component home screen — Create component

The component creation process is broken up into four sections; ‘Component type’, ‘Component details’, ‘Definition document’ and ‘Tags’. We will start from the top and work our way down starting with ‘Component type’. The component we’re building today is going to be a Build component, so leave that selected and we’ll move on to the details section.

Component type

In the details section we can select the operating system to use, specifying which specific versions of the operating system are compatible with this component, and provide a name, description and version number. At this time, operating systems are limited to either Linux, or Windows. Compatible versions of Linux are; Amazon Linux 2, Ubuntu 16, 18 and 20, CentOS Linux 7 and 8, Red Hat Enterprise Linux 7 and 8 and SUSE Linux Enterprise Server versions 12 and 15. For Windows the following versions of Windows Server are compatible with Image Builder; 2004, 2012R2, 2016, 2019, 20H2 and 2022.

Select Linux as the operating system, and specify Amazon Linux 2 in Compatible OS Versions. Give it a name, short description and select the default option for ‘KMS keys — optional’. This will be version 1.0.0 (Image Builder uses SemVer) and a short version description.

Component details

Two options are available to select from in the definition document section; ‘Define document content’ and write your component from scratch, or ‘Use example’ and edit an AWS provided example and make inline edits to the document. Select definition document and create the component with a build phase consisting of three steps; DownloadScript, RunScript and InstanceCleanup. I will explain each of the steps in more detail below.

name: imageBuilderDemo-apacheWebServer
description: 'This Image Builder component will install Apache web-server and configure the index.html file with a simple message'
schemaVersion: 1.0
phases:
- name: build
steps:
- name: DownloadScript
action: S3Download
onFailure: Abort
maxAttempts: 3
inputs:
- source: s3://imagebuilderdemo-scriptbucket/configureEC2.sh
destination: /tmp
- name: RunScript
action: ExecuteBash
onFailure: Abort
maxAttempts: 3
inputs:
commands:
- 'chmod +x {{ build.DownloadScript.inputs[0].destination }}'
- 'bash {{ build.DownloadScript.inputs[0].destination }}'
- name: InstanceCleanUp
action: ExecuteBash
onFailure: Abort
maxAttempts: 3
inputs:
commands:
- 'rm {{ build.DownloadBootstrap.inputs[0].destination }}'

Definition documents follow the standard YAML format. At the top we have a document name, a description and a schema version which at the time of this writing, 1.0 is the only version. Moving down the document, we have one phase (build), with three steps. In the first step, we are going to download our script from S3 to the /tmp directory on the instance.

- name: DownloadScript
action: S3Download
onFailure: Abort
maxAttempts: 3
inputs:
- source: s3://imagebuilderdemo-scriptbucket/configureEC2.sh
destination: /tmp/configureEC2.sh

This step of the build uses the ‘S3Download’ action, and is configured to abort if a failure is detected. It will attempt a maximum of three runs before it will signal your pipeline to fail. This step has been configured with two inputs, the source S3 bucket and script to be downloaded, and the destination on the EC2 instance, in this case /tmp. Please notice that for both the source, and destination you must name the file in addition to the source bucket and destination directory.

- name: RunScript
action: ExecuteBash
onFailure: Abort
maxAttempts: 3
inputs:
commands:
- 'chmod +x {{ build.DownloadScript.inputs[0].destination }}'
- 'bash {{ build.DownloadScript.inputs[0].destination }}'

The next step uses the ‘ExecuteBash’ action on the script we just downloaded. Two commands are being run, the first command (chmod) will make the script executable while the second runs the script. While the format seen in the commands may be a little confusing at first, eventually you become accustom to working with it. This format is called input/output chaining and is written in the following format ‘{{ phase.step.input/output[index position].variable }}’. In this case, we are identifying the file downloaded during the download step (as the first and only file of that step it is index 0) and the directory we downloaded it to.

- name: InstanceCleanUp
action: DeleteFile
onFailure: Abort
maxAttempts: 3
inputs:
- path: '{{ build.DownloadScript.inputs[0].destination }}'

The last step is instance cleanup. We don’t want to give our customers/users images with bootstrapping material left on them, we want to provide them with clean AMIs on launch. For this we use the ‘DeleteFile’ action, with the input/output chaining format location of our script. We’ll skip over tagging our component and click ‘Create’. You should see the following if no errors are caught;

Component screen with our newly created component

Congratulations! You have just created your very first Image Builder component!

Building a Recipe

An EC2 Image Builder recipe defines the base image to use as your starting point to create a new image, along with the set of components that you add to customize your image and verify that everything is working as expected.

Whereas an Image Builder component tells the service how to build an individual piece of an AMI; install and configure Apache, run a yum update, install the AWS CLI, etc. Recipes tell Image Builder how to build a full AMI by combining multiple components in one action. Image Builder gives you the ability to create Image and Container recipes. We’re going to build an Image recipe today, so go ahead and click the ‘Image recipe’ in the menu on the left-side of the screen, clicking on that should bring us to the following screen;

Recipe menu screen, create recipe

Click the ‘Create image recipe’ button to get started on creating a new recipe. There are six sections in creating a recipe; ‘Recipe details, ‘Base image’, ‘Instance configuration’, ‘Working directory’, ‘Components’ and ‘Tags’. We’ll start off with the recipe details;

Recipe details

In this section we give our recipe a name, version number and optional description. In the next section ‘Base image’, we’ll select the image and options we’re going to work with;

Base image selection

When selecting the base image to build your image from, there are lots of options. Available for use are either Image Builder managed instances, providing a custom AMI ID that either you or another organization maintain (please note that these images must have the SSM agent pre-installed) or you can import a virtual machine into Image Builder and use that image to build from. In the next section we’ll select which Operating system we’re building on, whether that image is Amazon managed, owned by you or has been shared with you.

This recipe will be configured to use Amazon Linux 2 using the Amazon-managed image. Under ‘Image name’ you can select from x86, ARM and ECS optimized images, we’re going to select x86 in this case. Lastly, under versioning options we will use the latest available OS version. You can also select to use a selected OS version, or specify by major.minor.patch if your particular needs require you to use a specific version of a given OS.

Instance configuration options

We’ll stick with the Instance configuration defaults. Leave the remove SSM agent check box unchecked, as we’d like the agent left installed on our instance. You’ll also notice that you can enter user data just like you would when spinning up an EC2 instance from the console, we are not going to utilize that here. Hopefully though, you’ve recognized the irony in that, our component we previously configured could have been done from this User data section.

Working directory selection

Stick with the default of /tmp for working directory. although you may change this to any directory you prefer for your given use-case.

Build components selection screen

You can add up to 20 build components to your recipe, in this example we’re using two; the Apache web server component we previously created (imageBuilder-apacheWebServer) and the update-linux component (Amazon managed) that will run a yum update on our instance.

Build Component — toggle between customer owned and Amazon managed components

At the top of the Build components selection box, is a drop down menu where you can toggle between ‘Amazon-managed’ and ‘Owned by me’ components. We’ll start with the Amazon-managed components and pick the update-linux component (you should be able to find this on page 6 of the component list). Then change the drop down to ‘Owned by me’ and select the imageBuilder-apacheWebServer component.

Selected components for your recipe

Below the Build components section we can see the selected components in the ‘Selected components’ section. Clicking on the ‘Expand all’ toggle allows you to select to either use the latest available component version, or specify a version to run. We will skip over the test component section for now and move on, however I may have to come and revisit this a later date with an article on image testing…

The next two steps are optional, Storage (volumes) and tags. We will skip over those for the sake of this article and create our recipe. These sections should hopefully be pretty self-explanatory.

Our newly created recipe in the recipe menu

…and just like that we’re 2/3rds of the way to being able to create our very first AMI!

Image Builder image pipelines provide an automation framework for creating and maintaining custom AMIs and container images.

Just like CodePipeline or Jenkins provide an automation framework for developers to manage their code through a Continuous Integration/Continuous Delivery methodology, pipelines within Image Builder allows us to automate the creation and update of AMI’s. An Image Builder pipeline pulls all the pieces that go into making an AMI together; base image, components, infrastructure configuration and distribution settings. Within pipelines you can schedule builds, enable change detection for base image and components and enable the ability to skip scheduled builds when there are no changes. Click on the ‘Image pipeline’ option in the menu on the left of the screen, then click on the ‘Create image pipeline’ button and we’ll create our very first pipeline.

Image pipeline menu, click ‘Create pipeline’ button to start creation process

Creating an image pipeline is broken down into five steps; ‘Specify pipeline details’, ‘Choose recipe’, ‘Define infrastructure configuration’, ‘Define distribution settings’ and ‘Review’. The ‘Specify pipeline details’ step is broken down into three parts; ‘General’, ‘Build schedule’ and ‘Tags’.

General details for your pipeline are entered here

This section should be pretty straight-forward. We give our pipeline a name, brief description and select if we want to enable, or disable enhanced metadata collection, which allows us to collect additional information during the creation of an image.

Build schedule, select from three options on when you’d like your pipeline to run

For this demo, select manual and launch the pipeline manually from the console. The other scheduling options are to run our pipeline on a schedule by choosing the frequency, day and time from a dropdown menu, or define it as a CRON expression. Both options give additional granularity by letting us select to run the pipeline either every time the schedule dictates, or ONLY if dependency updates have been made somewhere within your pipeline, i.e. you’ve made an update to one of your components. Skip over tags in this case (again) and click next.

Recipe selection, select the Image Builder recipe you want to run in your Pipeline

On the choose recipe screen we have the ability to select either an existing recipe, or to create a new recipe. As we’ve already previously created a recipe we’ll keep that selected and select our recipe under the recipe details section.

Details of the selected recipe are shown here

Under the details section we can review the details, and make sure everything looks right before clicking next and moving on.

Infrastructure configuration — three options to choose from at this step

Next we define the Infrastructure configuration the pipeline will use to build images. Infrastructure configuration is where we can define the IAM role, instance type, SNS topic, VPC, subnet(s) and security groups key pair to use as well as what S3 bucket to use for logs. We have the option of using an existing infrastructure configuration, creating a new configuration or creating a configuration using service defaults, which is the option we will choose for this demonstration.

Choose your image distribution options from the distribution settings step

For the sake of this demonstration we’re going to create a distribution using service defaults. For those users using Image Builder in enterprise settings, using the default may not be an option, so you can create your own distribution(s) for your images. If you’re getting to this screen and have already created a distribution, you can select the ‘Use existing distribution settings’ option, then select the configuration from the drop-down menu.

From the console you can also select to create a new distribution. Within this menu, you can define the region or regions you want your image distributed to, output name of the AMI and several options for sharing the image across the accounts you manage. Options exist to define target accounts through their account numbers, or for those who manage their account structure via Organizations, you can share the image within your Organization, or within specific organizational units of your Organization. These are fairly new features to Image Builder, and coming from someone who previously managed distribution by entering individual account numbers, I can tell you this will GREATLY simplify AMI management and distribution. If your AMI will be used in an EC2 auto-scaling group, you can add a launch template configuration so that in addition to creating/updating your AMIs, Image Builder can also update your launch configuration with the new or updated AMI after your pipeline runs. In this section, you can also attach any license configurations if you have them through License Manager to manage any licensed software you’re installing on your images. Finally, you can define Output AMI tags and tags for your distribution as well.

The final step in the process of creating a pipeline is the review step. On this page, we can review each of the previous steps, ensure that everything looks good before clicking on the ‘Create pipeline’ button to create our pipeline.

Pipeline menu showing our newly created Image Pipeline

We’ve created a component, a recipe and now a pipeline so I’m sure you’re wondering “what’s next Kevin”? We’re going to run our shiny brand new (with new car smell and all!) pipeline of course. From the Image pipeline screen above, click on the newly created pipeline so we can get started.

Let’s run our pipeline by clicking ‘Actions’ > ‘Run pipeline’

From the ‘Actions’ drop-down, select ‘Run pipeline’ to kick off our first run. In the Output images section of the screen, we should see the run show up on the menu, initially in the ‘Pending’ status. In short order that should change to ‘Building’.

Our pipeline is now in a building state — grab a drink and let the magic happen!

The job should take about 30 minutes to run, so while it does, and depending on the time (it’s 5:00 somewhere) of day grab a coffee, bottle of beer, glass of whiskey, whatever suits you and let the magic happen. If your job runs successfully you should be greeted with a similar screen;

Our pipeline has successfully run, and we have an AMI!

The output of this run is version 1.0.0/1, which means the first run of version 1.0.0 of this pipeline. The status of available means our pipeline has completed and the AMI is available for use. If we head to the EC2 console, select ‘AMIs’ under the Images section we should see our newly created AMI.

Our newly created AMI in the EC2 console > AMI’s

Click ‘Launch instance from image’ to launch a new EC2 instance and validate the pipeline. I selected a t3.small for my instance type, the defaults for ‘Configure instance’ and ‘Add storage’, and added a name to the instance via tagging. On the ‘Configure Security Group’ step, I’m going to select ‘Create a new security group’ and add a http rule to allow http traffic from anywhere.

Configuring a simple security group for instance validation

Keep in mind, this is a demo and for a live application I would lock my security groups down more than what I’m showing you here. This being a quick demo however, the instance won’t be up long enough to be a problem. Click the ‘Launch instance’ button, select the key pair (or create a new pair) so you can access your instance. If everything went as planned, you should be able to to the instances dashboard within the EC2 console, select your instance and open the IP address in a new browsing tab.

Copy the URL of the instance and paste it into your web browser

If the page doesn’t come up, you may have to change the https to http in the URL if your browser defaults to https, which these days it likely does. You should however be greeted with the following;

Our simple web page we built into the AMI, you should see this if everything went right

Basic? Sure, but for the sake of this demo I think we’ve proven the power and usefulness of EC2 Image Builder. Also, please don’t knock the lack of CSS here. Don’t forget to delete any resources created as a part of this demo when you’re finished, we don’t want to rack up unnecessary charges.

For demonstration purposes, what we came up with is pretty manageable. As we take this into enterprise deployments however, using default configurations and maintaining this manually becomes…well, unmanageable. As this series progresses we’ll dive into more advanced topics; managing everything with CloudFormation, troubleshooting, creating advanced components, etc.

Congratulations, you made it to the end! Again, I hope you find value in this article, especially if you are, or will be using EC2 Image Builder to manage your images. Stay tuned for more, because this won’t be my last post on the topic!

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store