AWS

AWS Cloudformation with Python

In the world of AWS, there are many ways to do many things. With more than a 100 services comprising the AWS services portfolio, finding a strategy to deploy, manage and maintain multiple services can be challenging. One method is to use AWS CloudFormation

Michael Hyland

• 5 min read
AWS Cloudformation with Python

In the world of AWS, there are many ways to do many things. With more than a 100 services comprising the AWS services portfolio, finding a strategy to deploy, manage and maintain multiple services can be challenging. One method is to use AWS CloudFormation.

With CloudFormation you are able to use a template to describe all the AWS services you require. CloudFormation will use this template to provision and configure the required resources. With this method of service deployment, one does not need to individually create and configure resources and figure out what depends on what, CloudFormation will deal with this all for you. The advantages of this are the following:

  1. Simplify Infrastructure Management
  2. Quickly Replicate Your Infrastructure
  3. Easily Control and Track Changes to Your Infrastructure
  4. Single source of truth for AWS resources that can be version controlled

This method is also known as Infrastructure As Code and can be used to integrate into other DevOps concepts like Continuous Delivery Pipelines.

The CloudFormation Template

The CloudFormation template is the nuts and bolts of CloudFormation deployments. A template can be written in YAML or JSON. The template anatomy comprises of a structure of 9 sections of which the “Resources” section is the only mandatory section. Each service or resource will have its own unique Property Type.

CloudFormation Stacks

All related AWS services created by CloudFormation are placed within a single unit called a Stack. A CloudFormation Stack will comprise of all the resources you have described in the template. These resources are managed via the stack. In other words, the collection of resources is created, updated and deleted by creating, updating and deleting CloudFormation Stacks.

Change Sets

Making changes to the stack can sometimes be tricky. It’s best to use change sets when updating your CloudFormation Stacks. Generating a change set will produce a summary of your proposed changes. This will allow you to see how your changes may impact your running resources. This is important when making changes to critical, production resources. When doing updates, its common that resources get replaced. Even for what can be considered trivial changes like a name change. So, for example, if you wanted to update the name of an RDS instance, a change set summary will let you know that the entire database will be replaced with the implications of data loss. This allows you to plan accordingly.

Stack Drift

Stack drift is when your resources are changed outside of CloudFormation. These changes are when resources created by CloudFormation and are part of the stack are edited directly using the underlying service that created the resource. This is called an out-of-band change. This is not uncommon since these changes can be accidental or intentional due to time sensitive operational events. Changes outside of the stack will complicate stack updates or deletion events. CloudFormation offers drift detection to identify stack resources that have been changed outside of CloudFormation management. This will allow you to take corrective action to ensure your resources are in sync with the definitions in the stack template.

Deploying a CloudFormation Stack Using Python

CloudFormation stacks can be deployed using multiple methods i.e. the AWS cli, the AWS web console and the API. As an exercise, I have developed a Python script, using the Boto 3 library, that deploys a stack when supplied a CloudFormation template. The repository for this script can be found here.

What the script does

The scripts name is create_stack.py. As of now it only works with a json formatted template. Running a help on the script will display the available arguments for the script.

./create_stack.py -h
usage: create_stack.py [-h] -t TEMPLATE -n NAME -r REGION [-p PARAMS]
[-u UPDATE]

optional arguments:
-h, --help show this help message and exit
-t TEMPLATE, --template TEMPLATE
location of the template file
-n NAME, --name NAME name of the stack
-r REGION, --region REGION
the aws region
-p PARAMS, --params PARAMS
the key value pairs for the parameters of the stack
-u UPDATE, --update UPDATE
use this argument only if the stack requires updates

The script takes 5 arguments:

  1. TEMPLATE
  2. NAME
  3. REGION
  4. PARAMS
  5. UPDATE

The PARAMS argument is optional since defaults can be configured in the template. When including the params option in the script execution use the following format:

-p "Keyname=keyvalue&Keyname=keyvalue&Keyname=keyvalue"

Example:

-p "KeyName=ec2_keypair&SSHLocation=165.0.98.114/32"

The UPDATE argument is optional and when invoked takes any one of these: True, T or t

The Provided Template

The provided template, which is present in the repository and called webstack.json, will create a full web server environment in AWS. The following resources will be created:

  1. VPC
  2. Subnet
  3. Internet Gateway
  4. VPC Gateway Attachment
  5. Route Table
  6. Route
  7. Subnet Route Table Association
  8. Security Group
  9. EC2 Instance

The setup deployed will resemble the following diagram:

More information on this particular template can be found here

Creating a New Stack or Updating an Existing Stack

Running the script to create or update will initially present you with the analysis of the provided template. First, it will output the template description. Then, it will show what resources and how many of them will be created. It will then prompt whether you want to go ahead and create the stack with the listed resources. In this example, a key pair called ec2_keypair will need to exist in the region that you are deploying to. The SSHLocation parameter will be the IP address range that will have permissions to connect to the EC2 instance.

./create_stack.py -t webstack.json -n WebStack -r us-east-1 -p "KeyName=ec2_keypair&SSHLocation=165.0.98.114/32"

If you choose no, the script will exit with a "cancelled" message to the console.

If you choose yes, the script will load the template to AWS CloudFormation. It will then report on the status of the stack until the creation is complete.

The output of the script will match what the AWS console reports.

If any resources fail to create for any reason, the script will error out, roll back will be initiated and the reason for failure will be outputted.

Conclusion

AWS CloudFormation offers a centralized and robust method of deploying AWS resources. The trickiest part is building the template to match your desired environment and then keeping the deployed stack in sync. AWS offers comprehensive documentation and customer test cases to help you on your CloudFormation journey. I hope this blog has also offered a decent introduction to CloudFormation and Python. Feel free to contact me for any feedback or questions.