Additional menu

CodingWithDrew

A Web and App Development Portfolio

CloudFormation
You are here: Home / Amazon Web Services / Learn CloudFormation with Drew

Learn CloudFormation with Drew

Table of Contents

Introduction

Would you like to manage your AWS resources and infrastructure as code without all the headaches of waiting for updates or unsupported resources?

Does Hashicorp Terraform feel just a little too complicated, unnecessary, and overkill since you are entirely on AWS?

Perhaps you are just getting into devops and want to learn the best and easiest way to manage AWS as code. Well, look no further.

This article is my notes on how to manage AWS using CloudFormation from zero knowledge to expert. We will cover how to write CloudFormation, YAML, AWSCLI, and Troposphere. You can probably cover this whole topic in less than 1 hour. I hope you enjoy reading it as much as I did learning it! Cheers. -Drew Karriker

What is CloudFormation?

To understand what CloudFormation is, first you'd need to know what AWS is.

AWS is a cloud provider from Amazon and I have a number of articles about AWS here: https://codingwithdrew.com/category/amazon-web-services/. For this guide, I assume you already have a basic working level of knowledge for most of the basic free tier services AWS services such as S3, IAM, EC2, and the like. I also assume you have an AWS account - be aware that some of these examples may not be free-tier eligible - always delete your CloudFormation Stacks after you are done learning.

AWS has so many services, it's overwhelming. Given that there are so many services and offerings from AWS, there is a need to be able to manage them all in an automated fashion instead of through the UI for each individual service. The more services there are, the more cumbersome the manual process becomes to the point of being untenable.

CloudFormation has entered the chat.

CloudFormation is a declarative way of outlining your AWS Infrastructure for any resource - most of which are supported. CloudFormation can create all your resources in the right order with the exact configuration in a single file. Better yet you can manage changes to the infrastructure with GitOps.

What are the benefits of AWS CloudFormation?

If you haven't heard the term "Infrastructure as code" before, you will. Basically it allows you to avoid manual processes for better control and to remove human error. You can version control your infrastructure - meaning you can implement GitOps and all changes with code reviewers and easily revert those changes.

Another Advantage is reduced costs. Each resources within a stack is staged with a tag. You can use that tag to see how much a stack is costing you! You can also estimate the costs of your resources using the CloudFormation templates. You can leverage cost saving strategies such as deleting templates at a specific time every day and then recreating them at another time.

Productivity can also be improved because you can destroy and re-create infrastructure on the cloud on the fly. It's declarative programming, so no need to figure out ordering and orchestration! Cloudformation is widely used so there is likely an existing template on the web - this can save a ton of time! This also means there is a ton of great documentation and help resources on this subject.

Separation of concerns - if you have many different stacks, apps, environments, and layers you can with CloudFormation.

Why CloudFormation over Ansible or Terraform?

I compared some of the main deltas below for ya:

CloudFormationOther Services
AWS native meaning it has and will always contain the latest options for AWS services.Must be updated everytime a new service or API option comes from AWS which can take a long time
State based and AWS figures out how to reach that stateInstruction based and it can be difficult to fully orchestrate your stacks

Introductory Example

We are going to create a simple EC2 instance and then we are going to add a security group to that EC2. No need to worry about the code syntax just yet. Just follow along with the copy paste examples.

On your local machine, create a file called ec2-example.yaml and open it in an editor.

Now paste the following YAML code into it:

    Resources:
        MyInstance:
            Type: AWS::EC2::Instance
            Properties:
            AvailabilityZone: us-west-2
            ImageId: ami-a4c7edb2
            InstanceType: t2.micro

What this YAML code does is it tells CloudFormation to create 1 EC2 instance on us-west-2 AZ. It's going to use the free-tier t2.micro instance size so no worries on cost here.

In AWS Management Console, head over to the services dropdown and search for "CloudFormation". You'll be prompted on what you'd like to create, select "Create new stack".

Make sure you are in us-west-2 AZ.

On the next screen, select "choose a file to upload to S3". Note that you can only use resources that are already in S3 storage.

Upload that new file you just created called ec2-example.yaml to S3.

Click next and provide a Stack name. Let's call it "intro".

Click next.

In the next page, you'll see "Options" and the ability to create a key and value. In the Key box, put: "Drew" and in the Value put "Learns". We will see this again in a moment.

Click Next.

We will be able to review our work - inside this page we will see our Template URL which has been uploaded to S3 in us-west-2.

Click on "Create".

In the next page that will appear, you will see a Stack Name: "Intro" with an orange Status of "CREATE_IN_PROGRESS".

If you click on the "Events" tab at the bottom, you will see a log of all the resources that we are going to create. The Status will not turn Green with "CREATE_COMPLETE" until all the resources in the ec2-example.yaml are created.

We can go to AWS Services drop down menu and select EC2 and in the EC2 instances dashboard we can see that the EC2 has been created. If we select that instance and go to the "Tags" tab, we can see our Key and Values we specified during our launch. We have also inherited 3 other tags associated with our CloudFormation template.

This means we can see what instances are tied to what and what resources associated with a CloudFormation configuration file costs.

What if we wanted to add an SSH security group? Can we just edit the CloudFormation file?

Turns out, you cannot. Instead what you have to do is create a new template. If you go back to your CloudFormation dashboard in AWS, and you click on the link for you Stack Name "Intro" a new page will appear.

Then you can click "Update Stack" on the top right. You can then edit the local file like this:

    Resources:
        MyInstance:
            Type: AWS::EC2::Instance
            Properties:
            AvailabilityZone: us-east-1a
            ImageId: ami-a4c7edb2
            InstanceType: t2.micro
            SecurityGroups:
                - !Ref SSHSecurityGroup 

        # our EC2 security group
        SSHSecurityGroup:
            Type: AWS::EC2::SecurityGroup
            Properties:
            GroupDescription: Enable SSH access via port 22
            SecurityGroupIngress:
            - CidrIp: 0.0.0.0/0
                FromPort: 22
                IpProtocol: tcp
                ToPort: 22

Then upload it to S3 again and click Next.

Bear in mind you cannot change your Stack Name "Intro". Click Next until you are on the Review page - you will have the "Update" option. You can preview the changes that will be made. If you are satisfied, you can click the "Update" button.

If you go back to the "Events" tab you will see the existing EC2 is terminated. This is great because it saw that you declared you only wanted 1 EC2 in your state and it deleted the old EC2 before creating a new one.

If you want to delete all the resources created by the CloudFormation all you have to do is go back to the CloudFormation Stack called "Intro" and right click on it's row and select "Delete Stack", a pop up will appear asking if you are sure you'd like to delete it. Then all the resources will be removed in the correct order without you having to figure it out!

Summary of the intro example:

  • Templates have to be uploaded to S3 and then referenced in CloudFormation.
  • To update a template, we cannot edit the existing template, we have to upload a new version of the template. It'll also tell us what it will update.
  • Stack names cannot be changed.
  • Deleting a CloudFormation stack will delete everything it created. This is super handy so there isn't any chance of human error in deleting resources that will cost money.

YAML stands for "YAML Aint Mark Up Language"

What is YAML?

YAML is a super easy writing/reading language similar to JSON. You can use JSON for CloudFormation but it's strongly recommended against.

YAML files leverage key-value pairs which are invoked with a :.

What can YAML do?

  • You can nest objects with a tab (2 spaces).
  • You can also leverage Arrays (identified by a - prefix).
  • You can use multiline strings (identified by a | prefix)
  • You can add Comments (Prefixed with a #) - this is something that JSON doesn't support (if you didn't know).

How can I create an S3 Bucket using CloudFormation?

We are going to use CloudFormation to provision S3 resources. S3 is free-tier eligible so don't worry about any charges on this one.

We are going to leverage this documentation: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket.html

Create a new local Yaml file and edit it to look like this:

        Resources:
            MyS3Bucket:
                Type: "AWS::S3::Bucket"
                Properties: {}

Now this is a very simple S3 resource. I'm not going to be redundant here and walk you through the CloudFormation upload template process we did in the first example, but step back through that process.

When you create your Stack Name, the new S3 Bucket will be named after it.

What if we wanted to add other properties to that S3?

We'll consider two different kinds up updates:

  • Non-Replacement - Updates with no interruption (adding AccessControl)
  • Replacement - updates such as updating the name of the bucket.

What is non-replacement?

Let's add AccessControl to our bucket's yaml file.

    Resources:
        MyS3Bucket:
            Type: "AWS::S3::Bucket"
            Properties:
                AccessControl: PublicRead

Now if you update the CloudFormation Stack, you will see that the S3 Bucket is going to be modifying the S3 Bucket and not replace the Bucket.

What is replacement?

Updates in the documentation that say "_Update requires: Replacement" will result in interruption of services because the existing resource will be destroyed and the a new resource added. You will lose data stored on the old S3 (at least potentially).

Let's see an example.

    Resources:
        MyS3Bucket:
            Type: "AWS::S3::Bucket"
            Properties:
                AccessControl: PublicRead
                BucketName: "Jeff-Bezos-1b2g21gg2"

If you update the S3 stack we created with this new CloudFormation yaml file we will delete the old bucket.

When you are done with this example, be sure to delete your Stack you created for this S3 example as a good practice.

What are the CloudFormation template options?

There are common parameters for all CloudFormation templates. Let's look at them:

  1. Tags - mean that every resource you have will share this tag. We set these in the 1st example above.
  2. Permissions - when you provide a resource, sometimes you'll need to add permissions (IAM Roles) to limit/enable specific users from performing certain actions.
  3. Notification Options - Allows AWS to notify emails or SNS topics when events happen.
  4. Rollback on Failure - Allows your CloudFormation to rollback to a previous state if something breaks on creation.
  5. Stack Policy - This protects resources in AWS CloudFormation from deletion/updating.

What is the CloudFormation Designer?

The CloudFormation Designer is a way to visualize your CloudFormation stack. It is very handy to quickly draft a CloudFormation Template.

It is well written to verify that your template is correctly implemented.

How do I use CloudFormation Designer?

From the AWS Management Console, go to the CloudFormation Service and create a new template. From there, at the top of the screen will be a button that says "Design Template".

The CloudFormation Designer will open up. It's a drag and drop feature. Also be sure that you choose a template language.

You can make relations to various resources. You can also visualize CloudFormation templates by clicking the Template tab at the bottom and pasting in it's contents into the Template and refreshing.

This is very useful to get started with CloudFormation Development.

What are the basic building blocks of CloudFormation?

Template Components - we will go into detail on each of these individually.

  • Parameters - the Dynamic inputs for your template
  • Resources - Your AWS resources declared in the template yaml file.
  • Mappings - The static variables for your template
  • Outputs - References to what has been created
  • Conditionals - Lists of conditions to perform resource creation
  • MetaData

Template Helpers are also integrated with each of these building blocks such as references and functions.

How do we automate deployments CloudFormation templates?

You can do it one of 3 ways:

  1. Editing Templates in a YAML file, leverage CI/CD tools to integrate templates into AWS
  2. Use AWS CLI to deploy templates

Parameters

What are Parameters?

Parameters are very powerful and can prevent errors from happening in your templates thanks to types. It's a way to provide inputs to your AWS CloudFormation Templates.

There are important to leverage if you want to reuse your templates for different applications or if some inputs cannot be determined ahead of time.

When should I use a parameter?

If you aren't sure if you should use a parameter, there a simple question you can ask yourself.

  • Is this CloudFormation resource configuration likely to change in the future?

If the answer is yes, then use a Parameter. You won't have to re-upload a template to change it's content!

How can you control a Parameter?

Parameters can be controlled by any of these things:

  • Type:
    • String
    • Number
    • CommaDelimitedList
    • List
    • AWS Parameter(to help catch invalid values - match against existing values)
  • Description
  • Constraints
  • ConstraintDescription (String)
  • Min/MaxLength
  • Min/MaxValue
  • Defaults
  • AllowedValues (array)
  • AllowedPattern (regexp)
  • NoEcho (Boolean)

Let's look at an example:

    Parameters:
        SecurityGroupDescription:
            Description: Security Group Description (Simple parameter)
            Type: String
        SecurityGroupPort:
            Description: Simple Description of a Number Parameter, with MinValue and MaxValue
            Type: Number
            MinValue: 1150
            MaxValue: 65535
        InstanceType:
            Description: WebServer EC2 instance type (has default, AllowedValues)
            Type: String
            Default: t2.small
            AllowedValues:
            - t1.micro
            - t2.nano
            - t2.micro
            - t2.small
            ConstraintDescription: must be a valid EC2 instance type.

In this example you can see that we set a min and max value for the Security Group's ports and we also create an array of Allowed values for EC2 instance types.

How do I reference a parameter?

The Fn::Ref function can be leveraged to reference parameters. These can be used anywhere in a temple and the shorthand for this is !Ref. It can also be used to reference other elements within the template.

Example:

    Parameters:
        MySecurityGroup:
            Type: "AWS:: EC2::SecurityGroup"
            Properties:
            GroupDescription: !Ref SecurityGroupDescription
            SecurityGroupIngress:
                -   CidrIp: !Ref SecurityGrouptIngressCIDR
                    FromPort: !Ref SecurityGroupPort
                    ToPort: !Ref SecurityGroupPort
                    IpProtocol: tcp
            VpcId: !Ref MyVPC

Resources

What are Resources?

Resources are the "Meat'n Taters" of your CloudFormation Template.

They represent the different AWS Components that will be created and configured - they are declared and can reference each other. AWS will figure out the creation, updates, and deletes of these resources for us so we don't have to write out the kitchen sink to create them!

FYI: There are over 200 types of resources. You can see them all here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-template-resource-type-ref.html

Resource Types are identified like this:

    Resource:
        MyCustomName:
            Type: AWS::aws-product-name::data-type-name

There is an optional attribute for Resources you should be aware of called "DependsOn:" which is very useful to draw a dependency between any two resources. For example, only create an ECS cluster after creating an Auto Scaling Group.

Another optional attribute for Resources is called "Deletion Policy:" which will protect resources from being deleted even if the CloudFormation Stack is deleted (example: You don't want your database or S3 to lose all its data).

Lastly, "MetaData:" is an optional attribute that allows you to put anything you want in it.

Mappings

What are Mappings?

Mappings are fixed variables within your CloudFormation Template that allow you to differentiate between different environments (dev/prod), regions, AMI types, and the like.

All values are hardcoded within the template.

When would you prefer to use mappings over a parameter?

Use Parameters when the values are really user specific. Mappings are great when you know in advanced all the values that can be taken and that they can be deduced from variables such as:

  • Region
  • Availability Zone
  • AWS Account
  • Environment (dev/staging/production)

They Allow safer control over the template

How do I access Mapping Values?

We will use Fn::FindInMap to return a named value from a specific key or shorthand FindInMap [MapName, TopLevelKey, SecondLevel Key]

Example:

    Mappings:
        RegionMap:
            us-east-1: #TopLevelKey
                "32": "ami-1234d" #SecondLevelKey
    Resources:
        myEC2-custom-name:
            Type: "AWS::EC2::Instance"
            Properties:
                ImageId: !FindInMap [RegionMap, !Ref "AWS::Region", 32]

Outputs

What are outputs in CloudFormation?

Outputs section declares optional outputs values that we can import into other stacks, this allows for cross stack referencing.

You can view the outputs in the AWS Console or in using the AWS CLI. They are very useful for networks defined in CloudFormation that need output variables such as VPC ID and the Subnet IDs.

Conditions

What are conditions used for?

Conditionals are used to control the creation of resources or outputs based on a condition. Conditions can be whatever you want them to be but common ones are Environment, Parameter values, and AWS Region. Each Condition can be used to reference another condition, parameter value, or mapping.

How do you define a condition?

You will need to choose any name for your condition and set it as the Logical ID.

Example:

    Conditions:
        Logical ID:
            Intrinsic function

You can then set the intrinsic function for the logical ID such as:

  • Fn::And
  • Fn::Equals
  • Fn::If
  • Fn::Not
  • Fn::Or

What is the Fn::GetAtt?

Fn::GetAtt is used to pull an attribute that is attached to any resource you create with CloudFormation. To know the attributes of your resources, the best place to look is the documentation from (AWS documentation)[https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-getatt.html]. Syntax for function name:

    Fn::GetAtt: [ logicalNameOfResource, attributeName ]

Shorthand:

    !GetAtt logicalNameOfResource.attributeName

Metadata (Optional)

What is Metadata and how do I use it?

Metadata section is optional and can use any arbitrary YAML you'd like that provides details about the template or resources.

Example:

    Metadata:
        Instances:
            Description: "Information about the instances"
        Databases:
            Description: "Information about the databases"

AWS CloudFormation Designer will automatically create Metadata to describe how the resources are laid out in your template, pretty neat!

CloudFormation init and EC2 User Data

This is one of the more complicated subjects to cover here but also probably the most important.

What is EC2 User Data?

If you were to create an EC2 instance from the AWS Web Console and if we were to write a user-data script that will be executed at the first boot of the instance to install php and mySql. The way this works is that the script will be run at boot time of the instance.

How do we use EC2 User Data?

If you were to log into AWS console and navigate to EC2 Dashboard then spin up a t2.micro instance with a linux AMI then on the Configure Instance step, at the bottom of the page is "Advanced Details" where you will find "User data".

In this field you could easily copy paste all the steps for installation of PHP 5.6 and MySQL to your new EC2.

It would look something like this:

    #!/bin/bash
    yum update -y
    yum install -y httpd24 php56 mysql55-server php56-mysqlnd
    service httpd start
    chkconfig httpd on
    groupadd www
    usermod -a -G www ec2-user
    chow -R root:www /var/www
    chmod 2775 /var/www
    find /var/www -type d -exec chmod 2775 {} +
    find /var/www -type f -exec chmod 0664 {} +
    echo "<?php phpinfo(); ?>" /var/www/html/phpinfo.php

Review and launch that instance.

You can see that security groups need to be set up and more but then you can access the site via the browser for your ec2 instance/phpinfo.php.

How do we use EC2 User Data in CloudFormation?

If we created an EC2 resource in Cloudformation with User Data, it would look like this:

Example:

    Resources:
        WebServer:
            Type: AWS::EC2::Instance
            Properties:
                ImageID: ami-a4c7edb2
                InstanceType: t2.micro
                KeyName: !Ref KeyName
                SecurityGroups:
                  - !Ref WebServerSecurityGroup
                UserData:
                    Fn:: Base64: |
                        #!/bin/bash
                        yum update -y
                        yum install -y httpd24 php56 mysql55-server php56-mysqlnd
                        service httpd start
                        chkconfig httpd on
                        groupadd www
                        usermod -a -G www ec2-user
                        chow -R root:www /var/www
                        chmod 2775 /var/www
                        find /var/www -type d -exec chmod 2775 {} +
                        find /var/www -type f -exec chmod 0664 {} +
                        echo "<?php phpinfo(); ?>" /var/www/html/phpinfo.php                        
        WebServerSecurityGroup:
            Type: AWS::EC2::SecurityGroup
            Properties:
                GroupDescription: "Enable HTTP access via port 80 + SSH access"
                SecurityGroupIngress:
                  - CidrIP:0.0.0.0/0
                    FromPort: '80'
                    IpProtocol: tcp
                    ToPort: '80'
                  - CidrIp: !Ref SSHLocation
                    FromPort: '22'
                    IpProtocol: tcp
                    ToPort: '22'

That | you are seeing in the Fn:: Base64: | isn't an error, what it means is "Everything after this pipe will be kept as is including the new lines".

Keep in mind: - CidrIp: !Ref SSHLocation, you can set as your own IP instead of using the reference.

Isn't that much easier to set up than doing it manually? Security groups and everything is already managed.

What are problems with EC2 User Data I should be aware of?

EC2 User data has limitations that you should be aware and how to work around them.

  • If we wanted to have a very large instance configuration, User Data can only accept so many characters so that's not good.
  • What about changing the EC2 instance state without terminating it and creating a new one?
  • How do we make EC2 User data more readable?
  • How do we know (signal) that our EC2 user-data script actually completed successfully?

CloudFormation Helper Scripts has entered the chat.

CloudFormation Helper Scripts

I'd like to preface this section with a general understanding that EC2 user data and helper scripts (cfn-init) are linked together with cloudformation init and that I provide an example at this end of this section showing how to use all of them together. This is a hard to follow section so I'll try to be as verbose as I can.

EC2 UserData is an imperative way to provision/boostrap the EC2 instance using SHELL syntax but has limitations. We just learned about these and I hope you have a good grasp on them before we push forward.

AWS::CloudFormation::Init is a declarative way to provision/bootstrap the EC2 instance using YAML. We are about to learn all about this.

AWS::CloudFormation::init is triggered inside UserData by way of helper scripts (cfn-init).

AWS::CloudFormation::init is useless unless it is triggered by UserData.

If that doesn't make sense (it likely won't just yet), just keep reading.

How do I implement CloudFormation Init?

Amazon Linux AMI comes with 4 Python (helper) scripts pre-installed.

  • cfn-signal: A Simple wrapper to singal an AWS CloudFormation Creation Policy or Wait Condition that will enable you to synchronize other resources in the stack with the application being ready.
  • cfn-get-metadata: A wrapper script making it easy to retrieve either all metadata defined for a resource or path to a specific key or subtree of the resource metadata.
  • cfn-hup: A daemon to check for updates to metadata and execute custom hooks when the changes are detected.
  • cfn-init: Most important to understand!! This is used to retrieve and interpret the resource metadata, installing packages, creating files, and starting services.

The typical workflow for these helper scripts is cfn-init, cfn-signal, then optionally cfn-hup.

How do I declare cfn-init?

This one is super powerful and complicated, there is a lot to unpack here so I'd definitely recommend bookmarking this section.

  • You can have multiple configs in your CloudFormation::init and run them sequentially
  • You create configsets with multiple configs
  • You invoke a config sets from your EC2 User data

A "config" in AWS Cloudformation contains the following and is executed in this order:

  • packages: install a list of packages on the OS from any of the following repositories: rpm, yum/apt, then rubygems/python. You can even specify a version or an empty array (latest)
  • groups: define user groups
  • users: define users and which groups they belong to
  • sources: These are an easy was to download whole compressed archives from the web and unpack them on the instance directly, typically a tar or zip file. Very handy if for example you have a set of standardized scripts for your instances that you store in S3
  • files: create files on the EC2 instance, using inline or can be pulled from a URL. These are super powerful because you have full control over any content you want.
  • commands: run a series of commands one at time in alphabetical order and even set a directory from which the command is run or leverage environment variables. You can also provide a test to control whether the command is executed or not. (Example test: If a file does not exist, command: the download command). This being executed after the File block is super helpful to execute that script if necessary.
  • services: Launch a list of sysvinit. This will launch a bunch of services at instance launch to ensure services are restarted when the files change or packages are updated by cfn-init.

One additional note, Fn::Sub or the shorthand !Sub (The sub function) allows you to use ${} to interpolate or "substitute" things like ${AWS::StackName} to pull References or Pseudo variables.

How do I invoke that cfn-signal?

After you have initialized your configuration using the cnf-init command, you can then use the cnf-signal command to let CloudFormation know that the resource creation has been successful. This is done in a CloudFormation CreationPolicy.

Here is an example:

    CreationPolicy:
        ResourceSignal:
            Timeout: PT5M

In the example above we are setting a maximum of 5 minutes wait time for the instance to come online and be self configured. If we don't hear back from cfn-signal by then, CloudFormation will fail and roll back to the previous version.

How do I invoke cfn-hup?

Cfn-hup can be used to tell the EC2 instance to look for MetaData changes every 15 minutes and apply the metadata configuration again.

What does it look like in practice?

        Resources:
            MyInstance:
                Type: "AWS::EC2::Instance"
                Metadata:
                    AWS::CloudFormation::Init:
                        config:
                            packages:
                                rpm:
                                    epel: "http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm"
                                yum:
                                    httpd: []
                                    php: []
                                    wordpress:[]
                                rubygems:
                                    chef:
                                      - "0.10.2"
                            groups:
                                groupOne: {}
                                groupTwo: 
                                    gid: "123"
                            users:
                                "apache":
                                    groups:
                                      - "groupOne"
                                      - "groupTwo"
                                    uid: "1234"
                                    homeDir: "/tmp"
                            sources:
                                "home/ec2-user/aws-cli": "https://github.com/aws/aws-cli/traball/master"
                                /etc/puppet: "https://github.com/user1/cfn-demo/tarball/master"
                            files:
                                "/tmp/setup.mysql":
                                    content: !Sub |
                                        CREATE DATABASE ${DBName};
                                        CREATE USER '${DBUsername}'@'localhost' IDENTIFIED BY '${DBPassword}';
                                        GRANT ALL ON ${DBName}.* TO `${DBUsername}'@'localhost';
                                        FLUSH PRIVILEGES;
                                    mode: "000644"
                                    owner: "root"
                                    group: "root"
                                "/etc/cfn/cfn-hup.conf":
                                    content: !Sub|
                                        [main]
                                        stack=${AWS::StackId}
                                        region=${AWS::Region}
                                        mode: "000400"
                                        owner: "root"
                                        group: groupOne
                                "/etc/cfn/hooks.d/cfn-auto-reloader.conf":
                                    content: !sub | 
                                        [cfn-auto-reloader-hook]
                                        triggers=post.update
                                        path=Resources.WebserverHost.Metadata.AWS::CloudFormation::Init
                                        action=/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource webServerHost
                            commands:
                                test:
                                    command: "echo \"$MAGIC\" > test.txt"
                                    env:
                                        MAGIC: "I come from an environment!"
                                    cwd: "~"
                                    test: "test ! -e ~/test.txt"
                                    ignoreErrors: "False"
                            services:
                                sysvinit:
                                    nginx:
                                        enabled: "true"
                                        ensureRunning: "true"
                                        files:
                                          - "etc/nginx/nginx.conf"
                                        sources:
                                          - "/var/www/html"
                                    sendmail:
                                        enabled: "false"
                                        ensureRunning: "false"
                Creation Policy:
                    ResourceSignal:
                        Timeout: PT5M
                Properties:
                    ImageId: ami-a4c7edb2
                    KeyName:
                        Ref: KeyName
                    InstanceType: t2.micro
                    SecurityGroups:
                    - Ref: WebserverSecurityGroup
                    UserData:
                        "Fn::Base64":
                            !Sub|
                                #!/bin/bash -xe
                                # Get the latest CloudFormation Package
                                yum update -y aws-cfn-bootstrap 
                                # start cfn-init
                                /opt/aws/bin/cfn-init -s ${AWS::StackID} -r WebserverHost --region ${AWS::Region}
                                # start the cfn-hup daemon to listen for changes to the ec2 instance metadata || error_exit "Failed cfn-init"
                                /opt/aws/bin/cfn-hup || error_exit "Failed cfn-hup"
                                /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackId} --resource WebServerHost --region ${AWS::Region}
        ...
    Resources:
        MyInstance:
            Type: "AWS::EC2::Instance"
            Metadata:
                AWS::CloudFormation::Init:
                    config:
                        packages:
                            rpm:
                                epel: "http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm"
                            yum:
                                httpd: []
                                php: []
                                wordpress:[]
                            rubygems:
                                chef:
                                  - "0.10.2"
                        groups:
                            groupOne: {}
                            groupTwo: 
                                gid: "123"
                        users:
                            "apache":
                                groups:
                                  - "groupOne"
                                  - "groupTwo"
                                uid: "1234"
                                homeDir: "/tmp"
                        sources:
                            "home/ec2-user/aws-cli": "https://github.com/aws/aws-cli/traball/master"
                            /etc/puppet: "https://github.com/user1/cfn-demo/tarball/master"
                        files:
                            "/tmp/setup.mysql":
                                content: !Sub |
                                    CREATE DATABASE ${DBName};
                                    CREATE USER '${DBUsername}'@'localhost' IDENTIFIED BY '${DBPassword}';
                                    GRANT ALL ON ${DBName}.* TO `${DBUsername}'@'localhost';
                                    FLUSH PRIVILEGES;
                                mode: "000644"
                                owner: "root"
                                group: "root"
                            "/etc/cfn/cfn-hup.conf":
                                content: !Sub|

I'm not entirely sure all of this example above will work, in fact I'd expect it wouldn't. The point of the example is to show you the various uses of cfn-init from the Metadata and then how to run it in your EC2 properties with solid examples of each section.

Logs for ec2-user data are in /var/log/cloud-init-output.log and logs for cfn-init are in /var/log/cfn-init.log. This can be really helpful if commands don't complete like you want them to.

CloudFormation Drift

CloudFormation allows you to create infrastructure but does not protect you from manual configuration changes.

How do we know if our resources have drifted from our declarative state?

A great resource on this topic can be located at https://docs.aws.amazon.com/AWSCloudFormation/Latest/UserGuide/using-cfn-stack-drift-resource-list.html

If you create a CloudFormation security group stack for example and then went into the security group and deleted it.

On the CloudFormation stacks if you select your newly created security group stack and select options > detect drift. It will change from "NOT CHECKED" to "DRIFTED". If you view the drift details, you will see that something has changed, what has changed, and when.

This can be super helpful in identifying troubles that may have happened after a manual change.

Nested Stacks

What is a Nested Stack?

Stacks that are part of other stacks, seems obvious right?

They allow you to isolate repeated patterns or common components in separate stacks and call them from other stacks. A load balancer configuration that is re-used or a Security Group is a good example.

How do I take advantage of nested stacks?

The easiest way to explain this is that by leveraging !GetAtt and your Outputs you can reference other stacks. You can also leverage an S3 URL with your cloudformationTemplateUrl. When you create the stack, the nested stacks will also be created.

When you make updates, you can upload them to your S3 bucket and then on your CloudFormation stack you can click Actions > update stack > select from S3.

It's important to know that you should never edit, delete, or apply changes to the nested stack directly. Always do changes to the top level stack.

When you delete the top level stack it will delete the nested stacks automatically.

A lot of people don't like nested stacks.

There are a lot of great nested stack examples at https://github.com/aws-samples/ecs-refarch-cloudformation/blob/master/master.yaml

When should you used Output Values over using Nested Stacks?

You will want to use Exported Stack Output Values if:

  • You have a central resource that is shared between many different stacks or stacks.
  • You need other stacks to be updated right away if a central resource is updated.

You will want to use Nested Stacks if:

  • Your resources are dedicated to one stack only and must be re-usable pieces of code

If you need to update a Nested stack, every Root stack will need to be manually updated.

Personally, I'd rather just use Output values whenever possible and avoid this argument all together.

What is a good resource to review some of the things I've already learned?

The two resources above will be an excellent review of a real world example you can skim over to understand how to use everything we have already learned so far.

Advanced CloudFormation Concepts

What you already know about CloudFormation from this article is enough to be very effective already but if you'd like to learn how to leverage Troposphere (python) to generate templates or be able to used the AWS CLI with it, you can keep reading here.

How do I use the AWS CLI to deploy CloudFormation Templates?

We can use the AWS CLI to create, update, or delete CloudFormation templates.

It is a great way to manage parameters in a file and to automate deployments.

If you don't already have AWSCLI installed, you can do it with a terminal command: pip3 install awscli --upgrade --user.

You should now see output when you type in the command aws into terminal.

You will need to set up AWS Security credentials and then create a new Access Key in order to use AWSCLI, download the file and show the keys.

Now Run: aws configure --profile drewlearns and provide the AWS Access Key ID you just copied then the Access Key Secret.

You can set the default region name to "us-west-2" and default output format to "json".

The sample-template.yaml file can be downloaded from: https://s3.us-west-2.amazonaws.com/cloudformation-examples-us-west-2/AWSCloudFormation-samples.zip - use the "EC2InstanceWithSecurityGroupSample.template" file.

You will see this file is in json format, you'll need to open it in an editor and copy it's content.

Convert that file to yaml now by pasting it using https://www.json2yaml.com/ and update the file name to be sample-template.yaml.

I've also already done this at the end of this article in "AppendixA"

Now, we have to pass our parameters to select the correct parameters.

Create a new file in the same directory as the sample-template.yaml called parameters.json.

We want to pass it an array of key value pairs. Copy paste the following into it:

    [
        {
            "ParameterKey": "InstanceType",
            "ParameterValue": "t2.micro"
        },
        {
            "ParameterKey": "KeyName",
            "ParameterValue": "Drew"
        },
        {
            "ParameterKey": "SSHLocation",
            "ParameterValue": "0.0.0.0/0"
        }
    ]

Now we need to create our CloudFormation template using the following command:

Before you run the command, make sure that you are in the directory that has your parameters.json file and the sample-template.yaml files.

aws cloudformation create-stack --stack-name example-cli-stack --template-body file://sample-template.yaml --parameters file://parameters.json --profile drewlearns --region us-west-2

The output of the command will be a key value pair in json format of the "StackId" which you can reference in AWS CloudFormation dashboard if you'd like.

There are some optional options you can pass in the above command:

  • --disable-rollback
  • --rollback-configuration <value>
  • --timeout-in-minutes
  • --capabilities <value>
  • --resource-types <value>
  • --role-arn <value>
  • --on-failure-<value>
  • --stack-policy-body <value>
  • --stack-policy-url <value>
  • --tags <value>
  • --client-request-token <value>
  • --enable-termination-protection
  • --cli-input-json <value>
  • --generate-cli-skeleton <value>

How do I prevent some resources from being deleted using CloudFormation?

With the DeletionPolicy attribute you can prevent resources from being destroyed or in some cases back them up before they are deleted.

DeletionPolicy can take the following values:

  • Delete: AWS CloudFormation will delete the resource and all of it's content if applicable during stack deletion. Does not apply to S3 Resources
  • Retain: AWS CloudFormation keeps the resource without deleting it or it's contents when the stack is deleted. This can be added to any resource type.
  • Snapshot: For resources that support SnapShots such as EC2 Volumes, ElastiCache, RDS, and the like - it will create a snapshot of the resource before deletion

What if I like to code in Python?

If you like to code in Python3 there is a tool called "Troposphere" that we can use to generate CloudFormation Templates! There is also a similar tool for GoLang called GoFormation I think.

Here is a link to the Troposphere github: https://github.com/cloudtools/troposphere

It has a ton of advantages! You can start using types in your templates, the generated CloudFormation will be valid, you can dynamically generate CloudFormation and you're allowed to leverage complex conditions to generate exactly what you need.

The disadvantage here is that it's a two step process to write CloudFormation. You'll first write your python code, then json, then upload.

Also writing in json instead of yaml isn't really my favorite. This used to be the only option now there is .to_yaml() function which cuts down on the steps.

This process also requires an understanding of Python which can be a harder hurdle to overcome compared to writing YAML.

If you don't already know python though, you are in luck! My previous article will teach you python from scratch! Here is a link: https://codingwithdrew.com/drew-learns-python-and-so-can-you/

To install Troposphere, just run pip3 install troposphere and follow the documentation in the read me.

After you write your CloudFormation in Python, you just run the file using python3 name_of_file.py and it will output the exact CloudFormation YAML that you need to copy paste into CloudFormation.

A lot of this process can be automated.

What are Custom Resources with AWS Lambda and how do I use them?

Custom resources enable you to write your own custom logic for provisioning templates. These will run anytime you create, update or delete stacks. A use case would be if you wanted to include resources that are not available as AWS CloudFormation Resource Types. Your Custom resources can be provisioned using AWS Lambda functions.

For great documentation on this, check out http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/walkthrough-custom-resources-lambda-lookup-amiids.html

This walk-through is rather lengthy but well worth running through.

Appendix A:

    ---
    AWSTemplateFormatVersion: '2010-09-09'
    Description: 'AWS CloudFormation Sample Template EC2InstanceWithSecurityGroupSample:
    Create an Amazon EC2 instance running the Amazon Linux AMI. The AMI is chosen based
    on the region in which the stack is run. This example creates an EC2 security group
    for the instance to give you SSH access. **WARNING** This template creates an Amazon
    EC2 instance. You will be billed for the AWS resources used if you create a stack
    from this template.'
    Parameters:
    KeyName:
        Description: Name of an existing EC2 KeyPair to enable SSH access to the instance
        Type: AWS::EC2::KeyPair::KeyName
        ConstraintDescription: must be the name of an existing EC2 KeyPair.
    InstanceType:
        Description: WebServer EC2 instance type
        Type: String
        Default: t2.small
        AllowedValues:
        - t1.micro
        - t2.nano
        - t2.micro
        - t2.small
        - t2.medium
        - t2.large
        - m1.small
        - m1.medium
        - m1.large
        - m1.xlarge
        - m2.xlarge
        - m2.2xlarge
        - m2.4xlarge
        - m3.medium
        - m3.large
        - m3.xlarge
        - m3.2xlarge
        - m4.large
        - m4.xlarge
        - m4.2xlarge
        - m4.4xlarge
        - m4.10xlarge
        - c1.medium
        - c1.xlarge
        - c3.large
        - c3.xlarge
        - c3.2xlarge
        - c3.4xlarge
        - c3.8xlarge
        - c4.large
        - c4.xlarge
        - c4.2xlarge
        - c4.4xlarge
        - c4.8xlarge
        - g2.2xlarge
        - g2.8xlarge
        - r3.large
        - r3.xlarge
        - r3.2xlarge
        - r3.4xlarge
        - r3.8xlarge
        - i2.xlarge
        - i2.2xlarge
        - i2.4xlarge
        - i2.8xlarge
        - d2.xlarge
        - d2.2xlarge
        - d2.4xlarge
        - d2.8xlarge
        - hi1.4xlarge
        - hs1.8xlarge
        - cr1.8xlarge
        - cc2.8xlarge
        - cg1.4xlarge
        ConstraintDescription: must be a valid EC2 instance type.
    SSHLocation:
        Description: The IP address range that can be used to SSH to the EC2 instances
        Type: String
        MinLength: '9'
        MaxLength: '18'
        Default: 0.0.0.0/0
        AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
        ConstraintDescription: must be a valid IP CIDR range of the form x.x.x.x/x.
    Mappings:
    AWSInstanceType2Arch:
        t1.micro:
        Arch: HVM64
        t2.nano:
        Arch: HVM64
        t2.micro:
        Arch: HVM64
        t2.small:
        Arch: HVM64
        t2.medium:
        Arch: HVM64
        t2.large:
        Arch: HVM64
        m1.small:
        Arch: HVM64
        m1.medium:
        Arch: HVM64
        m1.large:
        Arch: HVM64
        m1.xlarge:
        Arch: HVM64
        m2.xlarge:
        Arch: HVM64
        m2.2xlarge:
        Arch: HVM64
        m2.4xlarge:
        Arch: HVM64
        m3.medium:
        Arch: HVM64
        m3.large:
        Arch: HVM64
        m3.xlarge:
        Arch: HVM64
        m3.2xlarge:
        Arch: HVM64
        m4.large:
        Arch: HVM64
        m4.xlarge:
        Arch: HVM64
        m4.2xlarge:
        Arch: HVM64
        m4.4xlarge:
        Arch: HVM64
        m4.10xlarge:
        Arch: HVM64
        c1.medium:
        Arch: HVM64
        c1.xlarge:
        Arch: HVM64
        c3.large:
        Arch: HVM64
        c3.xlarge:
        Arch: HVM64
        c3.2xlarge:
        Arch: HVM64
        c3.4xlarge:
        Arch: HVM64
        c3.8xlarge:
        Arch: HVM64
        c4.large:
        Arch: HVM64
        c4.xlarge:
        Arch: HVM64
        c4.2xlarge:
        Arch: HVM64
        c4.4xlarge:
        Arch: HVM64
        c4.8xlarge:
        Arch: HVM64
        g2.2xlarge:
        Arch: HVMG2
        g2.8xlarge:
        Arch: HVMG2
        r3.large:
        Arch: HVM64
        r3.xlarge:
        Arch: HVM64
        r3.2xlarge:
        Arch: HVM64
        r3.4xlarge:
        Arch: HVM64
        r3.8xlarge:
        Arch: HVM64
        i2.xlarge:
        Arch: HVM64
        i2.2xlarge:
        Arch: HVM64
        i2.4xlarge:
        Arch: HVM64
        i2.8xlarge:
        Arch: HVM64
        d2.xlarge:
        Arch: HVM64
        d2.2xlarge:
        Arch: HVM64
        d2.4xlarge:
        Arch: HVM64
        d2.8xlarge:
        Arch: HVM64
        hi1.4xlarge:
        Arch: HVM64
        hs1.8xlarge:
        Arch: HVM64
        cr1.8xlarge:
        Arch: HVM64
        cc2.8xlarge:
        Arch: HVM64
    AWSInstanceType2NATArch:
        t1.micro:
        Arch: NATHVM64
        t2.nano:
        Arch: NATHVM64
        t2.micro:
        Arch: NATHVM64
        t2.small:
        Arch: NATHVM64
        t2.medium:
        Arch: NATHVM64
        t2.large:
        Arch: NATHVM64
        m1.small:
        Arch: NATHVM64
        m1.medium:
        Arch: NATHVM64
        m1.large:
        Arch: NATHVM64
        m1.xlarge:
        Arch: NATHVM64
        m2.xlarge:
        Arch: NATHVM64
        m2.2xlarge:
        Arch: NATHVM64
        m2.4xlarge:
        Arch: NATHVM64
        m3.medium:
        Arch: NATHVM64
        m3.large:
        Arch: NATHVM64
        m3.xlarge:
        Arch: NATHVM64
        m3.2xlarge:
        Arch: NATHVM64
        m4.large:
        Arch: NATHVM64
        m4.xlarge:
        Arch: NATHVM64
        m4.2xlarge:
        Arch: NATHVM64
        m4.4xlarge:
        Arch: NATHVM64
        m4.10xlarge:
        Arch: NATHVM64
        c1.medium:
        Arch: NATHVM64
        c1.xlarge:
        Arch: NATHVM64
        c3.large:
        Arch: NATHVM64
        c3.xlarge:
        Arch: NATHVM64
        c3.2xlarge:
        Arch: NATHVM64
        c3.4xlarge:
        Arch: NATHVM64
        c3.8xlarge:
        Arch: NATHVM64
        c4.large:
        Arch: NATHVM64
        c4.xlarge:
        Arch: NATHVM64
        c4.2xlarge:
        Arch: NATHVM64
        c4.4xlarge:
        Arch: NATHVM64
        c4.8xlarge:
        Arch: NATHVM64
        g2.2xlarge:
        Arch: NATHVMG2
        g2.8xlarge:
        Arch: NATHVMG2
        r3.large:
        Arch: NATHVM64
        r3.xlarge:
        Arch: NATHVM64
        r3.2xlarge:
        Arch: NATHVM64
        r3.4xlarge:
        Arch: NATHVM64
        r3.8xlarge:
        Arch: NATHVM64
        i2.xlarge:
        Arch: NATHVM64
        i2.2xlarge:
        Arch: NATHVM64
        i2.4xlarge:
        Arch: NATHVM64
        i2.8xlarge:
        Arch: NATHVM64
        d2.xlarge:
        Arch: NATHVM64
        d2.2xlarge:
        Arch: NATHVM64
        d2.4xlarge:
        Arch: NATHVM64
        d2.8xlarge:
        Arch: NATHVM64
        hi1.4xlarge:
        Arch: NATHVM64
        hs1.8xlarge:
        Arch: NATHVM64
        cr1.8xlarge:
        Arch: NATHVM64
        cc2.8xlarge:
        Arch: NATHVM64
    AWSRegionArch2AMI:
        af-south-1:
        HVM64: ami-064cc455f8a1ef504
        HVMG2: NOT_SUPPORTED
        ap-east-1:
        HVM64: ami-f85b1989
        HVMG2: NOT_SUPPORTED
        ap-northeast-1:
        HVM64: ami-0b2c2a754d5b4da22
        HVMG2: ami-09d0e0e099ecabba2
        ap-northeast-2:
        HVM64: ami-0493ab99920f410fc
        HVMG2: NOT_SUPPORTED
        ap-northeast-3:
        HVM64: ami-01344f6f63a4decc1
        HVMG2: NOT_SUPPORTED
        ap-south-1:
        HVM64: ami-03cfb5e1fb4fac428
        HVMG2: ami-0244c1d42815af84a
        ap-southeast-1:
        HVM64: ami-0ba35dc9caf73d1c7
        HVMG2: ami-0e46ce0d6a87dc979
        ap-southeast-2:
        HVM64: ami-0ae99b503e8694028
        HVMG2: ami-0c0ab057a101d8ff2
        ca-central-1:
        HVM64: ami-0803e21a2ec22f953
        HVMG2: NOT_SUPPORTED
        cn-north-1:
        HVM64: ami-07a3f215cc90c889c
        HVMG2: NOT_SUPPORTED
        cn-northwest-1:
        HVM64: ami-0a3b3b10f714a0ff4
        HVMG2: NOT_SUPPORTED
        eu-central-1:
        HVM64: ami-0474863011a7d1541
        HVMG2: ami-0aa1822e3eb913a11
        eu-north-1:
        HVM64: ami-0de4b8910494dba0f
        HVMG2: ami-32d55b4c
        eu-south-1:
        HVM64: ami-08427144fe9ebdef6
        HVMG2: NOT_SUPPORTED
        eu-west-1:
        HVM64: ami-015232c01a82b847b
        HVMG2: ami-0d5299b1c6112c3c7
        eu-west-2:
        HVM64: ami-0765d48d7e15beb93
        HVMG2: NOT_SUPPORTED
        eu-west-3:
        HVM64: ami-0caf07637eda19d9c
        HVMG2: NOT_SUPPORTED
        me-south-1:
        HVM64: ami-0744743d80915b497
        HVMG2: NOT_SUPPORTED
        sa-east-1:
        HVM64: ami-0a52e8a6018e92bb0
        HVMG2: NOT_SUPPORTED
        us-east-1:
        HVM64: ami-032930428bf1abbff
        HVMG2: ami-0aeb704d503081ea6
        us-east-2:
        HVM64: ami-027cab9a7bf0155df
        HVMG2: NOT_SUPPORTED
        us-west-1:
        HVM64: ami-088c153f74339f34c
        HVMG2: ami-0a7fc72dc0e51aa77
        us-west-2:
        HVM64: ami-01fee56b22f308154
        HVMG2: ami-0fe84a5b4563d8f27
    Resources:
    EC2Instance:
        Type: AWS::EC2::Instance
        Properties:
        InstanceType:
            Ref: InstanceType
        SecurityGroups:
        - Ref: InstanceSecurityGroup
        KeyName:
            Ref: KeyName
        ImageId:
            Fn::FindInMap:
            - AWSRegionArch2AMI
            - Ref: AWS::Region
            - Fn::FindInMap:
            - AWSInstanceType2Arch
            - Ref: InstanceType
            - Arch
    InstanceSecurityGroup:
        Type: AWS::EC2::SecurityGroup
        Properties:
        GroupDescription: Enable SSH access via port 22
        SecurityGroupIngress:
        - IpProtocol: tcp
            FromPort: '22'
            ToPort: '22'
            CidrIp:
            Ref: SSHLocation
    Outputs:
    InstanceId:
        Description: InstanceId of the newly created EC2 instance
        Value:
        Ref: EC2Instance
    AZ:
        Description: Availability Zone of the newly created EC2 instance
        Value:
        Fn::GetAtt:
        - EC2Instance
        - AvailabilityZone
    PublicDNS:
        Description: Public DNSName of the newly created EC2 instance
        Value:
        Fn::GetAtt:
        - EC2Instance
        - PublicDnsName
    PublicIP:
        Description: Public IP address of the newly created EC2 instance
        Value:
        Fn::GetAtt:
        - EC2Instance
        - PublicIp

Index: