Build and deploy static sites w/ Middleman, CircleCI and Amazon S3

Patrick Baselier – 24 December 2015
1482 words in about 6 minutes

Recently we migrated a number of sites from different CMS systems and made them static using Middleman, we switched to Contentful as CaaS and Netlify for building and hosting. Before coming up with this architecture, we investigated several alternatives. One of them was hosting static files on Amazon S3 and managing the deployment process with CircleCI. This article shows you how to set this up.

Architectural overview

Before we dive into the process of creating a site, setting up services and deploying continuously, let’s look at the following overview of how the different components are related to make sure it’s clear to you what you’ll be working on.

Developing the site is the only process that is done on your local machine (hence the solid line for the development box); all the others steps are facilitated by external services (dotted lines):

• The repo for our site is hosted on GitHub
• Changes pushed to GitHub will trigger a build process on CircleCI that generates a new collection of static files
• These files will then be automatically upload to Amazon S3 so a new site will be available to our end-users within seconds

You will need to have accounts for GitHub, CircleCI and Amazon S3 but all services are free (in some form). Middleman is open-source and you need to have Ruby and Git installed on your local machine.

Create a Middleman project

First we start with creating a project for our site. We use Middleman for this, although I expect Jekyll or Octopress will do as well. Open up a terminal, install Middleman and create your project. You can answer all questions asked by Middleman with n, since this not related to the subject of deploying and hosting:

1
2
3
4
5
$ gem install middleman
$ middleman version
Middleman 4.0.0
$ middleman init my-site
$ middleman s

Now when you open a new tab in your browser and navigate to http://localhost:4567 you’ll see that your Middleman site running.

Create a GitHub repo for your project

Your project needs to be added to GitHub so we can integrate it with CircleCI later on. Assuming you have created a GitHub account, let’s define a repo and push the project to it:

• Navigate to https://github.com/new a define a new repo. Consult the GitHub Help for the most recent instructions on this. In your terminal, add the following commands to create a local repo and push it to GitHub (make sure you replace <github-account> with your GitHub account):

1
2
3
4
5
$ git init
$ git add .
$ git commit -m "First commit"
$ git remote add origin git@github.com:<github-account>/my-site.git
$ git push -u origin master

Build the site

Building your site means generating static files by running Middleman’s build command. Although the build process will be run by CircleCI eventually, it’s a good practice to see this happen on your local machine.

1
$ middleman build

Notice how Middleman created a my-site/build subfolder that contains a .html file and some subfolders with assets. By the way, Middleman takes care of not adding the build folder to the repository, so you don’t have to worry about that yourself (if you’re interested: open the .gitignore file from the my-site folder to see that the /build folder is listed here).

Host your site on Amazon S3

Let’s setup a bucket on Amazon S3 so we can use it to host your static files put in the build folder. The Amazon site has a good step-by-step instruction on how to do this, so please follow this. You will probably need to choose a different name for your bucket than my-site because a bucket’s name needs to be unique globally. Don’t forget to use this name when defining the bucket policy!

When you have your bucket, let’s use the files from the build folder and upload them.

In the S3 Management Console, select the bucket to open it and click the Upload button.

Drag and drop all the files and folders from the build folder to S3 Console and click Start Upload.

To view your site, open a new tab and enter the url for your static site. You can retrieve the url by choosing the Properties button and opening the Static Website Hosting section. Do not close the S3 Console, we still need it later.

Use Middleman::S3Sync to upload to S3

Uploading the files to S3 needs to be automated so we can instruct CircleCI to invoke this step. Several solutions are available for this, we choose a gem called Middleman::S3Sync. Before adding this to our project, we need to create a user on Amazon that is granted to upload files to the bucket. Although Amazon has good instructions on how to accomplish this, it took me a while to figure it out and got it working, so I briefly guide you through the steps:

• Navigate to the IAM Management Console.
• Add a user (name it CircleCI) and write down or download the credentials. These are only provided once!
• Add the following policy for the user, replace my-site with the proper bucket name:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::my-site"
        },
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::my-site/*"
        }
    ]
}

Now we can setup our Middleman project. Again, the GitHub project has good instructions on this so it suffices to say:

1
2
3
4
5
6
7
8
9
10
11
12
13
# Gemfile
gem 'middleman-s3_sync', '~> 4.0.1.rc.3'
gem 'mime-types', '~> 3.0.0'

# config.rb
activate :s3_sync do |s3_sync|
  s3_sync.bucket                = 'my-site'
  s3_sync.region                = 'eu-west-1'
  s3_sync.aws_access_key_id     = 'AKIAxxx'
  s3_sync.aws_secret_access_key = 'YPDxxx'
end

$ middleman s3_sync

This last command should upload your static files from the build folder to S3. The values defined in the code above should match your settings. Consult your bucket for the details. The credentials are listed in the file you downloaded when creating the CircleCI user. Defining credentials is a bad practice and should never live in your codebase. We only needed here to test our upload process. Middleman::S3Sync support different ways of how credentials are defined, so it’s save to remove those 2 lines:

1
2
3
4
5
# config.rb
activate :s3_sync do |s3_sync|
  s3_sync.bucket                = 'my-site'
  s3_sync.region                = 'eu-west-1'
end

Use CircleCI for continuous deployment

Last step. We want our site to be build and uploaded to S3 every time a commit is pushed to GitHub. In real-life projects things will probably be more nuanced, you may have different environments for different purposes, you may want to run tests before the upload process can be started, etc. For now, we keep it simple.

When you use CircleCI for the first time, you need to grant CircleCI to access your GitHub repo’s. Both CircleCI and GitHub come with clear instructions on how to accomplish this.

Once have granted CircleCI, you can add a project to CircleCI that will be triggered on every push to GitHub:

1. In CircleCI, add a project
2. Select your GitHub account
3. Select your GitHub repo

CircleCI immediately starts building the project, but since we need to configure some settings first, the build will fail. Don’t worry about that for now. We will fix that right away:

• In CircleCI, open the project settings and select Environment variables
• Here you add the credentials needed for the upload to S3 (make sure you use CAPITALS when defining the names) you downloaded from Amazon.

Other CircleCI settings will be defined in a circle.yml file. Add this file to the root of your project:

1
2
3
4
5
6
7
8
machine:
  ruby:
    version: 2.1.2
deployment:
  production:
    branch: master
    commands:
      - bundle exec middleman s3_sync

Adding all these configuration settings does not change the content of the static files that your build will generate, so for illustrative purposes, open source/index.html.erb and change the title.

Now we should be ready to go. When we add, commit and push our changes to GitHub, it should automatically trigger a build on CircleCI which should update the site:

1
2
3
$ git add .
$ git commit -m "Setup CircleCI"
$ git push

When you look at your CircleCI dashboard, you will notice that a build is running. When finished, refresh the tab that has your site displayed to see that the changes are applied.

But there’s more…

Although we’ve only touched the surface of the tools used here, this already is a major step forward in automating the process of generating and deploying a static site.

You can do much more, so if this article got you interested, you definitely have to take a look at Middleman, CircleCI and Amazon S3. Or other tools we used such as Netlify, Contentful, EmberJS…

O man, if I only had more time.

Patrick Baselier

I’m a professional Ruby on Rails-, front-end- and unprofessional (that is: not professionally… yet) Ember developer from The Netherlands, I love sharing knowledge and one day I hope to be a more than a novice guitar player.