Jérôme Decoster

Jérôme Decoster

3x AWS Certified - Architect, Developer, Cloud Practionner

07 Jul 2020

Github Actions + Hugo + Terraform + S3

The Goal
  • Create a static blog with Hugo
  • Host it on S3 using a CloudFront CDN and a domain name managed by Route 53
  • Doing this manually from the web browser interface
  • Redoing the same thing with Terraform
  • Redoing it using Github Actions to automate the deployment of new posts
  • The GitOps approach makes it possible to update the site or the infrastructure according to the path of the commited files

    architecture.svg

    Install, explore and build the project

    Get the code from this github repository :

    # create a `tmp` directory in $HOME
    $ mkdir --parents $HOME/tmp
    
    # download the code
    $ git clone \
        --depth 1 \
        https://github.com/jeromedecoster/github-actions-hugo-terraform-s3.git \
        $HOME/tmp/aws
    
    # cd
    $ cd $HOME/tmp/aws
    

    We launch the site :

    $ make dev
    

    This script performs the following :

    $ hugo server \
    		--buildDrafts \
    		--disableFastRender
    

    We note the use of the --buildDrafts option which shows us all the posts, even the drafts, which will not appear in the build version.

    The website at http://localhost:1313 :

    site-dev-with-draft

    The draft: true attribute is defined in the Front Matter of the last 2 posts.

    Here is the example of the post black-and-brown-mallard-duck-surrounded-by-grass :

    ---
    title: "black and brown mallard duck surrounded by grass"
    description: "black and brown mallard duck surrounded by grass"
    date: "2020-01-04"
    tags: [black, brown]
    draft: true
    ---
    
    ![duck](duck.jpg)
    

    Here’s a blog content page :

    site-dev-content

    To see the site without the drafts we can do :

    $ make dev-nodraft
    

    This command simply do this :

    $ hugo server
    

    We see that the 2 most recent articles have disappeared :

    site-dev-without-draft

    Creating the S3 bucket

    I’m going to create a static S3 site that will be associated with my domain name jeromedecoster.net :

    hosted-zone

    I create a bucket www.jeromedecoster.net :

    s3-1-bucket-create

    I authorize all public access :

    s3-2-public-access

    The bucket is created :

    s3-3-bucket-created

    I activate the static website hosting and I indicate the index page and the error page :

    s3-4-static-site

    Creating the SSL certificate

    For my site to support HTTPS, I need to:

    1. Create an SSL certificate on AWS Certificate Manager.
    2. Create a CloudFront distribution and associate it with this certificate.

    In order to associate our SSL certificate with our CloudFront distribution, our certificate must be created in the us-east-1 region :

    certificate-1-us-east-1

    I start creating the certificate :

    certificate-2-request

    I enter 2 domain names jeromedecoster.net and *.jeromedecoster.net :

    certificate-3-domains

    I choose DNS validation method :

    certificate-4-validation

    The validation is now awaiting an action :

    certificate-5-pending

    I need to create a specific CNAME record in Route 53.

    I just need to click this button to validate the 2 domains at the same time :

    certificate-6-create-record

    I can see that a CNAME record was added in Route 53 after the click :

    certificate-7-route53

    After a few seconds, I refresh the interface to see that my certificate has been validated :

    certificate-8-issued

    Creation the CloudFront CDN

    We are now creating the CloudFront distribution :

    cloudfront-1-create

    Warning : for the Origin Domain Name parameter, you MUST NOT CHOOSE the S3 bucket listed in the drop-down list !

    cloudfront-2-list

    You must indicate the endpoint of the bucket given in the S3 properties :

    cloudfront-3-s3-endpoint

    The drop-down list suggest :

    • www.jeromedecoster.net.s3.amazonaws.com

    But you have to indicate :

    • www.jeromedecoster.net.s3-website.eu-west-3.amazonaws.com

    I also choose Redirect HTTP to HTTPS :

    cloudfront-4-origin-redirect

    For Alternate Domain Names, I indicate www.jeromedecoster.net.

    I choose Custom SSL Certificate and I select my certificate :

    cloudfront-5-ssl

    I leave the other parameters by default and I validate the creation of the distribution :

    cloudfront-6-in-progress

    I copy in my clipboard the Domain Name URL of my distribution :

    cloudfront-7-copy-id

    Finalization on Route 53

    I will now create a Record Set in Route 53 to route the traffic to my distribution :

    route53-1-create

    I indicate www and I create a type A Record Set.

    I choose Alias : Yes.

    I see that my CloudFront distribution does not appear in the drop-down list.

    This is not an error. It actually takes a while for this to become visible :

    route53-2-no-cloudfront

    I can however directly paste the URL Domain Name of my CloudFront distribution that I had copied to my clipboard a few steps above.

    I validate by clicking Create :

    route53-3-create-a

    I do the same thing again to create a type AAAA Record Set :

    route53-4-create-aaaa

    Putting the site online

    I will now generate the static site with Hugo :

    $ make build
    

    This command simply executes :

    $ rm public \
    		--force \
    		--recursive
    $ hugo
    

    Files are generated in the public directory :

    # -L level : Max display depth of the directory tree.
    # -F : Append a `/' for directories
    # --dirsfirst : List  directories  before  files.
    $ tree public -L 1 -F --dirsfirst
    public
    ├── categories/
    ├── css/
    ├── page/
    ├── posts/
    ├── tags/
    ├── index.html
    ├── index.xml
    └── sitemap.xml
    

    I will now upload these files to my bucket :

    $ make upload
    

    This command simply executes :

    $ cd public
    $ aws s3 sync --acl public-read . s3://www.$APEX_DOMAIN
    

    The variable $APEX_DOMAIN is declared in the make.sh file :

    #
    # variables
    #
    
    # AWS variables
    AWS_PROFILE=default
    AWS_REGION=eu-west-3
    # project name
    PROJECT_NAME=github-actions-hugo
    # apex domain name
    APEX_DOMAIN=jeromedecoster.net
    

    If I display the URL https://www.jeromedecoster.net in my browser :

    site-dev-without-draft

    Using Terraform

    We have just seen all the manual steps necessary to put a static site online.

    We are now going to recreate this site identically using Terraform and aws cli in order to automate this task.

    I start by destroying all my previously created resources :

    • The CloudFront distribution
    • The SSL certificate
    • The S3 bucket
    • The CNAME, A and AAAA Record Sets

    Important: I choose to NOT MANAGE the creation of the SSL certificate by Terraform but to do it with aws cli. I also do NOT CREATE a Hosted Zone with Terraform.

    Even if the creation / destruction of these resources is technically easy to do with Terraform, it seems preferable to manage it manually or via a bash script for the following reasons:

    1. You probably already have an existing a hosted zone. It is therefore not necessary to recreate another dynamically. Especially since each hosted area is billed monthly.

    2. Il est probable que vous possédiez déjà une hosted zone. Il n’est donc pas nécessaire d’en recréer une supplémentaire dynamiquement. D’autant que chaque hosted zone est facturée mensuellement.

    3. Creating a certificate to associate it with a hosted zone is not a recurring task. Once created, there is little chance that you will want to delete it. We therefore avoid removing it unnecessarily by executing terraform destroy.

    I create my certificate :

    $ make create-certificate
    

    This command executes this script :

    # create the certificate in us-east-1
    $ CERTIFICATE_ARN=$(aws acm request-certificate \
        --domain-name $APEX_DOMAIN \
        --subject-alternative-names *.$APEX_DOMAIN \
        --validation-method DNS \
        --query CertificateArn \
        --region us-east-1 \
        --output text)
    
    # get the CNAME datas for the DNS valitaion
    $ RESOURCE_RECORD=$(aws acm describe-certificate \
            --certificate-arn $CERTIFICATE_ARN \
            --query 'Certificate.DomainValidationOptions[0].ResourceRecord' \
            --region us-east-1)
    
    $ CNAME_NAME=$(echo "$RESOURCE_RECORD" | jq --raw-output '.Name')
    $ CNAME_VALUE=$(echo "$RESOURCE_RECORD" | jq --raw-output '.Value')
    
    # create the change Record Set file
    $ cat > CNAME.json << EOF
    {
      "Comment": " ",
      "Changes": [
        {
          "Action": "CREATE",
          "ResourceRecordSet": {
            "Name": "$CNAME_NAME",
            "Type": "CNAME",
            "TTL": 600,
            "ResourceRecords": [
              {
                "Value": "$CNAME_VALUE"
              }
            ]
          }
        }
      ]
    }
    EOF
    
    # get the hosted zone id
    $ HOSTED_ZONE_ID=$(aws route53 list-hosted-zones-by-name \
        --dns-name $APEX_DOMAIN \
        --profile $AWS_PROFILE \
        --query 'HostedZones[0].Id' \
        --output text)
    
    # add the CNAME record set
    $ aws route53 change-resource-record-sets \
        --hosted-zone-id $HOSTED_ZONE_ID \
        --change-batch file://CNAME.json
    

    We don’t want Terraform State files to be on our machine. We want them to be hosted on S3.

    We define it in the main.tf file :

    terraform {
      # 'backend-config' options must be passed like :
      # terraform init -input=false -backend=true \
      #   [with] -backend-config="backend.json"
      #     [or] -backend-config="backend.tfvars"
      #     [or] -backend-config="<key>=<value>"
      backend "s3" {}
    }
    

    So we create a specific bucket to receive the Terraform State files :

    $ make tf-setup-backend
    

    This command simply executes :

    # create a random name
    $ S3_BACKEND=$PROJECT_NAME-$(mktemp --dry-run XXXX | tr '[:upper:]' '[:lower:]')
    
    # write this name as a variable in a settings.sh file
    $ echo "S3_BACKEND=$S3_BACKEND" > settings.sh
    
    # create the bucket
    $ aws s3 mb s3://$S3_BACKEND --region $AWS_REGION
    

    We can now initialize Terraform :

    $ make tf-init
    

    This command simply executes :

    # set $S3_BACKEND
    $ source settings.sh
    
    # initialize terraform
    $ terraform init \
        -input=false \
        -backend=true \
        -backend-config="region=$AWS_REGION" \
        -backend-config="bucket=$S3_BACKEND" \
        -backend-config="key=terraform" \
        -reconfigure
    

    Let’s take a look at some excerpts from Terraform files.

    We use a trick to create resources in 2 different regions in main.tf :

    provider aws {
      region  = var.region
      profile = var.profile
    }
    
    provider aws {
      region  = "us-east-1"
      profile = var.profile
      alias   = "us_east_1"
    }
    

    I create a Record Set using an already existing Hosted Zone in route53.tf :

    data aws_route53_zone zone {
      name = var.apex_domain
    }
    
    resource aws_route53_record A {
      zone_id = data.aws_route53_zone.zone.zone_id
      name    = "www.${var.apex_domain}"
      type    = "A"
    
      alias {
        name                   = aws_cloudfront_distribution.distribution.domain_name
        zone_id                = aws_cloudfront_distribution.distribution.hosted_zone_id
        evaluate_target_health = false
      }
    }
    

    Creating the bucket for static website hosting in s3.tf :

    resource aws_s3_bucket bucket {
      bucket        = "www.${var.apex_domain}"
      force_destroy = true
      acl           = "public-read"
    
      website {
        index_document = "index.html"
        error_document = "error.html"
      }
    }
    
    data aws_iam_policy_document policy {
      statement {
        principals {
          type        = "*"
          identifiers = ["*"]
        }
        actions   = ["s3:GetObject"]
        resources = ["${aws_s3_bucket.bucket.arn}/*"]
      }
    }
    
    resource aws_s3_bucket_policy bucket_policy {
      bucket = aws_s3_bucket.bucket.id
      policy = data.aws_iam_policy_document.policy.json
    }
    

    Creation of the CloudFront distribution in cloudfront.tf :

    resource aws_cloudfront_distribution distribution {
      provider = aws.us_east_1
    
      origin {
        domain_name = aws_s3_bucket.bucket.website_endpoint
        origin_id   = aws_s3_bucket.bucket.website_endpoint
    
        custom_origin_config {
          http_port              = 80
          https_port             = 443
          origin_protocol_policy = "http-only"
          origin_ssl_protocols   = ["TLSv1", "TLSv1.1", "TLSv1.2"]
        }
      }
    
      enabled             = true
      is_ipv6_enabled     = true
      comment             = ""
      default_root_object = "index.html"
    
      aliases = ["www.${var.apex_domain}"]
    
      default_cache_behavior {
        allowed_methods  = ["GET", "HEAD"]
        cached_methods   = ["GET", "HEAD"]
        target_origin_id = aws_s3_bucket.bucket.website_endpoint
        compress         = true
    
        forwarded_values {
          query_string = false
    
          cookies {
            forward = "none"
          }
        }
    
        viewer_protocol_policy = "redirect-to-https"
        min_ttl                = 0
        default_ttl            = 86400
        max_ttl                = 31536000
      }
    
      viewer_certificate {
        acm_certificate_arn      = data.aws_acm_certificate.cert.arn
        ssl_support_method       = "sni-only"
        minimum_protocol_version = "TLSv1.2_2018"
      }
    
      restrictions {
        geo_restriction {
          restriction_type = "none"
        }
      }
    }
    

    Now we can create the infrastructure :

    $ make tf-apply
    

    Then upload the files :

    $ make upload
    

    If I display the URL https://www.jeromedecoster.net in my browser :

    site-dev-without-draft

    Website update with Github actions

    I create a workflow to rebuild the site and deploy it automatically after each update with cd.yml :

    Let’s look at some excerpts.

    The workflow will only be triggered if files are modified after a push action in these directories :

    name: cd
    
    on:
      push:
        paths:
          - "content/**"
          - "extra/**"
          - "layouts/**"
          - "static/**"
    

    We use the peaceiris/actions-hugo action to install Hugo.

    Then we generate the website very simply :

    - name: Setup Hugo
      uses: peaceiris/actions-hugo@v2
    
    - name: Build with Hugo
      run: hugo --minify
    

    We deploy the files in the S3 bucket:

    - name: Deploy static site to S3 bucket
      run: |
        cd public
        aws s3 sync \
          --delete \
          --acl public-read \
          . s3://www.jeromedecoster.net
    

    We launch a CloudFront invalidation on specific files to update our site quickly :

    - name: Cloudfront invalidation
      run: |
        aws cloudfront create-invalidation \
          --distribution-id ${{ secrets.DISTRIBUTION_ID }} \
          --paths "/" \
          "/index.html" \
          "/posts/" \
          "/posts/index.html" \
          "/tags/" \
          "/tags/index.html"
    

    I create project specific AWS accesses :

    $ create-user
    

    This command executes this script :

    # create the user
    $ aws iam create-user \
            --user-name $PROJECT_NAME
    
    # attach a PowerUserAccess policy
    $ aws iam attach-user-policy \
            --user-name $PROJECT_NAME \
            --policy-arn arn:aws:iam::aws:policy/PowerUserAccess
    
    # ... get the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY values ...
    
    # write these values in a secrets.sh file
    cat > secrets.sh << EOF
    AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
    AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
    EOF
    

    I add some secret variables to my github project :

    actions-secrets

    I modify a post that had remained in draft: true.

    I set now draft: false :

    actions-draft-false

    After the commit, the action is executed quickly :

    actions-cd

    I see the new blog post, my website is updated successfully :

    actions-site-updated

    Infrastructure update with Github actions

    I create a workflow to update the infrastructure with Terraform in infra.yml.

    Let’s look at some excerpts.

    The workflow will only be triggered if files are modified after a push action in this directory :

    name: infra
    
    on:
      push:
        paths:
          - "infra/**"
    

    Some of the variables used by Terraform are defined :

    env:
      AWS_ACCESS_KEY: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      AWS_REGION: eu-west-3
      TF_VAR_apex_domain: jeromedecoster.net
    

    Terraform is installed using the hashicorp/setup-terraform action, and then initialized with the remote state backend :

    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v1.1.0
    
    - name: Terraform Init
      timeout-minutes: 2
      run: |
        terraform init \
            -input=false \
            -backend=true \
            -backend-config="region=${{ env.AWS_REGION }}" \
            -backend-config="bucket=${{ secrets.S3_BACKEND }}" \
            -backend-config="key=terraform" \
            -reconfigure
    

    We apply the transformations :

    - name: Terraform Plan
      run: terraform plan -out=terraform.plan
    
    - name: Terraform Apply
      run: terraform apply -auto-approve terraform.plan
    

    To test this action I create an additional bucket by uncommenting these lines in the s3.tf file :

    # resource aws_s3_bucket bucket_test {
    #  bucket        = "test.${var.apex_domain}"
    #  force_destroy = true
    # }
    

    This bucket is useless for my project, it’s just to test the process.

    The action is quickly executed successfully :

    actions-infra

    The bucket has been created :

    actions-bucket-created