AWS and other interesting stuff

Elastic Beanstalk

In this article I’ll document the things I find out about Elastic Beanstalk as I test functionality.

Introduction

Terminology

  • Application
    • A collection of components like the environments, versions and configurations
  • Application Version
    • Part of an application
    • Each version is unique, but applications can have multiple versions
    • You can deploy multiple versions to test them
  • Environment
    • A hosting environment
    • Each environment has a URL
    • Scaling
      • You can set the type, and auto scaling settings: AZ, min, max, desired capacity, cooldown.
      • The type can be:
        • Load balancing, auto scaling
          • or
        • Single instance
      • Scaling Triggers can be set (which sets ASG scaling policies)
      • Time-based scaling can be configured
      • When load balancing, auto scaling is set you can set deployment preferences:
        • Ignore health check
        • Batch size (percentage or fixed)
  • Environment Configuration
    • Settings and parameters that define the environment and resources
  • Configuration template
    • You can save a configuration as a template
    • Used to create repeatable environment configurations

By default, Elastic Beanstalk creates scale-in and scale-out policies on NetworkOut

Environment Interface:

Each environment has the following options:

  • Dashboard
    • Health
    • Upload and Deploy
    • Configuration
    • Recent Events
  • Configuration
  • Logs - You can request logs, either the last 100 lines or the full logs, then download them.
    • http error and access logs
    • eb-activity log
  • Health - overall health and per instance
    • R/sec, 2xx, 3xx, 4xx, 5xx responses
    • Latency P99, P90, P75, P50, P10
    • Load Average
    • CPU utilization %
    • User
    • Sys
    • Idle
    • I/O Wait
  • Monitoring
    • Overview
    • Healthy Host Count
    • CPU, Latency, Requests, Network (In, Out)
    • More …
    • Time period can be set
    • Graphs for:
      • Environment health (by health code)
      • CPU Utilization (in percent)
      • Max Network In (in bytes)
      • Max Network Out (in bytes)
      • More … this page can be edited to add other graphs e.g. Auto Scaling Group > Network Out > Maximum
      • Time range can be set
  • Alarms
    • You can create custom alarms by clicking on the bell icon in for the graph on the Monitoring page
  • Managed Updates
    • Software and feature updates
  • Events
    • A log of Elastic Beanstalk events
  • Tags

Deploy:

  • Each deploy needs a unique Version Label.
  • Once you’ve deployed from the Dashboard you can use the Application Versions dropdown menu to deploy old versions.

Supported Platforms

  • Java
  • .NET
  • PHP
  • Node.js
  • Python
  • Ruby
  • Go

Webservers and Containers

  • Apache
  • IIS
  • Java SE
  • Nginx
  • Passenger
  • Puma
  • Tomcat
  • Docker

Supported Deployment Platforms

  • Git
  • IDEs (Eclipse, Visual Studio)
  • manual upload (WAR or ZIP files)

When To Use Elastic Beanstalk

  • Spend minimal time learning and setting up infrastructure
  • Quick prototyping and testing
  • Shorter application lifecycles

When Not To Use Elastic Beanstalk

  • When you need complete control over resource configurations
  • Existing applications can be difficult to fit in the Elastic Beanstalk model
  • Installing dependencies is different than simply using apt-get and can complicate things if you have a lot of dependencies.

Deployment

There are settings for both Application Deployments …

… and Configuration Updates:

  • Rolling based on health
    • Elastic Beanstalk waits until all instances in a batch pass health checks before moving to the next batch.
    • With basic health, an instance is consider healthy once it passes its ELB health checks
    • With enhanced, instance must pass 12 health checks over the course of 2 minutes (18 checks over 3 minutes for worker environments)
    • If a batch does not become healthy within the rolling update timeout (default is 30 minutes), the update is cancelled.
    • If the rolling update process fails, Elastic Beanstalk starts another rolling update to rollback to the previous configuration.
  • Rolling based on time
    • You set a pause time, and rather than checking health updates just pause between batches.

Blue Green Deployment

Blue Green deployment in a Elastic Beanstalk context means cloning an environment and swapping environment URLs.

Create Some Sample Source Bundles

I built blue_green_php_app-v1.0.zip, blue_green_php_app-v1.1.zip and blue_green_php_app-v1.2.zip using Git tags:

VERSION=v1.0 && \
  cd sample_blue_green_php_app && \
  git checkout tags/$VERSION && \
  zip -r ../blue_green_php_app-$VERSION.zip . -x ./*.git/* && \
  git checkout master && \
  cd ..

Create a bucket to store the bundles:

Note:

  • S3 region must be the same as the Elastic Beanstalk region
  • Elastic Beanstalk also supports CodeCommit (e.g. use SourceLocation to point to a specific commit):
$ aws s3api create-bucket --bucket blue-green-php-app-source --region ap-southeast-2

Upload the bundles

$ for file in $(ls -1 blue_green_php_app-v1.*.zip); do aws s3 cp $file s3://blue-green-php-app-source/; done

When you upload a bundle, you set a version label, e.g. v1.0

Setup

Create an application:

$ aws elasticbeanstalk create-application --application-name sample-blue-green-php-app

Create a simple single instance production environment in the application 1:

$ aws elasticbeanstalk create-environment --application-name sample-blue-green-php-app --environment-name production --solution-stack-name "64bit Amazon Linux 2016.09 v2.2.0 running PHP 7.0"

Create a simple single instance staging environment in the application:

$ aws elasticbeanstalk create-environment --application-name sample-blue-green-php-app --environment-name staging --solution-stack-name "64bit Amazon Linux 2016.09 v2.2.0 running PHP 7.0"

Create Application Versions From The 3 Source Bundles:

$ for i in $(seq 0 2); do aws elasticbeanstalk create-application-version --application-name sample-blue-green-php-app --version-label blue-green-php-app-v1.$i --source-bundle S3Bucket=blue-green-php-app-source,S3Key=blue_green_php_app-v1.$i.zip ; done

I couldn’t find an obvious way to deploy an application version to an environment via the aws cli, so I did it via the console.

Before CNAME update:

$ aws elasticbeanstalk swap-environment-cnames --source-environment-name staging --destination-environment-name production

After CNAME update:

Note:

  • the CNAME TTL is 900 seconds.
  • calling environments production and staging is confusing, as the production CNAME would point to the staging environment sometimes. It’d be better to call each environment something unique and have the CNAMEs dictate their current function.

All-at-once, Rolling And Immutable Deployment

Test Setup:

  • AllAtOnce disables rolling deployments and always deploy to all instances simultaneously.
  • Rolling enables standard rolling deployments.
  • RollingWithAdditionalBatch launches an extra batch of instances prior to starting the deployment to maintain full capacity.
  • Immutable performs an immutable update for every deployment. This creates a new temporary auto scaling group behind the same load balancer and adds new instances to it until the number match the old group (it checks the health of the first instance then adds all the rest at-once). If the new group is all healthy, it transfers all instances to the old group, then deletes the new group and old instances.
    • Immutable is the default for managed updates
  • Blue / Green deployment does an immutable update, but also creates a new load balancer and switches to it with a DNS change.
    • i.e. in Elastic Beanstalk you clone an environment then switch the DNS

RDS Data Tier

AWS recommends you don’t create an RDS instance as part of the environment, as it’ll be tied to the environment’s lifecycle: it’s unable to be moved and is deleted when the environment is. However, when you do create it as part of the environment it sets these environment variables:

# env | grep RDS
  RDS_HOSTNAME=aa1nt70t6hihih2.c7y0ze4j3jfd.ap-southeast-2.rds.amazonaws.com
  RDS_DB_NAME=ebdb
  RDS_PASSWORD=<REDACTED>
  RDS_USERNAME=developmentuser
  RDS_PORT=3306

Instead, they say you should setup the RDS instance separately and pass in your own setting for the environment variables above.

Or, even better, save the connection information to a file on S3 and use an instance profile role that has permissions to read the file. That way, even if someone has permissions to read the configuration settings for the environment, they won’t see the database connection details.

configuration

.ebextensions

.ebextensions are YAML configuration files that are bundled with the application bundle. You can configure:

  • Deployment Options
  • Software dependencies (and bootstrapping in general)
  • CloudWatch metrics and alarms
  • Auto Scaling and other service configurations
  • Creating other resources for your application like:
    • An RDS database
    • An SQS queue
    • SNS notifications
  • etc…

Note: to apply changes, you need to deploy a new application version.

The sections include:

  • option_settings
    • key value settings within name spaces that can be referenced in other files, and used by Elastic Beanstalk for its own configuration
  • Resources
    • Just like CloudFormation.
    • Uses Fn::GetOptionSetting to reference option settings
  • Files
    • e.g. source (download) or content, mode, owner group
  • Users
  • Groups
  • Packages
  • Services
  • Sources
  • Commands
  • Container_commands
  • Outputs

You can also bundle shell scripts with your application bundle and run them from the ebextensions

Elastic Beanstalk uses CloudFormation to deploy environments and resources, so we have access to resource types and configurations supported by CloudFormation.

For example .ebextensions/rolling-updates.config

option_settings:
  aws:elasticbeanstalk:command:
    DeploymentPolicy: Rolling
    BatchSizeType: Percentage
    BatchSize: 25

options_settings is the section to configure Elastic Beanstalk options.

.ebextensions/options.config

option_settings:
  - namespace:  aws:elasticbeanstalk:container:tomcat:jvmoptions
    option_name:  Xmx
    value:  256m
  - option_name: MYPARAMETER
    value: parametervalue
option_settings:
  aws:autoscaling:launchconfiguration:
    InstanceType: m1.small
    SecurityGroups: my-securitygroup
    EC2KeyName: my-keypair
    MonitoringInterval: "1 minute"
    ImageId: "ami-cbab67a2"
    IamInstanceProfile: "ElasticBeanstalkProfile"
    BlockDeviceMappings: "/dev/sdj=:100,/dev/sdh=snap-51eef269,/dev/sdb=ephemeral0"
option_settings:
  aws:elb:listener:443:
    ListenerProtocol: HTTPS
    SSLCertificateId: arn:aws:iam::123456789012:server-certificate/elastic-beanstalk-x509
    InstancePort: 80
    InstanceProtocol: HTTP
  aws:elb:listener:80:
    ListenerEnabled: false
option_settings:
  aws:elasticbeanstalk:command:
    BatchSize: '30'
    BatchSizeType: Percentage
  aws:elasticbeanstalk:sns:topics:
    Notification Endpoint: me@example.com
  aws:elb:policies:
    ConnectionDrainingEnabled: true
    ConnectionDrainingTimeout: '20'
  aws:elb:loadbalancer:
    CrossZone: true
  aws:elasticbeanstalk:environment:
    ServiceRole: aws-elasticbeanstalk-service-role
  aws:elasticbeanstalk:application:
    Application Healthcheck URL: /
  aws:elasticbeanstalk:healthreporting:system:
    SystemType: enhanced
  aws:autoscaling:launchconfiguration:
    IamInstanceProfile: aws-elasticbeanstalk-ec2-role
    InstanceType: t2.micro
    EC2KeyName: workstation-uswest2
  aws:autoscaling:updatepolicy:rollingupdate:
    RollingUpdateType: Health
    RollingUpdateEnabled: true

The Resources section can be set for more control over resources via CloudFormation. You can set this in another .ebextensions file e.g. resources.config. The Fn::GetOptionSetting function retrieves values from the option_settings configuration

Resources:
  AWSEBAutoScalingGroup:
    Metadata:
      AWS::CloudFormation::Authentication:
        S3Auth:
          type: "s3"
          buckets: ["elasticbeanstalk-us-west-2-123456789012"]
          roleName:
            "Fn::GetOptionSetting":
              Namespace: "aws:autoscaling:launchconfiguration"
              OptionName: "IamInstanceProfile"
              DefaultValue: "aws-elasticbeanstalk-ec2-role"
VERSION=v2.8 && \
  cd sample_blue_green_php_app && \
  git checkout tags/$VERSION && \
  zip -r ../blue_green_php_app-$VERSION.zip . -x ./*.git/* && \
  git checkout master && \
  cd ..
$ aws s3 cp blue_green_php_app-v2.8.zip s3://blue-green-php-app-source/
$ aws elasticbeanstalk create-application-version --application-name sample-blue-green-php-app --version-label blue-green-php-app-v2.8 --source-bundle S3Bucket=blue-green-php-app-source,S3Key=blue_green_php_app-v2.8.zip
# env | grep ^MY_AWESOME_PARAMETER
  MY_AWESOME_PARAMETER2=milk
  MY_AWESOME_PARAMETER3=bread
  MY_AWESOME_PARAMETER1=eggs

The .ebextensions files section is run before the code is deployed i.e. if you try to write to a file within the deployment directory it’ll be overwritten. I manually got it working by SSHing into the instance and running:

cp /tmp/config.php /var/www/html/

leader_only

The leader_only option runs on the first instance only e.g. in a 3 instance ASG:

sample_blue_green_php_app/.ebextensions/blue_green_php.config

container_commands:
  ...
  leader_test:
    command: echo "Follow me!" > /tmp/i-am-leader.txt
    leader_only: true
$ ssh -i ~/Downloads/SHTestKey.pem ec2-user@13.55.9.254 cat /tmp/i-am-leader.txt
cat: /tmp/i-am-leader.txt: No such file or directory

$ ssh -i ~/Downloads/SHTestKey.pem ec2-user@13.54.173.205 cat /tmp/i-am-leader.txt
Follow me!

$ ssh -i ~/Downloads/SHTestKey.pem ec2-user@13.55.13.216 cat /tmp/i-am-leader.txt
cat: /tmp/i-am-leader.txt: No such file or directory

Saved Configurations

  • Configurations can be saved and stored as objects in Amazon S3
  • Saved configurations belong to an application but they can be applied to new or existing environments for that application
  • They’re YAML formatted and define:
    • The environment’s platform configuration e.g. single or multi-container docker deployment
    • The tier - web or worker
    • Configuration settings
    • Resource tags

They exclude configuration that is applied via .ebextensions

Elastic Beanstalk uses precedence to determine which configuration is applied, from highest to lowest:

  • Settings applied directly to the environment
  • Saved Configurations
  • Configuration files

Testing Other Environment Configurations

Changing To An Auto Scaling, Load Balanced Environment

  • Changing from a single instance to an auto scaling, load balanced environment will replace the current instance.
  • You can then set
    • min, max, AZs, cooldown
    • Scaling Trigger e.g. CPUUtilization Average Percentage, which sets CloudWatch alarms for the auto scaling group:

CloudWatch Alarms:

Note: the alarms are not configured correctly. The 6,000,000 figure is a left-over from the inital network out alarm suggested in the form. This should be a percentage e.g. 70.

Load on an instance …

for i in $(seq 1 5); do dd if=/dev/urandom | bzip2 -9 >> /dev/null & done

… causes a scale event …

… and when stopped, a scale down:

Worker Tier

Worker tiers run a daemon that reads from a SQS queue and then POSTs that information to a URL that you specify:

sqsd      2096  0.2 17.0 961324 103440 ?       Sl   22:43   0:08 /opt/elasticbeanstalk/lib/ruby/bin/ruby /opt/elasticbeanstalk/lib/ruby/bin/aws-sqsd start

Killing httpd and running a netcat (nc) instead, and then adding a test message to the queue I get this:

# nc -l 80
POST / HTTP/1.1
Content-Type: application/json
User-Agent: aws-sqsd/2.3
X-Aws-Sqsd-Msgid: ad0b7bf4-6e5b-4cd7-93d9-8481b2124786
X-Aws-Sqsd-Receive-Count: 1
X-Aws-Sqsd-First-Received-At: 2016-12-11T23:37:51Z
X-Aws-Sqsd-Sent-At: 2016-12-11T23:37:51Z
X-Aws-Sqsd-Queue: awseb-e-gqkxngqhpj-stack-AWSEBWorkerQueue-1R2B3UT4CTB2J
X-Aws-Sqsd-Path:
X-Aws-Sqsd-Sender-Id: AIDAIAI2G6LSJPAVCN4HM
Host: localhost
Content-Length: 4

test

Docker

You can use preconfigured Docker environment:

  • Glassfish
  • Go
  • Python

or a custom one:

  • Docker
    • Can have a Dockerfile and/or a Dockerrun.aws.json (version 1) file
  • Multi-container Docker
    • Can only have a Dockerrun.aws.json (version 2)
      • You specify the “Image” in this file rather than a Dockerfile
        • i.e. You need to have custom Docker images in a private repo if you need to use them
      • Other options include AWSDockerRunVersion, Ports, Volumes, Logging, Authentication, containerDefinitions
        • The Logging option allows Elastic Beanstalk to automatically retrieve logs from the Docker instances

Dockerfile and Dockerrun.aws.json are bundled with the application code in the WAR/ZIP archive or Git repository.

In addition to the Dockerrun.aws.json file, we can also have configuration files in the .ebextensions directory.

option_settings:
  aws:elb:listener:8080
    ListenerProtocol: HTTP
    InstanceProtocol: HTTP
    InstancePort: 8080

Then map that port to our container from the host instance in the Dockerrun.aws.json file

{
    "name": "nginx-proxy",
    "image": "nginx",
    "essential": true,
    "memory": 128,
    "portMappings": [
      {
        "hostPort": 80,
        "containerPort": 80
      }
    ],
    ...
}

i.e ELB:8080 => Instance(host):8080 => Container:8080

.ebextensions can also be used to create:

  • DynamoDB tables
  • SQS queues
  • etc…

FYI: Environment variables are available within each container:

# docker exec -it 1ae18d2aefc4 env | grep ^TEST

TEST_VAR1=horse
TEST_VAR2=tree
TEST_VAR3=ball

Preconfigured Docker

Nginx is used to proxy requests to the docker instance:

# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
f4632be99728        b4982e9d9939        "/uwsgi-start.sh"   54 minutes ago      Up 54 minutes       8080/tcp            jolly_tesla
# cat /etc/nginx/conf.d/elasticbeanstalk-nginx-docker-upstream.conf
upstream docker {
	server 172.17.0.2:8080;
	keepalive 256;
}
# cat /etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy.conf
  ...
    server {
        listen 80;

        ...

        location / {
            proxy_pass            http://docker;
            proxy_http_version    1.1;

            proxy_set_header    Connection            $connection_upgrade;
            proxy_set_header    Upgrade                $http_upgrade;
            proxy_set_header    Host                $host;
            proxy_set_header    X-Real-IP            $remote_addr;
            proxy_set_header    X-Forwarded-For        $proxy_add_x_forwarded_for;
        }
    }

Generic Multi-Container Docker

This option starts an instance with docker running on it, and the Amazon ECS Agent:

# docker ps
CONTAINER ID        IMAGE                            COMMAND             CREATED             STATUS              PORTS               NAMES
d2ba0929b31e        amazon/amazon-ecs-agent:latest   "/agent"            2 minutes ago       Up 2 minutes                            ecs-agent

This allows container instances to connect to a cluster. Like all environments, you have an auto scaling, load balanced option.

Single Container Docker - Dockerfile

Dockerfile is only supported for Single Container Docker; Multi Container Docker requires images to be deployed to an online repository before creating the Elastic Beanstalk environment.

Using a simple PHP Dockerfile as an example:

FROM php:7.1-apache
COPY src/ /var/www/html/
EXPOSE 80

Build the image:

$ docker build -t blue-green-php-app .

Test it works locally:

$ docker run -d -p 80:80 blue-green-php-app

Create an application version to upload:

VERSION=v1.0 && \
  cd elastic_beanstalk_simple_docker && \
  git checkout tags/$VERSION && \
  zip -r ../elastic_beanstalk_simple_docker-$VERSION.zip . -x ./*.git/* && \
  git checkout master && \
  cd ..

Upload the application and deploy it, and it works!

Update v1.1 that has a Dockerrun.aws.json file.

{
  "AWSEBDockerrunVersion": "1",
  "Volumes": [
    {
      "HostDirectory": "/var/app/shared",
      "ContainerDirectory": "/mnt/shared"
    }
  ],
  "Logging": "/var/log/nginx"
}

Note: “You can provide Elastic Beanstalk with only the Dockerrun.aws.json file or in addition to the Dockerfile in a .zip file. When you provide both files, the Dockerfile builds the Docker image and the Dockerrun.aws.json file provides additional information for deployment as described later in this section. When you provide both the Dockerfile and the Dockerrun.aws.json file, do not specify an image in the Dockerrun.aws.json file. Elastic Beanstalk uses the image specified in the Dockerfile and ignores it in the Dockerrun.aws.json file.” - http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker_image.html

Confirming the container has the volume mounted …

# docker exec -i -t 4277206e9833 touch /mnt/shared/hello

… on the host:

# ls -al /var/app/shared/
total 8
drwxr-xr-x 2 root root 4096 Dec 12 01:55 .
drwxr-xr-x 4 root root 4096 Dec 12 01:40 ..
-rw-r--r-- 1 root root    0 Dec 12 01:55 hello

Confirming the Dockerfile and Dockerrun.aws.json are not deployed in the container:

$ docker exec -i -t 4277206e9833 ls -al /var/www/html
total 20
drwxr-xr-x 2 www-data www-data 4096 Dec 12 01:28 .
drwxr-xr-x 3 root     root     4096 Nov  8 23:22 ..
-rw-r--r-- 1 root     root       92 Dec  9 01:15 config.php
-rw-r--r-- 1 root     root      366 Dec 12 01:23 index.php
-rw-r--r-- 1 root     root       20 Dec  9 01:21 phpinfo.php

Multi Container Docker - Image From A Private Repository

I extended the Multicontainer Docker Tutorial by having 3 php containers load-balanced by Nginx on each instance, and 2 load-balanced instances for a total of 6 containers: https://github.com/SteveHoggNZ/elastic_beanstalk_docker/tree/v1.1

No Dockerfile is used. The images are referenced in Dockerrun.aws.json, which creates the 3 php-app containers and links them to the nginx-proxy.

Note: public repositories are used. If private ones were used then an authentication section in Dockerrun.aws.json could be set to refer to an S3 bucket an key that contains the authentication data.

# docker ps --format '{{.ID}} {{.Image}}\t\t\t{{.Command}}'

e83a2b5259f7 nginx			"nginx -g 'daemon off"
3c5d8b8d52a6 php:fpm			"php-fpm"
3137a3d4a777 php:fpm			"php-fpm"
6687a8a69d5b php:fpm			"php-fpm"
cf5288784822 amazon/amazon-ecs-agent:latest			"/agent"

Confirm nginx can talk to php:fpm

# docker exec -i -t e83a2b5259f7 grep php-app /etc/hosts

172.17.0.2	php-app1 6687a8a69d5b ecs-awseb-Custom-env-1-9yvkg5rq7n-1-php-app1-dec0acd3a293c2bc3b00
172.17.0.3	php-app2 3137a3d4a777 ecs-awseb-Custom-env-1-9yvkg5rq7n-1-php-app2-e2d3a5f8a9c7a6a3aa01
172.17.0.4	php-app3 3c5d8b8d52a6 ecs-awseb-Custom-env-1-9yvkg5rq7n-1-php-app3-b0ef9db6d1d6a5eaee01

Result:

Requests spread evenly across 2 instance running 3 containers each:

$ for i in $(seq 1 100); do curl http://multi-docker-php.ap-southeast-2.elasticbeanstalk.com/ -s | grep \<h3\> | cut -d \> -f 2; done | sort | uniq -c | sort -n
  16 3137a3d4a777
  16 cc763eb0fad5
  17 3c5d8b8d52a6
  17 44e2f44d2f3b
  17 6687a8a69d5b
  17 6f0e3bf5aced

Note: Elastic Beanstalk doesn’t seem to enable cross-zone load balancing by default.

Notes

There is a eb CLI that provides simpler commands than aws elasticloadbalancer

Activity Log

/var/log/eb-activity.log

Environment Variables

They’re set in /etc/profile.d/eb_envvars.sh

YAML Linting

http://www.yamllint.com/

References