AWS and other interesting stuff

Continuous Integration And Continuous Delivery - Examples

OpsWorks

The examples here are based on OpsWorks Best Practices

For descriptions of the deployment methods see part 1 - Deployment Methods

Maintaining Consistency

There are 2 deployment mechanisms with OpsWorks:

  1. New instances get the current app and cookbook code
  2. Existing instances must have the app and cookbook code manually deployed. Deploy command for apps, and Update Custom Cookbooks command for cookbooks.

That means you have to manage your source code carefully to avoid unintentionally running different code on different instances. e.g. running code from the master branch of a Git repository is not a good idea as it is a moving target. AWS recommends using S3 Archives as this guarantees your code is a static artifact e.g. get Jenkins to test and build these artifacts. If you use Git, you should use tags as a static reference.

Deployment

OpsWorks does not automatically deploy updated code on online instances, you must do that manually. That means you need to consider:

  • Deploying without compromising the site’s ability to handle traffic
  • Handling an unsuccessful deployment

You can deploy all-at-once, to all instances in a layer, however, the recommended way is to use a more robust strategy like Rolling Deployment or Blue Green Deployment (see below):

Testing

Test App Setup

To test various deployment methods I created a simple PHP app that connects to an RDS instance

I created 10 tags to experiment with:

for i in $(seq 0 9); do VERSION="v1.$i" ; sed -i "" 's/\$version = ".*";/$version = "'$VERSION'";/' index.php ; git add index.php; git commit -m "increment to version $VERSION"; git tag $VERSION ; git push origin $VERSION; done

The version number is shown on in the PHP app so I can easily see which version is deployed.

Setup Steps:

  • Create a stack for the application
  • Create a PHP Layer for the app with an ELB
  • Create an RDS Layer for the database
  • Setup an app
    • Use RDS as the data source
    • Use simple PHP app with tag v1.0 as the source
  • Create 3 instances on the PHP Layer - one in each AZ - and deploy the app to all 3 instances
  • Add custom chef recipes to configure the database connection for the app

After a deploy, the app shows this:

Note: I’m going to ignore the ‘\n’ after the messages.

Rolling Deployment

AWS recommend that you use ELB health checks to verify the deployment was successful, but you should point it to a page that checks dependencies and verifies everything is working correctly.

Remove The Instance From The Load Balancer

$ aws elb deregister-instances-from-load-balancer --load-balancer-name PHPBlueGreenDeploymentAppLB --instances i-0138af5cad4a30431

Update The App Revision Number

$ aws opsworks update-app --app-id f2966874-c39e-4caf-b0f2-4610478daead --app-source Revision=v1.1

Deploy To The Instance That Is Out Of The Load Balancer

Lookup the OpsWorks Instance ID using the EC2 Instance ID …

$ aws opsworks describe-instances --layer-id 618dab35-4723-4dd4-8af9-d92b56f9f9f8 | jq '.Instances[] | select(.Ec2InstanceId == "i-0138af5cad4a30431") | {InstanceId}'
{
  "InstanceId": "87ae918a-73e8-4655-ad30-c0601196c79c"
}

… then use that ID to do a deployment to the instance that out of the load balancer.

$ aws opsworks create-deployment --stack-id 4d7c757d-7170-4085-91fd-cbf98eae6f3b --app-id f2966874-c39e-4caf-b0f2-4610478daead --command Name=deploy --instance-ids 87ae918a-73e8-4655-ad30-c0601196c79c
{
    "DeploymentId": "fb4be884-4dfb-47a8-9797-074e2fa36b07"
}

As expected, the instance that is out of the load-balancer has its app updated and the other two don’t i.e. the load-balanced app is still the old version.

Register The Instance Back With The Load Balancer

I can then register the instance back with the load-balancer:

$ aws elb register-instances-with-load-balancer --load-balancer-name PHPBlueGreenDeploymentAppLB --instances i-0138af5cad4a30431

Use The Load Balancer’s Health Check To See How To Proceed

If It Comes Back As InService, Move On To The Next Instance
$ aws elb describe-instance-health --load-balancer-name PHPBlueGreenDeploymentAppLB --instances i-0138af5cad4a30431
{
    "InstanceStates": [
        {
            "InstanceId": "i-0138af5cad4a30431",
            "ReasonCode": "N/A",
            "State": "InService",
            "Description": "N/A"
        }
    ]
}
If It Comes Back As OutOfService, Pause And Alert Someone

The deployment will need manual intervention if this happens.

Deploy your code to multiple stacks and

Blue Green Deployment

The best practice deployment guide refers to a using multiple stacks for the different stages of deployment: Development, Staging and Production.

  • The development stack permissions set so that developers can access the servers
    • e.g. an SSH key for each developers
  • When you’re ready, you clone the Development Stack to make a Staging Stack
    • Important: don’t clone the permissions
  • When the staging stack is ready, you do a Blue Green deployment by promoting the Staging Stack to production and retiring the old Production Stack.

The documentation talks about:

  • Having a pool of ELB that you can attach to OpsWorks Stacks as-needs-be. You could also pre-warm them if traffic warrants it.
  • Gradually transferring traffic to the green stack using weighted DNS e.g. 5%, 10% gradually up to 50% then a full cutover. (This sounds like AB testing to me)
  • Monitoring for problems with the new stack and rolling back if needs-be.
  • Keeping the old blue stack around in-case you need to rollback completely.
  • Data Source
    • You can only register an RDS DB instance with one stack at a time, but you can switch an RDS DB instance from one stack to another.
    • Approaches:
    • Use the same database for both applications (Blue and Green)
      • Advantages
        • No downtime, no synchronisation
      • Disadvantages
        • Migrating to a new schema is difficult
        • You need to manage the security in some other way e.g. environment vars, or custom JSON
    • One database for both applications (Blue and Green)
      • Advantages
        • Each version has its own database, so schemas don’t need to be compatible
      • Disadvantages
        • Synchronisation of databases during the transition is complicated
        • You need to ensure synchronisation doesn’t cause significant downtime and performance issues.

In general, RDS is recommended due to its flexibility in any transition scenario i.e. it can be disassociated and associated.

Cloning An OpsWorks Stack

Cloning a stack won’t make a copy of the RDS instance or the ELB. For that reason, using CloudFormation to manage multiple sets of associated resources would be a better choice for production. I’m using the CLI as I’d like to get familiar with it.

I’m cloning a Development stack here, but the same process could be used to clone from Development to Staging as suggested in the steps above.

$ aws opsworks clone-stack --source-stack-id 4d7c757d-7170-4085-91fd-cbf98eae6f3b --service-role-arn "arn:aws:iam::<REDACTED>:role/aws-opsworks-service-role" --name "PHPBlueGreenApp-Dev"

You’re not able to clone apps that have RdsDbInstance or OpsworksMysqlInstance data source, so that needs to be manually setup on the new stack:

First, I need to create a new RDS instance though:

$ aws rds create-db-instance --db-name mydbdev --db-instance-identifier mydbdev --db-instance-class db.t2.micro --engine mysql --master-username mydbdev --master-user-password '<REDACTED>' --backup-retention-period 0 --allocated-storage 5

I forgot to set the correct security group, so I’ll do that now:

$ aws rds modify-db-instance --db-instance-identifier mydbdev --vpc-security-group-ids sg-4eb45729

Note: describe-db-instances didn’t return a DBInstanceArn attribute as promised, so I had to look that up in the console.

Then I need register the RDS instance (i.e. create an RDS Layer) for the new dev stack:

$ aws opsworks register-rds-db-instance --stack-id 22ce84ee-087d-4d29-b9dd-b4950d1bfcb3 --rds-db-instance-arn "arn:aws:rds:ap-southeast-2:<REDACTED>:db:mydbdev" --db-user mydbdev --db-password "<REDACTED>"

Now I need to create an ELB …

$ aws elb create-load-balancer --load-balancer-name PHPBlueGreenDeploymentAppLBDev --listeners Protocol=HTTP,LoadBalancerPort=80,InstancePort=80,InstanceProtocol=HTTP --availability-zones ap-southeast-2a ap-southeast-2b ap-southeast-2c

$ aws elb modify-load-balancer-attributes --load-balancer-name PHPBlueGreenDeploymentAppLBDev --load-balancer-attributes "CrossZoneLoadBalancing={Enabled=true},ConnectionDraining={Enabled=true,Timeout=300}"

$ aws elb configure-health-check --load-balancer-name PHPBlueGreenDeploymentAppLBDev --health-check Target=HTTP:80/index.php,Interval=30,UnhealthyThreshold=2,HealthyThreshold=2,Timeout=5

… and associate it with the PHP Layer:

$ aws opsworks attach-elastic-load-balancer --elastic-load-balancer-name PHPBlueGreenDeploymentAppLBDev --layer-id 06bca6a8-6895-4a61-999f-c6387f3ff413

Then I need create the App for the dev stack:

$ aws opsworks create-app --stack-id 22ce84ee-087d-4d29-b9dd-b4950d1bfcb3 --name PHPBlueGreenApp --type php --app-source Type=git,Url=https://github.com/SteveHoggNZ/sample_blue_green_php_app.git,Revision=v1.2 --data-source Type=RdsDbInstance,DatabaseName=mydbdev,Arn=arn:aws:rds:ap-southeast-2:<REDACTED>:db:mydbdev

Note: I’m using a new revision, v1.2

Then I add instances to the layer for each AZ:

$ aws opsworks create-instance --stack-id 22ce84ee-087d-4d29-b9dd-b4950d1bfcb3 --layer-ids 06bca6a8-6895-4a61-999f-c6387f3ff413 --instance-type t2.micro --availability-zone ap-southeast-2a
$ aws opsworks create-instance --stack-id 22ce84ee-087d-4d29-b9dd-b4950d1bfcb3 --layer-ids 06bca6a8-6895-4a61-999f-c6387f3ff413 --instance-type t2.micro --availability-zone ap-southeast-2b
$ aws opsworks create-instance --stack-id 22ce84ee-087d-4d29-b9dd-b4950d1bfcb3 --layer-ids 06bca6a8-6895-4a61-999f-c6387f3ff413 --instance-type t2.micro --availability-zone ap-southeast-2c

Then start each of the instances:

$ aws opsworks start-instance --instance-id 92687748-1abb-4b59-9ab8-998c61c5a9af
$ aws opsworks start-instance --instance-id b2fd9a76-0897-46f7-a4b1-30e80580400d
$ aws opsworks start-instance --instance-id 63193fca-46f7-4144-956e-46915e9261e6