AWS and other interesting stuff

OpsWorks

OpsWorks For DevOps Certification - Part 1 - Introduction

I haven’t used OpsWorks before, so I only have a general understanding of what it does.

I intend to learn in this way:

  1. Do some reading to get a general idea of what the tool is for and what you can do with it
  2. Solve a specific problem with it - get hands on by setting myself some challenges to accomplish
    • This tests my knowledge and assumptions about the tool; validating assumptions is great, but invalidating them is often where the best learning is.
  3. Dive-deep on the features to see what else is possible
    • Branching out from the specific example, try out other features and try to understand their purpose
    • Keep going until I am happy with my coverage of the features

In this first post of a two-part blog I’m working at stage 2.

In part 2 I will move on to stage 3 - a thorough investigation into the OpsWorks interface and concepts.

OpsWorks

OpsWorks is Chef + AWS’ bespoke orchestration

OpsWorks is AWS’ implementation of the CHEF configuration management and automation system with the additional power to create, update and delete AWS infrastructure components.

OpsWorks For Chef Automate vs AWS OpsWorks Stacks

Chef Automate is chef.io’s commercial offering that provides: workflow automation for continuous deployment, automated testing for compliance and security, and a user interface that gives you visibility into your nodes and their status.

AWS OpsWorks for Chef Automate is AWS’s managed service offering of Chef Automate.

It was announced at AWS re:Invent in November 2016, so it won’t be in the exams at the time this blog is written.

https://blog.chef.io/2016/12/01/inside-the-new-aws-opsworks-for-chef-automate-service/

The existing OpsWorks service is now called AWS OpsWorks Stacks.

Pick A Task

I decided to install MediaWiki. I chose MediaWiki as I’m familiar PHP, it has multiple versions available and it requires a database so I’ll need to setup an RDS layer.

Fundamentals 1

Challenge #1 - Setup MediaWiki as an Application and deploy it

  1. Create an App with https://releases.wikimedia.org/mediawiki/1.20/mediawiki-core-1.20.8.tar.gz as an HTTP Archive Application Source
  2. Deploy it and see what happens

    • /etc/httpd/sites-enabled/mediawiki.conf - DocumentRoot is /srv/www/mediawiki

      # ls -l /srv/www/mediawiki
      total 8
      lrwxrwxrwx 1 deploy apache current -> /srv/www/mediawiki/releases/20161205000142
      drwxr-xr-x 4 deploy apache releases
      drwxrwx--- 9 deploy apache shared
      
Visiting the instance IP in the browser shows MediaWiki. The application obviously isn’t installed properly - it’ll need a database - but its a start :-)

Challenge #2 - Deploy a more recent version

  1. Set Application Source to https://releases.wikimedia.org/mediawiki/1.21/mediawiki-core-1.21.11.tar.gz for the app
  2. Run deploy
That was easy
# ls -l /srv/www/mediawiki/current
lrwxrwxrwx 1 deploy apache 42 Dec  5 00:37 /srv/www/mediawiki/current -> /srv/www/mediawiki/releases/20161205003742

Challenge #3 - Rollback to the version from #1

The Deploy link for the App also has a Rollback option

That didn’t work:

Chef::Exceptions::RecipeNotFound: could not find recipe php-rollback for cookbook deploy

It looks like it’d be as simple as running this on each instance (assuming the instance has that release):

rm -f /srv/www/mediawiki/current && ln -s /srv/www/mediawiki/releases/20161205000142 /srv/www/mediawiki/current

I’ll look at this again later if needs be.

Challenge #4 - Add a new instance

That was easy enough

Observations:

  • The new instances have the application deployed, but there was no app option when creating a new instance so it must be an app or layer setting.
  • Apps > Deploy and Deployments > Deploy App take you to the same screen, except the App is selectable in the later.

Perhaps when you deploy to only one instance, then add a new instance to the layer it won’t have the application deployed? No, the app is deployed to new instances in the layer. When an app is deployed on an instance, it must get associated with the instance’s layer. That makes sense as you want all instances in a layer to be the same (that’s the whole point…)

Challenge #5 - Upgrade OS

I created instances with an older OS:

$ head -n 1 /etc/issue
Amazon Linux AMI release 2016.03

Updating the Stack’s Default OS won’t upgrade the instances.

Deployments > Run Command has an Upgrade Operating System option:

Observations:

  • After upgrading the OS, AWS recommend you run Setup to make sure the instance is setup correctly. i.e. run the recipes associated with the Setup Lifecycle hook.

That works as expected.

$ head -n 1 /etc/issue
Amazon Linux AMI release 2016.09

Fundamentals 2

Reference: http://docs.aws.amazon.com/opsworks/latest/userguide/gettingstarted.html

MediaWiki uses MySQL so that needs to be configured.

Challenge #6 - Setup an RDS Layer

You setup the RDS instance first then record the connection details in the layer settings.

That was easy enough. On deploy, the JSON configuration is recorded in /var/lib/aws/opsworks/chef

cat /var/lib/aws/opsworks/chef/2016-12-05-02-25-03-01.json | jq '.opsworks.stack.rds_instances[0].db_name'
"mediawiki"

Now how do I get the connection details configured in MediaWiki?

Challenge #7 - Configure MediaWiki with the RDS connection settings

RDS needs to be added as a data source for the application:

The application then needs to be deployed on existing instances.

The deployment JSON contains the database connection details now:

cat /var/lib/aws/opsworks/chef/2016-12-05-02-44-48-01.json | jq '.deploy.mediawiki | { application, deploy_to, group, database }'
{
  "application": "mediawiki",
  "deploy_to": "/srv/www/mediawiki",
  "group": null,
  "database": {
    "host": "opsworksmediawiki.<REDACTED>.ap-southeast-2.rds.amazonaws.com",
    "database": "mediawiki",
    "port": 3306,
    "username": "mediawiki",
    "password": "<REDACTED>",
    "reconnect": true,
    "data_source_provider": "rds",
    "type": "mysql"
  }
}

Chef’s node syntax is as follows:

[:deploy][:mediawiki][:database][:username]

Chef Recipe

A Chef recipe is basically a specialized Ruby application that performs tasks on an instance such as installing packages, creating configuration files, executing shell commands, and so on. Groups of related recipes are organized into cookbooks, which also contain supporting files such as templates for creating configuration files.

Reference: http://docs.aws.amazon.com/opsworks/latest/userguide/gettingstarted-db-recipes.html

The dbsetup.rb refers to default[:phpapp][:dbtable] but I don’t see that in the /var/lib/aws/opsworks/chef/ JSON, so where does it come from? It is a defined attribute in attributes/default.rb

The deploy variable refers to the current app node, [:deploy][:mediawiki] in my example.

I setup https://github.com/SteveHoggNZ/H4Cookbooks.git as a Custom Chef Recipe location for the Stack, and set the recipes for the deploy hook on the PHP Layer:

On first try, I got a error:

STDERR: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock'

I updated the dbsetup.rb file to include the -h mysql option, and while I was at it I made sure the RDS security group accepted MySQL traffic from 172.31.0.0/16

This still didn’t update the Chef recipes due to local caching, so I ran the update_custom_cookbooks command, deployed again and it worked.

Finally, I had to put the mediawiki.sql.gz file in files/default, and then copy that to the file system

cookbook_file '/tmp/mediawiki.sql.gz' do
  source 'mediawiki.sql.gz'
  mode '0400'
  action :create
end

… so that it could then be restored:

execute "mysql-setup" do
  command "cat /tmp/mediawiki.sql.gz | gzip -d | /usr/bin/mysql -h#{deploy[:database][:host]} -u#{deploy[:database][:username]} -p#{deploy[:database][:password]} #{deploy[:database][:database]}"
  not_if "/usr/bin/mysql -h#{deploy[:database][:host]} -u#{deploy[:database][:username]} -p#{deploy[:database][:password]} #{deploy[:database][:database]} -e'SHOW TABLES' | grep #{node[:mediawiki][:checktable]}"
  action :run
end

Reference: http://www.thegeekstuff.com/2016/06/chef-cookbook-directory-structure/

Note: As this wiki install isn’t intended for production, there are a few things missing from the setup. For example:

  • $wgSecretKey, and wgUpgradeKey in LocalSettings.php are hard coded. They would need to be generated on a per install basis.
  • Idempotence is handled in a very basic way in dbsetup.rb. A better way with timestamps would suit a production setup

Next … On To Part 2


I’ve learn’t a lot. Now on to part 2



Miscellaneous

Useful Getting Started guide

http://robinosborne.co.uk/2013/07/02/chef-for-developers/

Chef Testing

Test Kitchen

Chef Alternatives