I was considering how (having not worked in an environment where all this stuff has been really created from scratch) how I would go about creating an infrastructure from code, from scratch and make it reproducible.
If you read the press, this is what everyone is supposedly doing, but I haven’t worked anywhere where all of it sits together as it should – usually because of legacy infrastructure, multiple tools etc.
So, given that:-
1. There are seemingly hundreds of tools to choose from;
2. I know puppet already;
3. I have just complete my AWS developer associate certification, so I know a bit about that;
4. I have been trying out Terraform, and have been quite impressed;
I have come up with the following workflow using these tools, but the concepts should work with other varieties (NB: purely for EC2 instances at the moment):-
Terraform deploys base instance to EC2 puppet server
IAM role added to allow aws CLI to function (not needed for this, but for other admin)
Terraform puppet server bootstrap script:-
1. Installs aws tools
2. Copies s3 ux1_backup/puppet to the right places (i.e. recovers the puppet server config/data)
3. Starts puppet server
4. Starts puppet agent (which will install all other software and configure)
Terraform deploys base instances to other EC2 servers
1. On the instance: Installs puppet agent
2. On the instance: connects
3. On the puppet server: Check for a cert-signing request from the instance, sign if it looks correct
4. On the instance: ensure puppet agent is running and tests ok
5. Create a semaphore file to indicate recovery is needed (e.g. /recoverme)
All EC2 servers
1. Puppet deploys a backup/recovery script and schedules
2. The script checks for the recovery semaphore file on the instance – if there, recovers configuration and data from the most recent backup (NB: The S3 bucket should be configured with versioning on, copies to glacier ASAP), and clears the semaphore. If not there, runs a backup to the S3 bucket.
Seems like it should work, and once puppet is configured, it should be possible to destroy and recreate everything through terraform.
NB: Terraform config probably needs to be kept on github or similar – with instructions how to obtain the AWS credentials.
Now to try it out…