DevOps Next Steps: Infrastructure Automation
You helped to spread the word about DevOps throughout the organization. Now, Operations is in open communication with Development and is involved in the decision-making process. Where do we go from here?
There's still a ton of work to be done whether you're still in an on-prem data center or the cloud. Being on one of the major cloud providers obviously offers some advantages here but, let's start with some basic requirements that apply to both scenarios.
Our infrastructure should be:
Scalable. Accommodate the increase or reduction of instances or stacks.
Repeatable. Be able to reproduce an environment or instance to a point in time.
Testable. Provide a means for various forms of testing unit, integration, system, etc.
Maintainable. Easily understand and reproduce existing infrastructure.
Thankfully, various forms of tools exist today to make this possible Terraform, Configuration Management (CM) (Ansible, Chef, and others), and CloudFormation just to name a few. Choosing a tool or set of tools for automation is an important decision that is made as an organization. I encourage having discussions amongst the team and POC'ing multiple tools to find the best tool for your organization. It may be important to emphasize the distinction between automating infrastructure and provisioning infrastructure. Infrastructure automation refers to launching virtualized infrastructure while the latter, provisioning, refers to configuring software on infrastructure. I've seen the term provisioning used interchangeably but, I prefer to have a clear definition. For now, we'll want to focus on what I'll call infrastructure agnostic tools. It means the tools are not designed to support only VMware or one cloud provider. Instead, they provide a means to support multiple providers.
When it comes to pure infrastructure automation, I believe Terraform, an open source tool developed by Hashicorp (makers of Vagrant and Packer) has a leg up on the competition. Terraform maintains a global state of your infrastructure and you can either leverage their DSL or JSON to describe your infrastructure. The DSL is very readable, easy to learn, and the community does a great job of keeping the codebase up-to-date with the latest provisioner changes. The major advantages of Terraform are its wide variety of support across provisioners like AWS, Google, CloudFlare, VMware, many others, and it's strong preview mode. The multiple provisioner support eases the pain of switching to a different cloud provider or maintaining infrastructure in hybrid environments.
Going the CM route for infrastructure automation, however, has its advantages. The most important one being you can use the same tool for both launching and provisioning infrastructure. It means less education is required to support your infrastructure and the ability to pass dynamic values like IPaddresses, hostnames, load balancers, etc. without hard coding your configuration code. Some CM languages such as Chef will also allow you to write unit tests for your infrastructure code. All major players in CM support infrastructure automation some are less fleshed out than others but, comparing CM tools is an entirely different conversation to be had.
Regardless of what toolset you choose, testing is a must have. My current favorite for infrastructure testing is Test Kitchen. It's very versatile and supports multiple provisioners. I advise not getting too granular when testing infrastructure. Instead, focus on testing an instance/node and the stack as a whole, checking for criteria like open ports, endpoints, running services, network connectivity, etc.
Provisioning your infrastructure is the next chapter.