Terraform Project Structure

Getting started with Terraform is not that complex, however the lack of a good project structure becomes a problem quicker than you may think.

This post assumes you have basic knowledge about Terraform. Check out our previous post An introduction to Terraform if you would like a refresh.

Project structure

In the following project structure we split per environment. We leverage the use of modules to make sure we do not duplicate more than needed. This approach allows the environments to be different in size but not structure.

├── environments
│   ├── integration
│   │   └── main.tf
│   │   └── variables.tf
│   ├── production
│   │   └── main.tf
│   │   └── variables.tf
│   └── staging
│       └── main.tf
│       └── variables.tf
└── modules
    ├── app-asg
    │   └── main.tf
    └── network-common
        └── main.tf

Depending on the need and the complexity of your infrastructure, you may want to start by splitting your configuration into multiple components. By splitting these, you will both increase the speed Terraform can make changes and lower the risk of making accidental changes. This change, however, will require you to use remote state to reference resources from different environments.

├── environments
│   ├── production
│   │   ├── backend
│   │   │   └── main.tf
│   │   │   └── variables.tf
│   │   ├── frontend
│   │   │   └── main.tf
│   │   │   └── variables.tf
│   │   ├── network
│   │   │   └── main.tf
│   │   │   └── variables.tf
...

Modules

Modules allow you to abstract some implementation details from the user. In our case we can use modules to define for example an asg-app setup. This module will setup an Elastic LoadBalancer (ELB) and Auto Scaling Group (ASG). It will require an AMI and an instance count as arguments.

A module by itself is just a configuration that takes some variables that will be converted into arguments when used as a module.

variable "ami" {
  type = string
}

variable "instance_count" {
  type = number
}

resource "aws_instance" "example" {
  ami           = var.ami
  instance_type = "t2.micro"
}
...

To use this module, you reference the location where the module is located. You have multiple options to define the module source.

module "app" {
  source         = "../../modules/asg-app"
  ami            = "ami-123123123"
  instance_count = 12
}

Backends and Locks

When using AWS, you may want to choose S3 as a backend for your state storage. By using a backend, the state will be stored in a central location and will allow anyone in the team to work with this central state given the correct permissions.

terraform {
  backend "s3" {
    region         = "eu-west-1"
    bucket         = "company-terraform-state"
    dynamodb_table = "terraform_locks"
  }
}

Note the dynamodb_table setting. This setting makes sure a lock is created whenever Terraform is running. This will prevent others from altering the same resource at the same time.

$ terraform init ${ENVIRONMENT}/${COMPONENT} \
		-backend=true \
		-backend-config="key=${ENVIRONMENT}/${COMPONENT}/terraform.tfstate"
$ terraform plan ${ENVIRONMENT}/${COMPONENT}
$ terraform apply ${ENVIRONMENT}/${COMPONENT}

Makefile

This example shows the init command for a complex project structure. It sets the backend key to be unique for every component.

This may be a good time to create a helper script or Makefile to make sure the interaction with Terraform and its configuration gets abstracted away from the user.

An approach I like is to create a Makefile that can be called as follows:

$ make plan env=prod component=network

An example Makefile would look something like this:

plan: _init
	terraform plan environments/$(env)/$(component)
	
apply: _init
	terraform plan environments/$(env)/$(component)

_init:
	rm -rf .terraform
	terraform init \
		-backend=true \
		-backend-config="key=$(env)/$(component)/terraform.tfstate" \
		environments/$(env)/$(component)

Reference Remote State

If you need to access data from another component, you can use the remote state.

In this example we will have an app component that needs the vpc_id that is defined in a different component.

In the network component we define an output to specify we want to expose the value to other components.

output "vpc_id" {
  value = "${aws_vpc.main.id}"
}

Now we can setup a reference to the remote state in our app component.

data "terraform_remote_state" "network" {
  backend = "s3"

  config {
    bucket = "company-terraform-state"
    key    = "prod/network/terraform.tfstate"
    region = "eu-west-1"
  }
}

Now that we have a reference setup, we can use the values defined as outputs.

  vpc_id = data.terraform_remote_state.network.vpc_id

Registry

Building a full configuration for every component in your infrastructure is no longer needed. Terraform has created a registry where you can use community modules that will get you setup in no time. These modules can help you skip a lot of the hard steps in setting up your projects.

Note that I would suggest only using verified modules if possible and make sure to lock the version of your module once you are happy with the result as changes by the community may cause unwanted changes in your infrastructure.