We have already seen how we can build infrastructure with the basics of Terraform, some primary commands and fundamentals in the first part of this series. Now let’s see how Terraform Modules work and the benefits of it. I highly recommend to read first part of the series before starting with the second part.
As you manage your infrastructure with Terraform, increasingly complex configurations will be created. There is no intrinsic limit to the complexity of a single Terraform configuration file or directory, so it is possible to continue writing and updating your configuration files in a single directory. However, if you do, you may encounter one or more of the following problems:
- Understanding and navigating the configuration files will become increasingly difficult.
- Updating the configuration will become riskier, because an update to one block may cause unintended consequences to other blocks of your configuration.
- Duplication of similar blocks of configuration may increase, for example, when you configure separate dev/staging/production environments, which will cause an increasing burden when updating those parts of your configuration.
- If you want to share parts of your configuration between projects and teams, cutting and pasting blocks of configuration between projects could be error-prone and hard to maintain.
What are modules for?
Here are some of the ways that modules help solve the problems listed above:
- Organize configuration: Modules make it easier to navigate, understand, and update your configuration by keeping related parts of your configuration together. Even moderately complex infrastructure can require hundreds or thousands of lines of configuration to implement. By using modules, you can organize your configuration into logical components.
- Encapsulate configuration: Another benefit of using modules is to encapsulate configuration into distinct logical components. Encapsulation can help prevent unintended consequences—such as a change to one part of your configuration accidentally causing changes to other infrastructure—and reduce the chances of simple errors like using the same name for two different resources.
- Re-use configuration: Writing all of your configuration without using existing code can be time-consuming and error-prone. Using modules can save time and reduce costly errors by re-using configuration written either by yourself, other members of your team, or other Terraform practitioners who have published modules for you to use. You can also share modules that you have written with your team or the general public, giving them the benefit of your hard work.
- Provide consistency and ensure best practices: Modules also help to provide consistency in your configurations. Consistency makes complex configurations easier to understand, and it also helps to ensure that best practices are applied across all of your configurations. For example, cloud providers offer many options for configuring object storage services, such as Amazon S3 (Simple Storage Service) or Google’s Cloud Storage buckets. Many high-profile security incidents have involved incorrectly secured object storage, and because of the number of complex configuration options involved, it’s easy to accidentally misconfigure these services.
Using modules can help reduce these errors. For example, you might create a module to describe how all of your organization’s public website buckets will be configured, and another module for private buckets used for logging applications. Also, if a configuration for a type of resource needs to be updated, using modules allows you to make that update in a single place and have it been applied to all cases where you use that module.
What is a Terraform module?
A Terraform module is a set of Terraform configuration files in a single directory. Even a simple configuration consisting of a single directory with one or more .tf files is a module. When you run Terraform commands directly from such a directory, it is considered the root module. So, in this sense, every Terraform configuration is part of a module. You may have a simple set of Terraform configuration files like this:
$ tree my-module/
.
├── LICENSE
├── README.md
├── main.tf
├── variables.tf
├── outputs.tf
In this case, when you run Terraform commands from within the my-module directory, the contents of that directory are considered the root module.
Calling modules
Terraform commands will only directly use the configuration files in one directory, which is usually the current working directory. However, your configuration can use module blocks to call modules in other directories. When Terraform encounters a module block, it loads and processes that module’s configuration files.
A module that is called by another configuration is sometimes referred to as a “child module” of that configuration.
Local and remote modules
Modules can be loaded from either the local filesystem or a remote source. Terraform supports a variety of remote sources, including the Terraform Registry, most version control systems, HTTP URLs, and Terraform Cloud or Terraform Enterprise private module registries.
Module best practices
In many ways, Terraform modules are similar to the concepts of libraries, packages, or modules found in most programming languages, and they provide many of the same benefits. Just like almost any non-trivial computer program, real-world Terraform configurations should almost always use modules to provide the benefits mentioned above.
It is recommended that every Terraform practitioner use modules by following these best practices:
- Start writing your configuration with a plan for modules. Even for slightly complex Terraform configurations managed by a single person, the benefits of using modules outweigh the time it takes to use them properly.
- Use local modules to organize and encapsulate your code. Even if you aren’t using or publishing remote modules, organizing your configuration in terms of modules from the beginning will significantly reduce the burden of maintaining and updating your configuration as your infrastructure grows in complexity.
- Use the public Terraform Registry to find useful modules. This way you can quickly and confidently implement your configuration by relying on the work of others.
- Publish and share modules with your team. Most infrastructure is managed by a team of people, and modules are an important tool that teams can use to create and maintain infrastructure. As mentioned earlier, you can publish modules either publicly or privately. You will see how to do this in a later section of this article.
Use modules from the Registry
Assumptions: I’m assuming that you are running all these following commands from the Google Cloud Shell.
Open the Terraform Registry page for the Terraform Network module.
The page includes information about the module and a link to the source repository. The right side of the page includes a dropdown interface to select the module version and instructions for using the module to provision infrastructure.
When you call a module, the source argument is required. Terraform will search for a module in the Terraform registry that matches the given string. You could also use a URL or local file path for the source of your modules. See the Terraform documentation for a list of possible module sources.
The other argument shown here is the version. For supported sources, the version will let you define what version or versions of the module will be loaded. We will use the exact version here. You can read about more ways to specify versions in the module documentation.
git clone https://github.com/terraform-google-modules/terraform-google-network.git
cd terraform-google-network/examples/simple_project
cat main.tf It will look something like below.
module "test-vpc-module" {
source = "terraform-google-modules/network/google"
version = "~> 4.0.1"
project_id = var.project_id # Replace this with your project ID in quotes
network_name = "my-custom-mode-network"
mtu = 1460
subnets = [
{
subnet_name = "subnet-01"
subnet_ip = "10.10.10.0/24"
subnet_region = "us-west1"
},
{
subnet_name = "subnet-02"
subnet_ip = "10.10.20.0/24"
subnet_region = "us-west1"
subnet_private_access = "true"
subnet_flow_logs = "true"
},
{
subnet_name = "subnet-03"
subnet_ip = "10.10.30.0/24"
subnet_region = "us-west1"
subnet_flow_logs = "true"
subnet_flow_logs_interval = "INTERVAL_10_MIN"
subnet_flow_logs_sampling = 0.7
subnet_flow_logs_metadata = "INCLUDE_ALL_METADATA"
}
]
}It includes a few important blocks:
provider "google"defines your provider.subnetsare defined directly as an array with values. However, you can parameterize with variables.module "test-vpc-module"defines a Virtual Private Cloud (VPC), which will provide networking services for the rest of your infrastructure.
Let’s see how to Set values for input variables
Some input variables are required, which means that the module doesn’t provide a default value; an explicit value must be provided in order for Terraform to run correctly.
- Within the module
"test-vpc-module"block, review the input variables you are setting. Each of these input variables is documented in the Terraform registry input tab. The required inputs for this module are:
network_name: The name of the network being createdproject_id: The ID of the project where this VPC will be createdsubnets: The list of subnets being created
In order to use most modules, you will need to pass input variables to the module configuration. The configuration that calls a module is responsible for setting its input values, which are passed as arguments to the module block. Apart from the source and the version, most of the arguments to a module block will set variable values.
Retrieve the project Id.
gcloud config list --format 'value(core.project)'Fill in the project Id and Network Name in the variables.tf file. The Subnets are already defined in the main.tf file.
variable "project_id" {
description = "The project ID to host the network in"
default = "Replace your PROJECT_ID here"
}
variable "network_name" {
description = "The name of the VPC network being created"
default = "example-vpc"
}Define root output values
Modules also have output values, which are defined within the module with the output keyword. You can access them by referring to module.<MODULE NAME>.<OUTPUT NAME>. Like input variables, module outputs are listed under the outputs tab in the Terraform registry.
Module outputs are usually either passed to other parts of your configuration or defined as outputs in your root module.
cat output.tfThe output looks something like below code.
output "network_name" {
value = module.test-vpc-module.network_name
description = "The name of the VPC being created"
}
output "network_self_link" {
value = module.test-vpc-module.network_self_link
description = "The URI of the VPC being created"
}
output "project_id" {
value = module.test-vpc-module.project_id
description = "VPC project id"
}
output "subnets_names" {
value = module.test-vpc-module.subnets_names
description = "The names of the subnets being created"
}
output "subnets_ips" {
value = module.test-vpc-module.subnets_ips
description = "The IP and cidrs of the subnets being created"
}
output "subnets_regions" {
value = module.test-vpc-module.subnets_regions
description = "The region where subnets will be created"
}
output "subnets_private_access" {
value = module.test-vpc-module.subnets_private_access
description = "Whether the subnets will have access to Google API's without a public IP"
}
output "subnets_flow_logs" {
value = module.test-vpc-module.subnets_flow_logs
description = "Whether the subnets will have VPC flow logs enabled"
}
output "subnets_secondary_ranges" {
value = module.test-vpc-module.subnets_secondary_ranges
description = "The secondary ranges associated with these subnets"
}
output "route_names" {
value = module.test-vpc-module.route_names
description = "The routes associated with this VPC"
}Provision infrastructure
Run the following command from inside the simple_project directory.
terraform init
terraform plan
terraform apply -auto-approveUnderstand how modules work
When using a new module for the first time, you must run either terraform init or terraform get to install the module. When either of these commands is run, Terraform will install any new modules in the .terraform/modules directory within your configuration’s working directory. For local modules, Terraform will create a symlink to the module’s directory. Because of this, any changes to local modules will be effective immediately, without your having to re-run terraform get.
Clean up your infrastructure
Let’s clean up our infrastructure and start from scratch.
terraform destroy
Build your own module
Although using existing Terraform modules correctly is an important skill, every Terraform practitioner will also benefit from learning how to create modules. If we create every Terraform configuration with the assumption that it may be used as a module, then it will help you design your configurations to be flexible, reusable, and composable.
Terraform treats every configuration as a module. When you run terraform commands, or use Terraform Cloud or Terraform Enterprise to remotely run Terraform, the target directory containing Terraform configuration is treated as the root module.
Module structure
Terraform treats any local directory referenced in the source argument of a module block as a module. A typical file structure for a new module is:
$ tree my-module/
.
├── LICENSE
├── README.md
├── main.tf
├── variables.tf
├── outputs.tf
Note: None of these files are required or has any special meaning to Terraform when it uses your module. You can create a module with a single .tf file or use any other file structure you like.
Each of these files serves a purpose:
LICENSEcontains the license under which your module will be distributed. When you share your module, the LICENSE file will let people using it know the terms under which it has been made available. Terraform itself does not use this file.README.mdcontains documentation in markdown format that describes how to use your module. Terraform does not use this file, but services like the Terraform Registry and GitHub will display the contents of this file to visitors to your module’s Terraform Registry or GitHub page.main.tfcontains the main set of configurations for your module. You can also create other configuration files and organize them in a way that makes sense for your project.variables.tfcontains the variable definitions for your module. When your module is used by others, the variables will be configured as arguments in the module block. Because all Terraform values must be defined, any variables that don’t have a default value will become required arguments. A variable with a default value can also be provided as a module argument, thus overriding the default value.outputs.tfcontains the output definitions for your module. Module outputs are made available to the configuration using the module, so they are often used to pass information about the parts of your infrastructure defined by the module to other parts of your configuration.
Be aware of these files and ensure that you don’t distribute them as part of your module:
terraform.tfstateandterraform.tfstate.backupfiles contain your Terraform state and are how Terraform keeps track of the relationship between your configuration and the infrastructure provisioned by it.- The
.terraformdirectory contains the modules and plugins used to provision your infrastructure. These files are specific to an individual instance of Terraform when provisioning infrastructure, not the configuration of the infrastructure defined in.tffiles. *.tfvarsfiles don’t need to be distributed with your module unless you are also using it as a standalone Terraform configuration because module input variables are set via arguments to the module block in your configuration.
Note: If you are tracking changes to your module in a version control system such as Git, you will want to configure your version control system to ignore these files. For an example, see this .gitignore file from GitHub.
Refer to this repository for the completed terraform code created in this article.
Create a module
Run following commands to create a module structure.
mkdir my-terraform-code
cd my-terraform-code
touch main.tf
mkdir -p modules/gcs-bucket
touch main.tf
touch variables.tf
touch outputs.tf
touch README.md
touch LICENSEYour current directory (my-terraform-code) structure will look something like this:
modules/
└── gcs-bucket
├── LICENSE
├── README.md
├── main.tf
├── outputs.tf
└── variables.tf
Add the following lines to the README.md file.
# My module - GCS bucket
This module provisions Cloud Storage bucket hosting static contents.Add the following lines to LICENSE file.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.Add this Cloud Storage bucket resource to your main.tf file inside the modules/gcs-bucket directory:
resource "google_storage_bucket" "bucket" {
name = var.name
project = var.project_id
location = var.location
storage_class = var.storage_class
labels = var.labels
force_destroy = var.force_destroy
uniform_bucket_level_access = true
versioning {
enabled = var.versioning
}
dynamic "retention_policy" {
for_each = var.retention_policy == null ? [] : [var.retention_policy]
content {
is_locked = var.retention_policy.is_locked
retention_period = var.retention_policy.retention_period
}
}
dynamic "encryption" {
for_each = var.encryption == null ? [] : [var.encryption]
content {
default_kms_key_name = var.encryption.default_kms_key_name
}
}
dynamic "lifecycle_rule" {
for_each = var.lifecycle_rules
content {
action {
type = lifecycle_rule.value.action.type
storage_class = lookup(lifecycle_rule.value.action, "storage_class", null)
}
condition {
age = lookup(lifecycle_rule.value.condition, "age", null)
created_before = lookup(lifecycle_rule.value.condition, "created_before", null)
with_state = lookup(lifecycle_rule.value.condition, "with_state", null)
matches_storage_class = lookup(lifecycle_rule.value.condition, "matches_storage_class", null)
num_newer_versions = lookup(lifecycle_rule.value.condition, "num_newer_versions", null)
}
}
}
}The provider documentation is here.
Navigate to the variables.tf file in your module and add the following code:
variable "name" {
description = "The name of the bucket."
type = string
}
variable "project_id" {
description = "The ID of the project to create the bucket in."
type = string
}
variable "location" {
description = "The location of the bucket."
type = string
}
variable "storage_class" {
description = "The Storage Class of the new bucket."
type = string
default = null
}
variable "labels" {
description = "A set of key/value label pairs to assign to the bucket."
type = map(string)
default = null
}
variable "bucket_policy_only" {
description = "Enables Bucket Policy Only access to a bucket."
type = bool
default = true
}
variable "versioning" {
description = "While set to true, versioning is fully enabled for this bucket."
type = bool
default = true
}
variable "force_destroy" {
description = "When deleting a bucket, this boolean option will delete all contained objects. If false, Terraform will fail to delete buckets which contain objects."
type = bool
default = true
}
variable "iam_members" {
description = "The list of IAM members to grant permissions on the bucket."
type = list(object({
role = string
member = string
}))
default = []
}
variable "retention_policy" {
description = "Configuration of the bucket's data retention policy for how long objects in the bucket should be retained."
type = object({
is_locked = bool
retention_period = number
})
default = null
}
variable "encryption" {
description = "A Cloud KMS key that will be used to encrypt objects inserted into this bucket"
type = object({
default_kms_key_name = string
})
default = null
}
variable "lifecycle_rules" {
description = "The bucket's Lifecycle Rules configuration."
type = list(object({
# Object with keys:
# - type - The type of the action of this Lifecycle Rule. Supported values: Delete and SetStorageClass.
# - storage_class - (Required if action type is SetStorageClass) The target Storage Class of objects affected by this Lifecycle Rule.
action = any
# Object with keys:
# - age - (Optional) Minimum age of an object in days to satisfy this condition.
# - created_before - (Optional) Creation date of an object in RFC 3339 (e.g. 2017-06-13) to satisfy this condition.
# - with_state - (Optional) Match to live and/or archived objects. Supported values include: "LIVE", "ARCHIVED", "ANY".
# - matches_storage_class - (Optional) Storage Class of objects to satisfy this condition. Supported values include: MULTI_REGIONAL, REGIONAL, NEARLINE, COLDLINE, STANDARD, DURABLE_REDUCED_AVAILABILITY.
# - num_newer_versions - (Optional) Relevant only for versioned objects. The number of newer versions of an object to satisfy this condition.
condition = any
}))
default = []
}Add an output to your module in the outputs.tf file inside your module:
output "bucket" {
description = "The created storage bucket"
value = google_storage_bucket.bucket
}Like variables, outputs in modules perform the same function as they do in the root module but are accessed in a different way. A module’s outputs can be accessed as read-only attributes on the module object, which is available within the configuration that calls the module.
Return to the main.tf in your root directory and add a reference to the new module:
module "gcs-bucket" {
source = "./modules/gcs-bucket"
name = var.name
project_id = var.project_id
location = "us-east1"
lifecycle_rules = [{
action = {
type = "Delete"
}
condition = {
age = 365
with_state = "ANY"
}
}]
}Create a variables.tf file (in your root directory):
touch variables.tfAdd the following code to the variables.tf file and define the variables project_id and name:
variable "project_id" {
description = "The ID of the project in which to provision resources."
type = string
default = "YOUR PROJECT ID HERE"
}
variable "name" {
description = "Name of the buckets to create."
type = string
default = "YOUR (UNIQUE) BUCKET NAME HERE"
}Create an outputs.tf file (in your root directory):
touch outputs.tfAdd the following code in the outputs.tf file:
output "bucket-name" {
description = "Bucket names."
value = "module.gcs-bucket.bucket"
}Note: The name of your storage bucket must be globally unique. Using your name and the date is usually a good way to create a unique bucket name. You can also use your Project ID.
Install the local module
Whenever you add a new module to a configuration, Terraform must install the module before it can be used. Both the terraform get and terraform init commands will install and update modules. The terraform init command will also initialize backends and install plugins.
terraform init
terraform plan
terraform apply -auto-approveNow your bucket is ready to upload your static website files.
Clean up the website and infrastructure
terraform destroy
Conclusion
Let me know your queries and feedback in the comments. I will cover more topics in the next part of the series like storing states in bucket, importing existing infrastructure in terraform state, etc.