Terraform Custom Providers: Expanding Infrastructure Control

Terraform Introduction

For those in the world of DevOps and IT, Terraform is a familiar tool that simplifies the process of deploying, modifying, and overseeing both on-premises and cloud-based infrastructure. Acting as an Infrastructure-as-Code (IaC) solution, Terraform lets us define the desired infrastructure we want using the human-readable HashiCorp Configuration Language (HCL). These configuration files can be easily shared, versioned, and reused, boosting team productivity.

Terraform is a widely embraced tool that offers a unified way to manage various infrastructure components across different cloud providers and more. Getting started with Terraform is straightforward, primarily requiring knowledge of basic terraform CLI commands. However, setting up a Terraform environment might present initial challenges, including deciding where and by whom Terraform should be run and ensuring the secure storage of sensitive configuration and state files.

Another benefit of Terraform is its flexibility. HashiCorp designed Terraform to be easily extendable, allowing it to accommodate different cloud providers and services through a modular architecture.

Components

The architecture of Terraform can be divided into two main parts:

• Terraform Core: This is the engine that runs every terraform CLI command. It provides a user-friendly interface, communicates with providers, manages resource state, and builds dependency graph[link].
• Terraform Plugins: These individual providers (~plugins) encapsulate the details of how to interact with cloud providers, SaaS services, and APIs. Triggered by the core through Remote Procedure Calls (RPC), providers operate independently, with their own versioning and releases.

Each provider defines the resources and data sources it can manage. Within the Terraform configuration, each resource you manage is associated with a provider that implements its functionality. Typically, providers don't directly interact with API endpoints when managing resources or services. Instead, they rely on a Go library that abstracts the service's API and handles authentication with the infrastructure provider. The diagram below illustrates this architecture. While debugging this layered setup can be complex, the separate responsibilities of each component make maintenance much easier.

Knowing When to Create Your Own Provider

With over 3400 publicly available providers accessible through the official Terraform Registry, many common use cases have been covered, from standard cloud resource management to interesting implementations like Grafana dashboards. You'll find various engaging and even entertaining contributions among the community-supported providers - like one that lets you order real pizza.

Creating a custom provider becomes useful in specific scenarios, despite the wealth of existing options. For instance, if your company offers a unique service that you want to manage declaratively using a familiar tool, or if you encounter a public service that lacks a dedicated provider, or even if you're developing an existing provider and need to test it locally, a custom solution might be the way to go.

Developing Your Own Provider

HashiCorp provides two options for creating a custom provider. The first approach involves using a Go SDK, which is more suited for maintaining older providers. However, the recommended method, as suggested in the official documentation, is to use the Plugin Framework. This approach is also encouraged for migrating legacy providers if possible. The Plugin Framework offers additional features and a higher level of abstraction. You can find a comprehensive list of features in the documentation.

In the next section, we'll walk through an example of creating a custom provider for an online text sharing tool, PasteBin. This example will provide key insights and references for a deeper dive.

Configuring the Location of the Provider Binary

As Terraform practitioners, we usually rely on the public Terraform Registry, where providers satisfy version requirements are available. While this is the default scenario during the terraform init command, Terraform also considers local providers. You can explicitly specify this using the -plugin-dir flag(s), although the default location is the .terraform.d/plugins directory. Another option is configuring the .terraformrc file in your home directory (or a specific location using the TF_CLI_CONFIG_FILE environment variable).

This approach simplifies setting up the development environment. The following snippet demonstrates how to define a provider under development, prompting Terraform to search within the default directory of the installed Go binary.

	
	provider_installation {
		   dev_overrides {
		      "adesso.eu/terraform/pastebin" = "/Users/user.name/go/bin"
		  }
		   direct {}
		}
	

	
	func (p *PastebinProvider) Schema(ctx context.Context, req provider.SchemaRequest, resp *provider.SchemaResponse) {
		    resp.Schema = schema.Schema{
		        Attributes: map[string]schema.Attribute{
		            "username": schema.StringAttribute{
		                Optional:            true,
		                MarkdownDescription: "Pastebin User's name for personal pastes",
		            },
		            "password": schema.StringAttribute{
		                Optional:            true,
		                Sensitive:           true, // obscure the value in CLI output
		            },
		            "token": schema.StringAttribute{
		                Optional:            true,
		                Sensitive:           true,
		            },
		        },
		    }
		}
	

These attributes allow you to specify parameter types, define if they're optional or required, and more. You can find a comprehensive list of attributes in the corresponding Go module documentation. Users can then configure custom parameters in the usual manner. Given the sensitivity of managing secrets in Terraform, an alternative approach could involve reading credentials from environment variables within the Configure function.

	
	provider "pastebin" {
		  username = "MyUser"
		  password = "MySecurePassword"
		  token    = "MySecureToken"
		}
	

Creating Resource and Data Source Settings

Providers are responsible for managing resources and data sources. While a provider might manage zero or more resources and data sources, most providers handle a few. These elements constitute the fundamental building blocks orchestrated within Terraform's .tf files.

Resources define the creation and management of resources in third-party services. Whether it's a virtual machine or a paste on Pastebin, resources must implement resource.Resource interface. This interface encompasses functions such as Metadata, Schema, Configure, Create, Read, Update, and Delete.

Resource definition begins with defining the attributes of the resource, which can either be configured or read-only. This structure guides Terraform's handling of resource states.

	
	type PasteResourceModel struct {
		    Title       types.String `tfsdk:"title"`
		    Url         types.String `tfsdk:"url"`
		    Content     types.String `tfsdk:"content"`
		    Id          types.String `tfsdk:"id"`
		    LastUpdated types.String `tfsdk:"last_updated"`
		}
	

Just as we saw in provider settings, schema definition is crucial for resources. The function has the same goal, offering the schema and outlining each variable while incorporating metadata that supports different stages of the resource lifecycle. This aids documentation generation and validates resource definitions in Terraform configuration files.

	
	func (r *PasteResource) Schema(ctx context.Context, req resource.SchemaRequest, resp *resource.SchemaResponse) {
		    resp.Schema = schema.Schema{
		        Attributes: map[string]schema.Attribute{
		            "id": schema.StringAttribute{
		                Computed:            true,
		                MarkdownDescription: "Resource identifier",
		                PlanModifiers: []planmodifier.String{
		                    stringplanmodifier.UseStateForUnknown(),
		                },
		            },
		            "last_updated": schema.StringAttribute{
		                Computed: true,
		            },
		            "title": schema.StringAttribute{
		                Optional: true,
		            },
		            "url": schema.StringAttribute{
		                Computed: true,
		            },
		            "content": schema.StringAttribute{
		                Required: true,
		            },
		        },
		    }
		}
	

Functions like Create, Read, Update, and Delete correspond to their actions within the resource management cycle. Below is a simplified version of the Create function. Terraform reads the defined plan into our previously defined model. Based on this data, we interact with the service or infrastructure provider APIs using the integrated Go client (r.client.CreatePaste(params)). Once a resource is successfully created, Terraform populates calculated attributes and saves them in the state.

	
	func (r *PasteResource) Create(ctx context.Context, req resource.CreateRequest, resp *resource.CreateResponse) {
		    // Read Terraform plan data into the model
		    var data PasteResourceModel
		    resp.Diagnostics.Append(req.Plan.Get(ctx, &data)...) 
		    // ... Error handling
		    // Create a new Paste
		    pasteKey, err := r.client.CreatePaste(pastebin.NewCreatePasteRequest(data.Title.ValueString(), data.Content.ValueString(), pastebin.ExpirationNever, pastebin.VisibilityUnlisted, "go"))  
		    // ... Error handling
		    // Map response body to schema and populate Computed attribute values
		    data.Id = types.StringValue(pasteKey)
		    data.LastUpdated = types.StringValue(time.Now().Format(time.RFC850))
		    data.Url = types.StringValue("https://pastebin.com/" + pasteKey)
		    // Save data into Terraform state
		    resp.Diagnostics.Append(resp.State.Set(ctx, &data)...)
		}
	

Data sources used like resources but only need to implement the Read method within the datasource.DataSource interface.

Conclusion and Next Steps

This article provides a high-level overview of the Terraform plugin architecture and offers a glimpse into what it's like to create your own provider or contribute to an existing one. While we've covered a lot, there are some aspects we haven't delved into due to space limitations. For example, topics like creating tests, setting up proper logging, importing existing resources, generating documentation, versioning, and releasing are all intriguing areas worth exploring.

I hope you found this article engaging and that it sparked ideas on how you and your business can make the most of Terraform. If you're interested, I encourage you to explore HashiCorp's tutorial and use that experience to develop your own provider. If you have queries or seek external guidance, our team is here to assist. Happy Terraforming!