Tags: #tf #terraform #fastly
Summary
Store configuration, NOT data.
Why?
Data is going to be stored in the tfstate file and because of how the terraform command-line interface works (e.g. it communicates with a separate ‘core’ process over gRPC) there’s a size limitation of 4mb going over the wire.
Additional Comments
Populating resources with data externally from Terraform is very dangerous, as you’re side-stepping the protection guarantees that Terraform is designed to provide. If Terraform’s tfstate file doesn’t know about data that’s been created/pushed externally (i.e. data not defined within a terraform file and CRUD’ed using the terraform CLI), then Terraform is going to delete that data on the next plan/apply operation because (as far as Terraform is concerned) it shouldn’t exist.
Consider the Fastly terraform provider. We’ve seen the following situations:
Edge Dictionary loses records: Fastly provides an ‘edge dictionary’ which can be created via terraform, but once created the values can be updated via Fastly API calls. This means if you were to define initial key/values in terraform for the edge dictionary, it would mean any new content added outside of terraform (which is the whole benefit of an edge dictionary, to allow dynamic updates without requiring a redeploy of a service) will cause the new content to be lost whenever a new terraform apply is actioned (unless you set ignore_changes as stated in the Fastly documentation) as terraform’s tfstate file will not contain the new edge dictionary elements that were added via API calls.
Large number of ACL records causes gRPC error: A customer had lots of ACL records data defined in their terraform HCL and so eventually the tfstate file would be pushed via gRPC to the ‘core’ process and that’s when a gRPC error would bubble up (due to the 4mb limitation). The solution is to use terraform to manage the existence of an ACL ‘container’ and then manage the population of ACL records via some other service or tool (e.g. a script that uses the Fastly API to CRUD ACL records). To prevent terraform from getting confused you would need to set ignore_changes so it knows that any additional data discovered shouldn’t be stored in the tfstate file.
NOTE: if you manage data in terraform, and only add
ignore_changesuntil much later (e.g. after adding 50k ACL records you finally addignore_changes), then you’ll discover that terraform’s tfstate file will still know about the data and try to move it around (e.g.terraform planwill attempt to pull down 50k records and pass it internally to its ‘core’ process, thus triggering the gRPC 4mb limit error). The only way to solve the issue from the point is to remove the data from the tfstate file usingterraform rm.