Terraform Backend State Infrastructure

This directory contains Terraform configuration to provision the AWS infrastructure needed to store Terraform state files remotely. This includes an S3 bucket for state storage with file-based locking.

• S3 Bucket: Stores Terraform state files with versioning enabled and file-based locking
  • Name format: {prefix}-{account-id}-s3-tfstate
  • Versioning enabled (allows recovery of previous state versions)
  • Encryption at rest (AES256)
  • Private access (no public access)
  • IAM-based access control
  • Force destroy enabled (allows bucket deletion even if not empty)
• State Locking: File-based locking using use_lockfile = true in the backend configuration
  • Location: Lock file is stored in the same S3 bucket as the state file
  • Benefits: Simpler setup, lower cost, lock file stored alongside state in S3
• Security Features:
  • S3 Bucket Policy: Grants access only to the specified IAM principal
  • Public Access Block: Prevents any public access to the S3 bucket
  • Encryption: All data encrypted at rest

• OIDC Authentication: GitHub Actions authentication via AWS SSO/OIDC (no access keys needed)
• Automated State Management: Automatic state file upload and download via GitHub Actions workflows
• Local Scripts: Bash scripts for local development with automatic role assumption
• Dynamic Principal Detection: Automatically uses current caller's ARN for bucket policy
• Multi-Account Support: Designed for Account A (state storage) and Account B (resource deployment) setups

• AWS Account A (State Account) with appropriate permissions for S3 bucket creation
• GitHub repository with Actions enabled
• AWS SSO/OIDC configured (see AWS IAM Setup section)
• Terraform >= 1.2.0 (for local execution)
• AWS CLI V2
• For local execution: AWS Secrets Manager secret named github-role containing role ARNs

Required Secrets

Configure these at: Repository → Settings → Secrets and variables → Actions → Secrets

• AWS_STATE_ACCOUNT_ROLE_ARN
  • Type: Secret
  • Description: ARN of the IAM role in Account A (State Account) that trusts GitHub OIDC provider
  • Format: arn:aws:iam::ACCOUNT_A_ID:role/github-actions-state-role
  • Used for: GitHub Actions workflows (authenticating AWS API calls via OIDC)
  • ⚠️ Note: For local script execution, ensure the same role ARN is stored in AWS Secrets Manager secret 'github-role' with key 'AWS_STATE_ACCOUNT_ROLE_ARN'
• GH_TOKEN
  • Type: Secret
  • Description: GitHub Personal Access Token (PAT) with repo scope
  • Used for: Creating/updating the BACKEND_BUCKET_NAME repository variable after provisioning
  • Why needed: The default GITHUB_TOKEN may not have permissions to write repository variables

Required Variables

Configure these at: Repository → Settings → Secrets and variables → Actions → Variables

• AWS_REGION
  • Type: Variable
  • Description: AWS region where resources will be created
  • Example values: us-east-1, us-west-2, eu-west-1
  • ⚠️ Important: This should match the region in your variables.tfvars file
• BACKEND_PREFIX
  • Type: Variable
  • Description: The prefix that will be created once the state file is saved in the bucket
  • Example values: backend_state/terraform.tfstate
• BACKEND_BUCKET_NAME (Auto-generated)
  • Type: Variable
  • Description: The dynamically generated S3 bucket name
  • How it's created: Automatically set by the provisioning workflow after the bucket is created
  • ⚠️ Note: You don't need to create this manually - it's created automatically

Step 1: Create OIDC Identity Provider

1. Navigate to AWS IAM Console → Identity providers (left sidebar)
2. Click Add provider
3. Select OpenID Connect
4. Provider URL: https://token.actions.githubusercontent.com
5. Click Get thumbprint (AWS will automatically fetch GitHub's certificate thumbprint)
6. Audience: sts.amazonaws.com
7. Click Add provider

Step 2: Create IAM Role

1. Navigate to AWS IAM Console → Roles (left sidebar)
2. Click Create role
3. Under Trusted entity type, select Web identity
4. Under Web identity, select the Identity Provider: token.actions.githubusercontent.com
5. Audience: Select sts.amazonaws.com from the dropdown
6. Click Add condition to restrict which repositories can assume this role:
   • Condition key: token.actions.githubusercontent.com:sub
   • Operator: StringLike
   • Value: repo:YOUR_ORG/YOUR_REPO:* (replace with your GitHub organization and repository name)
7. Add permissions: Create or attach a policy with S3 permissions for the state bucket
8. Name the role: github-actions-state-role (or your preferred name)
9. Click Create role
10. Copy the Role ARN and set it as AWS_STATE_ACCOUNT_ROLE_ARN GitHub secret

Step 3: S3 Bucket Policy (Automatic)

The bucket policy in main.tf automatically uses the current caller's ARN via data.aws_caller_identity.current.arn. No configuration needed!

• When running via GitHub Actions: The workflow automatically detects the assumed role's ARN and uses it
• When running locally: The scripts automatically assume the IAM role and Terraform detects the assumed role's ARN
• The bucket policy always grants access to the current caller, eliminating the need for manual ARN configuration

For local development or testing, ensure you have:

• GitHub CLI installed and authenticated:

  # Install GitHub CLI (if not already installed)
  # macOS: brew install gh
  # Linux: See https://cli.github.com/manual/installation
  
  # Authenticate with GitHub
  gh auth login

• Required tools installed:
  • AWS CLI V2
  • Terraform >= 1.2.0
  • GitHub CLI (gh)
  • jq (for JSON parsing)
• AWS Secrets Manager configured:
  • Secret named github-role must exist in AWS Secrets Manager
  • Secret must contain JSON with key AWS_STATE_ACCOUNT_ROLE_ARN
  • Your AWS credentials must have secretsmanager:GetSecretValue permission
  • Example secret JSON structure:

    {
      "AWS_STATE_ACCOUNT_ROLE_ARN": "arn:aws:iam::<account-id>:role/<role-name>"
    }

This is the recommended approach as it handles state file upload automatically.

⚠️ Note: GitHub Actions workflows retrieve the role ARN directly from GitHub repository secrets (AWS_STATE_ACCOUNT_ROLE_ARN). This differs from local script execution, which uses AWS Secrets Manager.

env    = "prod"
region = "us-east-1"
prefix = "mycompany-tf"

1. Go to GitHub → Actions tab
2. Select "TF Backend State Destroying" workflow
3. Click "Run workflow" → "Run workflow"

⚠️ Warning: This permanently deletes the S3 bucket.

For local development or testing, use the provided automation scripts. These scripts handle role assumption, Terraform operations, state file management, and repository variable updates automatically.

⚠️ IMPORTANT:

• Secret Retrieval: The local bash scripts (get-state.sh and set-state.sh) retrieve the role ARN from AWS Secrets Manager (secret named 'github-role' with key 'AWS_STATE_ACCOUNT_ROLE_ARN'), not from GitHub repository secrets.
• GitHub Actions: The GitHub Actions workflows retrieve the role ARN directly from GitHub repository secrets (AWS_STATE_ACCOUNT_ROLE_ARN).
• Role Assumption: The scripts automatically assume the IAM role retrieved from AWS Secrets Manager. The S3 bucket policy grants access to the role ARN (used by GitHub Actions), not your local user ARN. Terraform will automatically detect and use the assumed role's ARN.

Use the set-state.sh script to provision infrastructure and upload the state file:

cd tf_backend_state
./set-state.sh

What the script does automatically:

1. Retrieves AWS_STATE_ACCOUNT_ROLE_ARN from AWS Secrets Manager (secret 'github-role', key 'AWS_STATE_ACCOUNT_ROLE_ARN')
2. Retrieves AWS_REGION from GitHub repository variables (defaults to us-east-1)
3. Retrieves BACKEND_PREFIX from GitHub repository variables
4. Assumes the IAM role with temporary credentials
5. Checks if infrastructure already exists:
   • If not exists: Runs terraform init, validate, plan, and apply
   • If exists: Downloads existing state file from S3 (if available)
6. Saves bucket name to GitHub repository variable BACKEND_BUCKET_NAME
7. Uploads terraform.tfstate to S3 (only if infrastructure was just provisioned)

If you need to download the state file from S3 (e.g., after running via GitHub Actions), use the get-state.sh script:

cd tf_backend_state
./get-state.sh

What the script does automatically:

1. Retrieves AWS_STATE_ACCOUNT_ROLE_ARN from AWS Secrets Manager (secret 'github-role', key 'AWS_STATE_ACCOUNT_ROLE_ARN')
2. Retrieves AWS_REGION from GitHub repository variables (defaults to us-east-1)
3. Assumes the IAM role with temporary credentials
4. Retrieves BACKEND_BUCKET_NAME and BACKEND_PREFIX from GitHub repository variables
5. Downloads terraform.tfstate from S3 if it exists

To destroy the infrastructure, use Terraform directly (after downloading the state file if needed):

cd tf_backend_state

# Download state file if needed
./get-state.sh

# Destroy infrastructure
terraform plan -var-file="variables.tfvars" -destroy -out terraform.tfplan
terraform apply -auto-approve terraform.tfplan

⚠️ Warning: This permanently deletes the S3 bucket and all resources.

After Provisioning

• The state file is automatically uploaded to: s3://{bucket-name}/{prefix}
• The bucket name is saved as BACKEND_BUCKET_NAME repository variable
• The variable is accessible to all workflows via ${{ vars.BACKEND_BUCKET_NAME }}

State File Location

• Path in S3: {prefix}
• Versioning: Enabled, so you can recover previous versions if needed

• S3 Bucket Name: {prefix}-{account-id}-s3-tfstate
  • Includes account ID to ensure global uniqueness
  • Example: mycompany-tf-123456789012-s3-tfstate

Authentication Flow

• GitHub Actions:
  1. GitHub Actions workflow runs
  2. GitHub OIDC provider issues token
  3. AWS validates token against OIDC Identity Provider
  4. AWS allows assuming IAM role
  5. Terraform uses assumed role credentials
• Local Scripts:
  1. Script retrieves role ARN from AWS Secrets Manager
  2. Script assumes IAM role using AWS CLI
  3. Terraform uses assumed role credentials

Access Control

• S3 bucket policy grants access to the IAM role ARN (used by GitHub Actions)
• Principal ARN is automatically detected from current caller (assumed role)
• Public access is completely blocked

• OIDC Authentication: No access keys needed - GitHub Actions authenticates via OIDC tokens
• IAM Role-Based Access: Bucket policy restricts access to specific IAM principal
• Public Access Block: Prevents any public access to the S3 bucket
• Encryption at Rest: All data encrypted using AES256
• Versioning: State file versioning enabled for recovery capabilities
• Secrets Management: Role ARNs stored in AWS Secrets Manager (local) or GitHub Secrets (CI/CD)

• Use OIDC: Prefer OIDC authentication over access keys for GitHub Actions
• Restrict Role Trust: Use condition keys to restrict which repositories can assume the IAM role
• Minimal Permissions: Grant only necessary S3 permissions to the IAM role
• Never Commit State Files: State files contain sensitive information and are in .gitignore
• Rotate Credentials: Regularly review and rotate IAM role permissions
• Monitor Access: Enable CloudTrail to monitor S3 bucket access

• State File: The state file contains sensitive information. Never commit it to version control (it's in .gitignore).
• Bucket Deletion: The bucket has force_destroy = true, meaning it can be deleted even if it contains files. Use with caution.
• Costs:
  • S3: Minimal cost for storage (typically < $1/month for small projects)
  • No additional costs for state locking (uses file-based locking in S3)
• Backup: State file versioning is enabled, so you can recover previous versions from the S3 console if needed.
• Multiple Environments: If you need multiple environments (dev, staging, prod), you can:
  • Use different prefixes in variables.tfvars
  • Or create separate Terraform workspaces
  • Or use separate repositories/variables for each environment

"Resource not accessible by integration" error

• Cause: GH_TOKEN doesn't have proper permissions or doesn't exist
• Solution: Create a PAT with repo scope and store it as GH_TOKEN secret

"Access Denied" when accessing S3

• Cause: The IAM principal doesn't have S3 permissions, or there's a mismatch between the caller and the bucket policy
• Solution:
  • The bucket policy automatically uses the current caller's ARN via data.aws_caller_identity.current.arn. Verify this matches your expectations:
    • Run aws sts get-caller-identity to see your current ARN
    • Ensure the caller has S3 permissions for the state bucket
  • For GitHub Actions: Verify the IAM role ARN used in AWS_STATE_ACCOUNT_ROLE_ARN secret matches the assumed role
  • Check that the OIDC trust relationship is correctly configured (for GitHub Actions)
  • For local scripts: Ensure the role ARN in AWS Secrets Manager matches the role ARN in GitHub secrets

OIDC Authentication Issues

• Cause: GitHub OIDC provider not configured correctly or role trust policy incorrect
• Solution:
  • Verify OIDC Identity Provider exists in Account A
  • Check role trust policy includes correct repository name
  • Ensure AWS_STATE_ACCOUNT_ROLE_ARN secret contains the correct role ARN (for GitHub Actions)

AWS Secrets Manager Issues (Local Scripts)

• Cause: Local scripts cannot retrieve secret from AWS Secrets Manager
• Common issues and solutions:
  • Secret doesn't exist: Ensure secret named github-role exists in AWS Secrets Manager
  • Access denied: Your AWS credentials must have secretsmanager:GetSecretValue permission for the github-role secret
  • Key not found: Ensure the secret JSON contains key AWS_STATE_ACCOUNT_ROLE_ARN
  • Invalid JSON: Verify the secret value is valid JSON format
  • Wrong region: Ensure your AWS CLI is configured to the correct region where the secret exists
• Verification:

  # Test secret retrieval manually
  aws secretsmanager get-secret-value --secret-id github-role --query SecretString --output text | jq .

Bucket name conflicts

• Cause: Another account is using the same prefix
• Solution: Use a more unique prefix in variables.tfvars

State file not found during destroy

• Cause: The state file wasn't uploaded or the bucket name variable is incorrect
• Solution: Verify BACKEND_BUCKET_NAME variable exists and contains the correct bucket name

GitHub Repository: talorlik/tf_backend_state

This project is licensed under the MIT License.

See the LICENSE file for details.