The Setup

Picture this: you’ve got a function that needs to upload a bunch of files to S3, but you don’t want to be a complete savage and upload them one at a time. You also don’t want to be an idiot and try to upload 10,000 files simultaneously and bring down your entire infrastructure. What you need is something in between - a civilized approach that says “let’s do 10 at a time, and when one finishes, we can start the next.”

This is where Go’s concurrency patterns come in handy, assuming you can actually figure out what they’re doing.

Channels - The Magic Mailboxes

Channels in Go are basically message-passing tubes. Think of them like those pneumatic tubes at old-school banks, except instead of cash deposits, you’re sending Go structs around.

resultChan chan<- UploadResult  // Send-only channel
semaphore chan struct{}         // Bidirectional channel

The <- arrow tells you which direction messages can flow:

• chan<- Type = send-only, you can put things in, but not take them out
• <-chan Type = receive-only, you can take things out, but not put them in
• chan Type = bidirectional

It’s like having one-way streets for your data, which prevents you from accidentally reading from a channel you should only be writing to. Very civilized.

Semaphores - The Bouncer Pattern

Here’s where it gets clever. A semaphore is basically a bouncer with a fixed number of VIP wristbands:

semaphore := make(chan struct{}, 10)    // 10 wristbands available
semaphore <- struct{}{}                 // to start work, take a wristband

// do work here

<-semaphore                             // when done, return the wristband

The struct{}{} bit is just an empty struct that takes up zero memory. We don’t care about the data - we just care about counting how many operations are running.

When the channel buffer is full (all 10 wristbands are taken), any new goroutine trying to send to it will block and wait. When someone finishes and takes a message out, the next goroutine automatically unblocks and continues. Go’s runtime handles all this blocking/unblocking automatically, which is pretty awesome.

WaitGroups - The Attendance Tracker

WaitGroups are like a teacher doing attendance. You tell it “expect N students to finish their work” and it’ll then wait until all N report back that they’re done.

This is crucial because if your main() function ends while goroutines are still running, the program just exits and kills everything mid-flight.

The Retry Logic

The upload function also has some retry logic with exponential backoff, which is just a fancy way of saying “if something fails, wait a bit and try again, but wait longer each time.” Which I suppose takes just as long to say.

waitTime := time.Second * time.Duration(1<<attempt) // 1, 2, 4, 8 seconds...

That 1<<attempt is bit shifting - it doubles the wait each retry. Much more elegant than hardcoding a bunch of sleep values.

Putting It All Together

Here’s a complete example that shows how all these pieces work together:

package main

import (
    "fmt"
    "sync"
    "time"
)

func main() {
    files := []string{ "file1.txt", "file2.txt", "file3.txt", "file4.txt", "file5.txt" }

    // create a semaphore that allows a max of 2 concurrent uploads
    semaphore := make(chan struct{}, 2)

    // create a waitgroup to track completion
    var wg sync.WaitGroup

    // create a channel to collect results
    resultChan := make(chan string, len(files))

    // start the uploads
    for _, file := range files {
        wg.Add(1) // expect one more worker to finish
        go uploadFile(file, semaphore, &wg, resultChan)
    }

    // wait for all uploads to complete
    wg.Wait()
    close(resultChan)

    // collect and print results
    fmt.Println("Upload results:")
    for result := range resultChan {
        fmt.Println(result)
    }
}

func uploadFile(filename string, semaphore chan struct{}, wg *sync.WaitGroup, results chan<- string) {
    defer wg.Done() // called when the function exits

    // grab a semaphore slot (blocks if all slots are taken)
    semaphore <- struct{}{}
    defer func() { <-semaphore }() // release the slot when done

    fmt.Printf("Starting upload: %s\n", filename)

    // retry logic with exponential backoff
    maxRetries := 3
    var err error

    for attempt := 0; attempt <= maxRetries; attempt++ {
        if attempt > 0 {
            // wait before retry with exponential backoff
            waitTime := time.Second * time.Duration(1<<attempt) // 1s, 2s, 4s, 8s...
            fmt.Printf("Retrying upload %s (attempt %d/%d) after %v\n",
                filename, attempt+1, maxRetries+1, waitTime)
            time.Sleep(waitTime)
        }

        // simulate upload work with chance of failure
        err = simulateUpload(filename, attempt)
        if err == nil {
            break // success!
        }

        fmt.Printf("Upload failed for %s: %v\n", filename, err)
    }

    // send results back
    if err != nil {
        results <- fmt.Sprintf("✗ Failed %s after %d attempts: %v", filename, maxRetries+1, err)
    } else {
        results <- fmt.Sprintf("✓ Uploaded %s", filename)
    }

    fmt.Printf("Finished upload: %s (success: %v)\n", filename, err == nil)
}

// simulateUpload simulates an upload that might fail
func simulateUpload(filename string, attempt int) error {
    // simulate work
    time.Sleep(1 * time.Second)

    // simulate failure on first attempt for some files
    if (filename == "file2.txt" || filename == "file4.txt") && attempt < 2 {
        return fmt.Errorf("network timeout")
    }

    return nil // success
}

If you run this, you should see that only two files upload at a time due to the semaphore buffer, but all five will eventually complete before the program exits.

Why This Pattern Works

The whole setup gives you:

• Controlled concurrency: Only N operations at once
• Automatic queuing: Extra work waits in line
• Graceful handling: No errors when you hit limits, just blocking
• Clean coordination: Everything waits for everything else to finish

It’s actually quite elegant once you wrap your head around it. Much better than the Python equivalent where you’d probably end up with a ThreadPoolExecutor and a bunch of boilerplate.

Final Thoughts

Go’s concurrency primitives are pretty well designed for this sort of thing. The channel-based approach feels weird coming from other languages, but it does force you to think about data flow and coordination in a more explicit way.

Now I just need to remember all this the next time I’m staring at Go code wondering why there are arrows pointing at random struct definitions.

The key takeaway is: channels are mailboxes, semaphores are bouncers with limited wristbands, and WaitGroups are attendance trackers. Everything else is just implementation details.

Future me will thank me for writing this down.

Probably.

What is PrivateLink?

AWS PrivateLink is a networking service that enables you to securely expose your applications or services to other VPCs or on-premises networks, without requiring them to traverse the public internet or use VPN connections. With PrivateLink, you can create private endpoints in a consumer VPC that connect directly to services in a service provider VPC, using private IP addresses. This allows you to keep all traffic between the consumer and provider within the AWS network, improving security and reducing latency.

Claude 3 Opus

OK, But Why Do We Want It?

There are several reasons to make your service available via PrivateLink such as:

• Security: By keeping traffic within the AWS network and avoiding the public internet, you reduce the risk of data breaches or unauthorized access to your service. Consumers can access your service without needing to open up their firewall or whitelist your public IPs.
• Simplified Networking: PrivateLink eliminates the need for the likes of VPNs or VPC Peering. Consumers can easily create an endpoint in their VPC and start using your service, without any additional setup.
• Improved Performance: Because traffic stays within the AWS network, PrivateLink connections offer lower latency and more consistent performance compared to connections over the public internet.

Game Plan

This is what we’re going to build.

We create a VPC Endpoint Service that users can then create Endpoints from in their own VPCs. The Endpoint Service points to a Network Load Balancer. We point this NLB to an Application Load Balancer that handles our SSL encryption. From there, the ALB forwards the request to an EC2 instance running our application.

Using this we can create a VPC Endpoint in our consumer VPC. The EC2 instance in the consumer VPC can access the service running on the app servers running in the service VPC via the endpoint without needin to go out over the internet.

Base Terraform

The first thing we’re going to need is some base Terraform configuration.

terraform {
  required_version = "~> 1.5"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.39"
    }

    tls = {
      source  = "hashicorp/tls"
      version = "~> 4.0"
    }
  }
}

We’ll need the AWS provider, and because this is only a demo and we’re too lazy to get a signed certificate, we’ll make use of the TLS provider.

data "aws_availability_zones" "available" {
  state = "available"
}

We want a list of AZs in our region for later use.

Creating the Service

Our service is going to be made up of an EC2 instance behind an Application Load Balancer. We also need a Network Load Balancer for the VPC Endpoint interface.

Network

The service sits in its own VPC. We’re going to create a subnet for each AZ.

resource "aws_vpc" "service" {
  cidr_block = "10.0.0.0/16"
}

resource "aws_subnet" "service" {
  count = length(data.aws_availability_zones.available.names)

  vpc_id                  = aws_vpc.service.id
  cidr_block              = cidrsubnet(aws_vpc.service.cidr_block, 4, count.index)
  availability_zone       = data.aws_availability_zones.available.names[count.index]
  map_public_ip_on_launch = true
}

We also want a Internet Gateway

resource "aws_internet_gateway" "service" {
  vpc_id = aws_vpc.service.id
}

resource "aws_route_table" "service" {
  vpc_id = aws_vpc.service.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.service.id
  }
}

resource "aws_route_table_association" "service" {
  count = length(aws_subnet.service)

  subnet_id      = aws_subnet.service[count.index].id
  route_table_id = aws_route_table.service.id
}

We’re going to use a Security Group in multiple places. It needs HTTP/S from inside the VPC, and access out to the intenet so we can install some software on the EC2 instance running our service.

resource "aws_security_group" "service" {
  name   = "service"
  vpc_id = aws_vpc.service.id
}

resource "aws_vpc_security_group_ingress_rule" "service_https" {
  security_group_id = aws_security_group.service.id

  description = "HTTPS from Service VPC"
  cidr_ipv4   = aws_vpc.service.cidr_block
  from_port   = 443
  to_port     = 443
  ip_protocol = "tcp"
}

resource "aws_vpc_security_group_ingress_rule" "service_http" {
  security_group_id = aws_security_group.service.id

  description = "HTTP from Service VPC"
  cidr_ipv4   = aws_vpc.service.cidr_block
  from_port   = 80
  to_port     = 80
  ip_protocol = "tcp"
}

resource "aws_vpc_security_group_egress_rule" "service_all" {
  security_group_id = aws_security_group.service.id

  description = "Allow all egress"
  cidr_ipv4   = "0.0.0.0/0"
  ip_protocol = "-1"
}

Service Compute

Speaking of an EC2 instance…

data "aws_ami" "amazon_linux_2023" {
  owners      = ["amazon"]
  most_recent = true

  filter {
    name   = "name"
    values = ["al2023-ami-2023.*-x86_64"]
  }

  filter {
    name   = "architecture"
    values = ["x86_64"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

resource "aws_instance" "service" {
  ami                    = data.aws_ami.amazon_linux_2023.id
  instance_type          = "t3.nano"
  subnet_id              = aws_subnet.service[0].id
  vpc_security_group_ids = [aws_security_group.service.id]
  user_data              = file("${path.module}/userdata.sh")

  key_name = var.key_name

  tags = {
    Name = "PrivateLink Demo Service"
  }
}

The userdata for the instance will simply install and start Nginx.

#!/bin/sh

sudo yum install -y nginx.x86_64
sudo systemctl start nginx

Application Load Balancer

With our instance running, we can create an Application Load Balancer to route traffic to it.

resource "aws_lb" "service_alb" {
  name               = "service-alb"
  internal           = true
  load_balancer_type = "application"
  security_groups    = [aws_security_group.service.id]
  subnets            = aws_subnet.service[*].id
}

We need to create a Target Group containing our EC2 instance.

resource "aws_lb_target_group" "service_alb" {
  name     = "service"
  port     = 80
  protocol = "HTTP"
  vpc_id   = aws_vpc.service.id
}

resource "aws_lb_target_group_attachment" "service_alb" {
  target_group_arn = aws_lb_target_group.service_alb.arn
  target_id        = aws_instance.service.id
  port             = 80
}

The Listener for the ALB needs a certificate if we want to use HTTPS. We’re going to use the TLS Terraform provider for this, but generatlly you’d want a cetificate signed by a trusted CA.

resource "tls_private_key" "service" {
  algorithm = "RSA"
}

resource "tls_self_signed_cert" "service" {
  private_key_pem = tls_private_key.service.private_key_pem

  subject {
    common_name  = aws_lb.service_alb.dns_name
    organization = "ACME Demos"
  }

  validity_period_hours = 72 # 3 days

  allowed_uses = [
    "key_encipherment",
    "digital_signature",
    "server_auth",
  ]
}

resource "aws_acm_certificate" "service" {
  private_key      = tls_private_key.service.private_key_pem
  certificate_body = tls_self_signed_cert.service.cert_pem
}

Finally we can add our listener to the Load Balancer.

resource "aws_lb_listener" "service_alb" {
  load_balancer_arn = aws_lb.service_alb.arn
  port              = "443"
  protocol          = "HTTPS"
  ssl_policy        = "ELBSecurityPolicy-2016-08"
  certificate_arn   = aws_acm_certificate.service.arn

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.service_alb.arn
  }
}

Network Load Balancer

Now that we have our Application Load Balancer set up and pointing to the EC2 instace, we can put the Network Load Balancer in front of it.

resource "aws_lb" "service_nlb" {
  name                             = "service-nlb"
  load_balancer_type               = "network"
  enable_cross_zone_load_balancing = false
  internal                         = true
  subnets                          = aws_subnet.service[*].id
}

The Load Balancer needs to target the ALB.

resource "aws_lb_target_group" "service_nlb" {
  name        = "service-nlb"
  port        = "443"
  protocol    = "TCP"
  target_type = "alb"
  vpc_id      = aws_vpc.service.id

  health_check {
    matcher  = "200"
    path     = "/"
    port     = "443"
    protocol = "HTTPS"
  }
}

resource "aws_lb_target_group_attachment" "service_nlb" {
  port             = aws_lb_listener.service_nlb.port
  target_group_arn = aws_lb_target_group.service_nlb.arn
  target_id        = aws_lb_listener.service_alb.load_balancer_arn
}

And connecting the listener to the LB.

resource "aws_lb_listener" "service_nlb" {
  load_balancer_arn = aws_lb.service_nlb.arn
  port              = 443
  protocol          = "TCP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.service_nlb.arn
  }
}

Endpoint Service

The last thing to do is to create the VPC Endpoint Service

resource "aws_vpc_endpoint_service" "service" {
  acceptance_required        = false
  network_load_balancer_arns = [aws_lb.service_nlb.arn]
}

And share it with our target account. This is the current account for now but can be changed as needed.

data "aws_caller_identity" "current" {}

resource "aws_vpc_endpoint_service_allowed_principal" "allowed_aws_accounts" {
  vpc_endpoint_service_id = aws_vpc_endpoint_service.service.id
  principal_arn           = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"
}

Consuming the Service

With the service created we can move on to standing up an EC2 instance to consume it.

Network

First thing we’re going to want to do is create a network. This is essentially the same as the Service VPC.

resource "aws_vpc" "consumer" {
  cidr_block = "10.0.0.0/16"
}

resource "aws_subnet" "consumer" {
  count = length(data.aws_availability_zones.available.names)

  vpc_id                  = aws_vpc.consumer.id
  cidr_block              = cidrsubnet(aws_vpc.service.cidr_block, 4, count.index)
  availability_zone       = data.aws_availability_zones.available.names[count.index]
  map_public_ip_on_launch = true
}

resource "aws_internet_gateway" "consumer" {
  vpc_id = aws_vpc.consumer.id
}

resource "aws_route_table" "consumer" {
  vpc_id = aws_vpc.consumer.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.consumer.id
  }
}

resource "aws_route_table_association" "consumer" {
  count = length(aws_subnet.consumer)

  subnet_id      = aws_subnet.consumer[count.index].id
  route_table_id = aws_route_table.consumer.id
}

Consumer Compute

We need an EC2 instance we can log into. We’ll take the name of a key as a variable.

variable "key_name" {
  description = "Name of the Key Pair to use for the EC2 instances"
  type        = string
}

resource "aws_instance" "consumer" {
  ami                    = data.aws_ami.amazon_linux_2023.id
  instance_type          = "t3.nano"
  subnet_id              = aws_subnet.consumer[0].id
  vpc_security_group_ids = [aws_security_group.consumer.id]

  key_name = var.key_name

  tags = {
    Name = "PrivateLink Demo Consumer"
  }
}

resource "aws_security_group" "consumer" {
  name   = "consumer"
  vpc_id = aws_vpc.consumer.id
}

resource "aws_vpc_security_group_ingress_rule" "consumer_ssh" {
  security_group_id = aws_security_group.consumer.id

  description = "SSH from the internet"
  cidr_ipv4   = "0.0.0.0/0"
  from_port   = 22
  to_port     = 22
  ip_protocol = "tcp"
}

resource "aws_vpc_security_group_egress_rule" "consumer_all" {
  security_group_id = aws_security_group.consumer.id

  description = "Allow all egress"
  cidr_ipv4   = "0.0.0.0/0"
  ip_protocol = "-1"
}

output "consumer_public_ip" {
  description = "Public IP address of the Consumer instance"
  value       = aws_instance.consumer.public_ip
}

PrivateLink

Our VPC Endpoint is going to need a Security Group that allows ingress from the VPC.

resource "aws_security_group" "consumer_privatelink" {
  name   = "consumer_privatelink"
  vpc_id = aws_vpc.consumer.id
}

resource "aws_vpc_security_group_ingress_rule" "consumer_privatelink_https" {
  security_group_id = aws_security_group.consumer_privatelink.id

  description = "HTTPS from Consumer VPC"
  cidr_ipv4   = aws_vpc.consumer.cidr_block
  from_port   = 443
  to_port     = 443
  ip_protocol = "tcp"
}

resource "aws_vpc_security_group_egress_rule" "consumer_privatelink_all" {
  security_group_id = aws_security_group.consumer_privatelink.id

  description = "Allow all outbound"
  cidr_ipv4   = "0.0.0.0/0"
  ip_protocol = "-1"
}

And we can finally create the VPC Endpoint.

resource "aws_vpc_endpoint" "consumer_privatelink" {
  service_name       = aws_vpc_endpoint_service.service.service_name
  security_group_ids = [aws_security_group.consumer_privatelink.id]
  subnet_ids         = aws_subnet.consumer[*].id
  vpc_endpoint_type  = "Interface"
  vpc_id             = aws_vpc.consumer.id
}

output "consumer_endpoint_dns_names" {
  description = "Domain names of the Consumer VPC Endpoint"
  value       = [for dns_entry in aws_vpc_endpoint.consumer_privatelink.dns_entry : "https://${dns_entry.dns_name}"]
}

Testing

With all this deployed you should be good to test it out. Connect to the Consumer EC2 instance using the public IP address in the Terraform outputs. Once connected, cURL the first VPC Endpoint URL in the consumer_endpoint_dns_names output variable. You should see the standard Nginx landing page.

Conclusion

In this post, we walked through the process of creating VPC endpoints with AWS PrivateLink using Terraform. We started by setting up a service provider EC2 instance inside a VPC behind an Application Load Balancer and Network Load Balancer. We then created a VPC Endpoint Service and shared it with a consumer VPC, allowing the consumer to securely access the service over the AWS network.

By using PrivateLink, we were able to improve the security, performance, and operational efficiency of our service. Consumers can now access the service without exposing it to the public internet or configuring complex network settings. Traffic stays within the AWS network, reducing latency and increasing compliance.

One important aspect of the setup is the allowed principal configuration, which controls which AWS accounts or IAM users can create endpoints to connect to the service. Here we’ve manually specified the allowed principal based on the current AWS account ID. However, in a real-world scenario, you may want to automate this depending on how new consumers are to be added to the system.

AWS PrivateLink is a useful service for securely sharing resources across VPCs and accounts. It can help achieve networking goals while maintaining high levels of security and performance. The specific implementation details will depend on your use case and requirements, but the general principles outlined in this post should provide a good starting point for understanding and working with PrivateLink.

If you have any issues with the Terraform, you can find a working implementation on my GitHub.

Rather than the Terraform docs for the function showing up as the first result I got a StackOverflow (or something similar) post of someone asking the same thing. Instead of simply recommending the Terraform function the top response recommended checking out AWS’s IP Address Manager. IPAM is a feature of the VPC service that lets you plan, track, and monitor IP addresses across your workloads.

This looked cool so I wanted to play around with it a bit.

Lab 01 - Intro to IPAM

Reading through the docs the first thing we’re going to want to do is set up an IPAM pool and a VPC in a single account and a single region. In this case we’re going to go with ap-southeast-2 as that’s my closest, but you can go where ever you want. This is what we’re going to set up.

To start with we need some standard Terraform config.

terraform {
  required_version = "~> 1.5"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.39"
    }
  }
}

provider "aws" {
  region = "ap-southeast-2"

  default_tags {
    tags = {
      Stack = "IPAM Lab"
    }
  }
}

We then create our root IPAM. Note that we need to list all regions we want to create pools in along with the region the IPAM is created in. Currently we’re only using our local region but we’re going to want to do more later on.

data "aws_region" "current" {}

resource "aws_vpc_ipam" "test" {
  description = "IPAM Test"

  "operating_regions" {
    region_name = data.aws_region.current.name
  }

  tags = {
    Name = "IPAM Test"
  }
}

Apply this and we can see the IPAM in the AWS console.

Next up we add an IPAM Pool and a CIDR. This is the pool of IP addresses the IPAM can dish out. We’re going to name this after the account ID which we can pull from the aws_caller_identity data block.

data "aws_caller_identity" "main" {}

resource "aws_vpc_ipam_pool" "account" {
  description    = "IPAM Test - Account Pool"
  address_family = "ipv4"
  ipam_scope_id  = aws_vpc_ipam.test.private_default_scope_id

  tags = {
    Name = data.aws_caller_identity.main.account_id
  }
}

resource "aws_vpc_ipam_pool_cidr" "account" {
  ipam_pool_id = aws_vpc_ipam_pool.account.id
  cidr         = "10.0.0.0/16"
}

Apply this and you should now see the Private Pool

But you’ll notice this pool isn’t associated with a region. We’re going to need a region pool if we want to be able to use it to create a VPC. We’ll create this as a child of the root pool.

resource "aws_vpc_ipam_pool" "region" {
  description         = "IPAM Test - Region Pool - ${data.aws_region.current.name}"
  address_family      = "ipv4"
  ipam_scope_id       = aws_vpc_ipam_pool.account.ipam_scope_id
  locale              = data.aws_region.current.name
  source_ipam_pool_id = aws_vpc_ipam_pool.account.id

  tags = {
    Name = data.aws_region.current.name
  }
}

Delegate this pool a chunk IP addresses from the root pool. We’re going to use a /20. Then we’ll create a VPC using a /24 CIDR.

resource "aws_vpc_ipam_pool_cidr" "region" {
  ipam_pool_id   = aws_vpc_ipam_pool.region.id
  netmask_length = 20
}

resource "aws_vpc" "vpc" {
  ipv4_ipam_pool_id   = aws_vpc_ipam_pool.region.id
  ipv4_netmask_length = 24
  depends_on          = [aws_vpc_ipam_pool_cidr.region]

  tags = { Name = data.aws_region.current.name }
}

With that, we’ve now got a pool of 4096 addresses assigned to Sydney and a VPC with 256 addresses.

Lab 02 - Let’s go multi-region

This time around we’re going to add additional regions. The example we’re building will create a VPC in us-east-1 and us-west-2.

Rather than define every resource for each region manually we’re going to create a region module and re-use it. First thing to do is set up the base TF requirements. We need the provider for the main region and one for the region we’re going to deploy to.

terraform {
  required_version = "~> 1.5"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.39"

      configuration_aliases = [aws.primary, aws.region]
    }
  }
}

We need to know the ID of the parent IPAM pool, and the size of the VPC we’re going to create.

variable "parent_pool_id" {
  type        = string
  description = "ID of the parent IPAM Pool"
}

variable "vpc_cidr_size" {
  type        = number
  description = "Size of the VPC CIDR block"
}

Pull some information from the account and existing resources.

data "aws_region" "region" { provider = aws.region }

data "aws_vpc_ipam_pool" "parent" {
  provider = aws.primary
  ipam_pool_id = var.parent_pool_id
}

From here we can create the Pool and CIDR for the region. Note that these are created in the primary region where the parent IPAM Pool resides. When you open up the IPAM console in the target region you won’t be able to see the pool, however if you go to manually create a VPC in the region you will see the regional pool as an option when selecting the IPAM-allocatied IPv4 CIDR block option.

resource "aws_vpc_ipam_pool" "region" {
  provider = aws.primary

  description         = "Region Pool - ${data.aws_region.region.name}"
  address_family      = "ipv4"
  ipam_scope_id       = data.aws_vpc_ipam_pool.parent.ipam_scope_id
  locale              = data.aws_region.region.name
  source_ipam_pool_id = data.aws_vpc_ipam_pool.parent.ipam_pool_id

  tags = { Name = data.aws_region.region.name }
}

resource "aws_vpc_ipam_pool_cidr" "region" {
  provider = aws.primary

  ipam_pool_id   = aws_vpc_ipam_pool.region.id
  netmask_length = var.vpc_cidr_size
}

Then we can create the VPC in the target region

resource "aws_vpc" "vpc" {
  provider = aws.region
  ipv4_ipam_pool_id   = aws_vpc_ipam_pool.region.id
  ipv4_netmask_length = var.vpc_cidr_size
  depends_on          = [aws_vpc_ipam_pool_cidr.region]

  tags = { Name = data.aws_region.region.name }
}

Back in the parent terraform remove the aws_vpc_ipam_pool.region, aws_vpc_ipam_pool_cidr.region, and aws_vpc resources.

We’re going to add the following providers

provider "aws" {
  region = "us-west-2"
  alias  = "region01"

  default_tags {
    tags = { Stack = "IPAM Lab" }
  }
}

provider "aws" {
  region = "us-east-1"
  alias  = "region02"

  default_tags {
    tags = { Stack = "IPAM Lab" }
  }
}

data "aws_region" "region01" { provider = aws.region01 }
data "aws_region" "region02" { provider = aws.region02 }

locals {
  operating_regions = distinct([
    data.aws_region.current.name,
    data.aws_region.region01.name,
    data.aws_region.region02.name,
  ])
}

What we’re doing here is building a list of regions we want to create pools in along with the region we’re creating the IPAM in. We then remove any duplicates.

We do it this way so you change what region region01 and region02 are actually sitting in without having to make any changes to other parts of the code. They can even be the same region as the root pool. We can then iterate over this local and update the IPAM to operate in each region.

resource "aws_vpc_ipam" "test" {
  description = "IPAM Test"

  dynamic "operating_regions" {
    for_each = local.operating_regions
    content {
      region_name = operating_regions.value
    }
  }

  tags = {
    Name = "IPAM Test"
  }
}

Then We create an instance of the region module for each region.

module "region01" {
  source = "./region"

  providers = {
    aws.primary = aws,
    aws.region = aws.region01,
  }

  parent_pool_id = aws_vpc_ipam_pool.account.id
  vpc_cidr_size  = 24
}

module "region02" {
  source = "./region"

  providers = {
    aws.primary = aws,
    aws.region = aws.region02,
  }

  parent_pool_id = aws_vpc_ipam_pool.account.id
  vpc_cidr_size  = 24
}

Give this an apply and you should then be able to see a couple of regional pools.

Note that in this example, the VPC is the same size as the regional pool, taking up the entire allocation.

Lab 03 - Baking it into the AWS Org

The next move would be to enable IPAM for the entire AWS Organization (assuming you’re using one). In our example we’ll have a central Shared Services account, and two Test accounts. We will delegate the IPAM for the organization to the Shared Services accounts.

To do this first you’re going to want to create these three accounts. Then to delegate access log into your root account, browse to AWS Organizations > Services. Look for Amazon VPC IP Address Manager. Click on “Enable trusted access”

Follow this with selecting “Show the option to enable trusted access” and entering “enabled” when asked. Next, go the IPAM console, Planning, and Organization Settings. Click on Delegate. Enter the account ID for your Shared Services account and click on Save Changes.

Switch to the Shared Services account and you’ll see this.

Awesome. So same as the first two labs, we need to set up some Terraform.

terraform {
  required_version = "~> 1.5"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.39"
    }

    random = {
      source  = "hashicorp/random"
      version = "~> 3.5"
    }
  }
}

provider "aws" {
  region = "ap-southeast-2"
  default_tags {
    tags = { Stack = "IPAM Lab" }
  }
}

We’re going to set up three spokes. Two in account one across two different regions, and one in account two.

locals {
  account_id_1 = "111111111111"
  account_id_2 = "222222222222"
}

provider "aws" {
  region = "ap-southeast-2"
  alias  = "spoke01"

  assume_role {
    role_arn = "arn:aws:iam::${local.account_id_1}:role/SharedServices-AdministratorAccess"
  }

  default_tags {
    tags = { Stack = "IPAM Lab" }
  }
}

provider "aws" {
  region = "ap-southeast-2"
  alias  = "spoke02"

  assume_role {
    role_arn = "arn:aws:iam::${local.account_id_2}:role/SharedServices-AdministratorAccess"
  }

  default_tags {
    tags = { Stack = "IPAM Lab" }
  }
}

provider "aws" {
  region = "us-west-2"
  alias  = "spoke03"

  assume_role {
    role_arn = "arn:aws:iam::${local.account_id_1}:role/SharedServices-AdministratorAccess"
  }

  default_tags {
    tags = { Stack = "IPAM Lab" }
  }
}

But before we get to that we need to configure the Organization level settings. This involves creating the IPAM and the shared pool, creating the Resource Access Manager share and sharing it to the Organization, and creating the root CDIR block.

variable "total_pool_cidr" {
  type        = string
  description = "CIDR range for the entire pool"
  default     = "10.0.0.0/8"
}

resource "aws_vpc_ipam" "main" {
  description = "IPAM Lab"

  dynamic "operating_regions" {
    for_each = local.operating_regions
    content { region_name = operating_regions.value }
  }

  tags = { Name = "IPAM Lab" }
}

resource "aws_vpc_ipam_pool" "shared" {
  description = "Org Pool"

  address_family = "ipv4"
  ipam_scope_id  = aws_vpc_ipam.main.private_default_scope_id
}

resource "aws_ram_resource_share" "ipam_pool" {
  name                      = "IPAM Lab"
  allow_external_principals = false
  permission_arns           = [
    "arn:aws:ram::aws:permission/AWSRAMDefaultPermissionsIpamPool"
  ]
}

resource "aws_ram_resource_association" "ipam_pool" {
  resource_arn       = aws_vpc_ipam_pool.shared.arn
  resource_share_arn = aws_ram_resource_share.ipam_pool.arn
}

data "aws_organizations_organization" "org" {}

resource "aws_ram_principal_association" "org" {
  principal          = data.aws_organizations_organization.org.arn
  resource_share_arn = aws_ram_resource_share.ipam_pool.arn
}

resource "aws_vpc_ipam_pool_cidr" "shared" {
  ipam_pool_id = aws_vpc_ipam_pool.shared.id
  cidr         = var.total_pool_cidr
}

The next step is to create a Pool per region. We do this by creating a list of regions for each spoke, then removing all duplicates. From there we can iterate over the list. But first we need our Region module. In a directory named region create the following.

terraform {
  required_version = "~> 1.5"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.39"
    }
  }
}

data "aws_region" "curent" {}

variable "parent_pool_id" {
  description = "ID of the parent address pool"
  type        = string
}

variable "netmask_length" {
  description = "Length of the Netmask to allocate to this region"
  type        = number
}

variable "region" {
  description = "Region to create the address pool for"
  type        = string
}

data "aws_vpc_ipam_pool" "parent" {
  ipam_pool_id = var.parent_pool_id
}

data "aws_region" "deploy" {}

resource "aws_vpc_ipam_pool" "deploy" {
  description         = "Region Pool - ${var.region}"
  address_family      = "ipv4"
  ipam_scope_id       = data.aws_vpc_ipam_pool.parent.ipam_scope_id
  locale              = var.region
  source_ipam_pool_id = data.aws_vpc_ipam_pool.parent.ipam_pool_id
}

resource "aws_vpc_ipam_pool_cidr" "deploy" {
  ipam_pool_id   = aws_vpc_ipam_pool.deploy.id
  netmask_length = var.netmask_length
}

output "ipam_pool_id" {
  description = "ID of the VPC IPAM pool"
  value = aws_vpc_ipam_pool.deploy.id
}

What we’re doing here is taking a Parent IPAM pool and a CIDR length and creating a child IPAM pool for the region.

We call this with the following:

data "aws_region" "main" {}
data "aws_region" "spoke01" { provider = aws.spoke01 }
data "aws_region" "spoke02" { provider = aws.spoke02 }
data "aws_region" "spoke03" { provider = aws.spoke03 }

locals {
  operating_regions = distinct([
    data.aws_region.main.name,
    data.aws_region.spoke01.name,
    data.aws_region.spoke02.name,
    data.aws_region.spoke03.name,
  ])
}

module "regions" {
  for_each = toset(local.operating_regions)

  source = "./region"

  parent_pool_id = aws_vpc_ipam_pool.shared.id
  region         = each.value
  netmask_length = 13
}

locals {
  regional_pools = {
    for k, v in module.regions : k => v.ipam_pool_id
  }
}

As described earlier, what we’re doing here is creating an aws_region data block for each region. We then use these to create a list of each region name, removing the duplicates using the distinct() Terraform function. A Region module instance is then created for each region in the list. We then create a regional_pools map linking the name of the region to the IPAM Pool ID for later reference.

Now that we’ve got a pool for each region we’re operating in, we need to create a child pool for each account operating in each region. First create an account module.

Update: There used to be a third tier pool here for the account/region combo. It didn’t work and I ripped it out while troubleshooting and didn’t keep the code (I should probably keep all this stuff in git). Instead we’re placing the VPC directly in the region pool.

terraform {
  required_version = "~> 1.5"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.39"
    }
  }
}

variable "vpc_netmask_length" {
  type        = number
  description = "Netmask length for the default VPC"
  default     = 24
}

variable "regional_pools" {
  type = map(string)
  description = "Dict of IPAM pool IDs for each region"
}

data "aws_caller_identity" "deploy" {}
data "aws_region" "deploy" {}

resource "aws_vpc" "vpc" {
  ipv4_ipam_pool_id = var.regional_pools[data.aws_region.deploy.name]
  ipv4_netmask_length = var.vpc_netmask_length

  tags = {
    Name = "${data.aws_caller_identity.deploy.account_id}/${data.aws_region.deploy.name}"
  }
}

Then back in the root we want one instance of this module per spoke provider.

module "spoke01" {
  depends_on = [aws_ram_principal_association.org]

  source = "./account"

  providers = {
    aws = aws.spoke01
  }

  regional_pools = local.regional_pools
}

module "spoke02" {
  depends_on = [aws_ram_principal_association.org]

  source = "./account"

  providers = {
    aws = aws.spoke02
  }

  regional_pools = local.regional_pools
}

module "spoke03" {
  depends_on = [aws_ram_principal_association.org]

  source = "./account"

  providers = {
    aws = aws.spoke03
  }

  regional_pools = local.regional_pools
}

With this we should be good to apply. It will set up the root IPAM Pool, a sub-pool for each region, then a VPC.

And cue the dramatic entrance of chaos and calamity, right on schedule

│ Error: creating EC2 VPC: operation error EC2: CreateVpc, https response error StatusCode: 400, RequestID:
│ 4ca0edc8-06b2-40a3-9297-cf1a80a897a4, api error InvalidIpamPoolId.NotFound: The ipam-pool ID
│ 'ipam-pool-0aaf0f8e9e9010072' does not exist

So what I was trying to do here wasn’t going to work. I had thought sharing the root IPAM Pool at the Organization level would make it available to all accounts. Technically it does, but child pools are considered their own resource. You would need to share each one out to the Organization.

My original thought was to create a pool per region, then split that up. That way if something went wrong you could instantly tell where in the world the problem was based on the IP address. Probably a hang up from when I was managing servers across multiple datacenters where finding them was easier going Region->Environment vs Environment->Region.

Though technically we could continue on this way. You would just need to create a RAM share for each regional pool. But that sounds like a pain in the ass. You could go the other way, create pool per environment then create regional pools inside it, but then you end up with the same problem of needing to create a share per environment.

No, I think I’m just going to skip the two tiered approach here and create a pool per environment/region.

Lab 04 - Simplifying things a bit

We’re going to build off of Lab 03 (no point letting all that go to waste)

Trash the account and regional stuff. You should be down to just the root IPAM pool and the RAM share.

Side tangent

I used to be against mono-repos. I like having things split up as required. However a couple of recent projects have me reconsidering that. Having to coordinate changes across three or more stacks has started to get difficult.

Having said that, I think I might still split this one one. The Org level IPAM is shared out to all accounts in the AWS Organization. You don’t have to update the share each time you add a new account. This would be my main concern against splitting it up and it doesn’t apply here.

But if we do split it up, then each account/region becomes their own environment. The only thing that we need to pass in is the root IPAM ID and that isn’t likely to ever change.

Yeah, I think I like that. However, I’m going to take it in as a variable rather than read it from a remote state.

Back to it

OK, so we’ve got more to rip out. Get rid of the providers for the spokes. Add a variable for a list of regions to operate in. We’re then going to create a local that takes this list, adds the current region to it, and removes any duplicates.

variable "operating_regions" {
  description = "List of regions to create the Pool for"
  type        = list(string)
  default     = ["ap-southeast-2", "us-west-2"]
}

data "aws_region" "current" {}

locals {
  operating_regions = distinct(concat([data.aws_region.current.name], var.operating_regions))
}

Add an output to the stack for the IPAM ID.

output "ipam_id" {
  description = "ID of the VPC IPAM"
  value       = aws_vpc_ipam.main.id
}

With that done you should be good to apply. The IPAM ID will be spit out at the end. We’re going to need this later.

In another directory we’re going to start from scratch for a spoke.

terraform {
  required_version = "~> 1.5"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.39"
    }
  }
}

provider "aws" {
  region = "ap-southeast-2"

  default_tags {
    tags = { Stack = "IPAM Lab" }
  }
}

variable "region" {
  description = "ID of the AWS region to deploy to"
  type        = string
}

variable "iam_role" {
  description = "ARN of the IAM role in the target account to assume"
  type        = string
}

provider "aws" {
  alias  = "deploy"
  region = var.region

  assume_role {
    role_arn = var.iam_role
  }

  default_tags {
    tags = {
      Stack = "IPAM Lab"
    }
  }
}

Next up we want to create the IPAM pool

variable "parent_pool_id" {
  description = "ID of the parent IPAM pool"
  type        = string
}

variable "netmask_length" {
  description = "Length of the Netmask to allocate to this spoke"
  type        = number
}

variable "name" {
  description = "Name of the spoke"
  type        = string
  default     = null
}

data "aws_caller_identity" "deploy" {
  provider = aws.deploy
}

data "aws_vpc_ipam_pool" "parent" {
  ipam_pool_id = var.parent_pool_id
}

locals {
  pool_name = "${data.aws_caller_identity.deploy.account_id}/${var.region}"
}

resource "aws_vpc_ipam_pool" "pool" {
  description         = "Spoke Pool - ${local.pool_name}"
  address_family      = "ipv4"
  ipam_scope_id       = data.aws_vpc_ipam_pool.parent.ipam_scope_id
  locale              = var.region
  source_ipam_pool_id = data.aws_vpc_ipam_pool.parent.ipam_pool_id
}

We give that an apply to see how we’re going and … damn. Another road block.

│ Error: creating IPAM Pool: UnauthorizedOperation: You are not authorized to perform this operation. User:
│ arn:aws:sts::1111111111111:assumed-role/SharedServices-AdministratorAccess/aws-go-sdk-1714091794708223000 is not
│ authorized to perform: ec2:CreateIpamPool on resource: arn:aws:ec2::222222222222:ipam-pool/ipam-pool-053687ec45ef90dfc
│ because no resource-based policy allows the ec2:CreateIpamPool action.

A quick check of the policy on the RAM share and indeed, remote account cannot create sub-pools. Bugger.

At this point I’m already a bit annoyed we have to tell IPAM which regions we may possible want to operate in in advance, I really don’t want to have to deal with sharing the sub-pools.

What if we don’t share the root IPAM pool, but create a single sub-pool that uses the entirety of the root and share that?

…

And it occurs to me that a pool needs to be tied to a region for us to be able to make any real use of it. Which may explain why I can’t do a lot with the root pool share in the other accounts/regions. This is what I get for taking so long to write this post.

It’s at this point it occurs to me that we can use a single RAM share and include all regional Pools in it. We don’t have to create a share for each pool.

Okay back to the root stack we create a pool per region.

resource "aws_vpc_ipam_pool" "shared" {
  for_each = toset(local.operating_regions)

  description = "Shared Pool - ${each.value}"

  address_family      = "ipv4"
  ipam_scope_id       = aws_vpc_ipam_pool.org.ipam_scope_id
  source_ipam_pool_id = aws_vpc_ipam_pool.org.id
  locale              = each.value
}

Update the RAM share for these.

resource "aws_ram_resource_association" "ipam_pool" {
  for_each = aws_vpc_ipam_pool.shared

  resource_arn       = each.value.arn
  resource_share_arn = aws_ram_resource_share.ipam_pool.arn
}

Now we need to rethink subnetting. We have to split up the Org subnet into each region, but we may also want to include additional regions at a later date. I’m thinking we go for a /12. This gives us 16 regions or private scopes. If we want to create /16 VPCs, we can get 16 of them per region.

Back in the org stack

resource "aws_vpc_ipam_pool_cidr" "shared" {
  for_each = toset(local.operating_regions)

  ipam_pool_id = aws_vpc_ipam_pool.shared[each.value].id
  cidr         = cidrsubnet(aws_vpc_ipam_pool_cidr.org.cidr, 4, index(local.operating_regions, each.value))
}

Apply that and you should be able to see our two regional pools

I’ve also over complicated the spokes. Let’s rip all that out and start again.

terraform {
  required_version = "~> 1.5"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.39"
    }
  }
}

provider "aws" {
  region = "ap-southeast-2"

  default_tags {
    tags = { Stack = "IPAM Lab" }
  }
}

variable "netmask_length" {
  description = "Length of the Netmask to allocate to this spoke"
  type        = number
}

We can retrieve the IPAM Pool using the name. This way we don’t have to go looking for the right pool for the region we’re deploying into.

data "aws_region" "current" {}

data "aws_vpc_ipam_pool" "pool" {
  filter {
    name   = "description"
    values = ["Shared Pool - ${data.aws_region.current.name}"]
  }
}

We can then create a VPC and some subnets

resource "aws_vpc" "vpc" {
  ipv4_ipam_pool_id   = data.aws_vpc_ipam_pool.pool.id
  ipv4_netmask_length = 20
}

data "aws_availability_zones" "available" {
  state = "available"
}

resource "aws_subnet" "public" {
  count = length(data.aws_availability_zones.available.names)

  vpc_id            = aws_vpc.vpc.id
  availability_zone = data.aws_availability_zones.available.names[count.index]
  cidr_block        = cidrsubnet(aws_vpc.vpc.cidr_block, length(data.aws_availability_zones.available.names), count.index)

  tags = {
    Name = "Shared Pool - ${data.aws_availability_zones.available.names[count.index]}"
  }
}

resource "aws_internet_gateway" "gateway" {
  vpc_id = aws_vpc.vpc.id
}

resource "aws_default_route_table" "table" {
  default_route_table_id = aws_vpc.vpc.default_route_table_id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.gateway.id
  }
}

Spin this up and you should be able to see the allocation in the IPAM console.

Switch over to the VPC console in us-west-2 and you should be able to see our subnets.

Closing thoughts

In the end I quite like AWS IPAM. Not sure I’m ever going to have a need for it though. I personally haven’t come across an AWS network large enough. And while it you could also use it to manage your on-premise network too, that seems like something that’d be easier to include from the start. But who knows, someone may decide it’s worth importing their existing environment into it.

Not sure how I feel about the requirements for a pool to be dedicated to a single region. However it does provide that Region->Environment link I was looking for.

Actually thinking about it, my current project may have been able to make use of it. But I feel it’s probably not going to be worth retroactively implementing. Oh well

If you have any difficulty following along with any of this I’ve stashed everything on GitHub.

SSH via Systems Manager

First step is to set up port forwarding via the Systems Manager agent. This will connect local port 2222 to port 22 on the remote device.

export NODE_ID=""
aws ssm start-session --target $NODE_ID --document-name AWS-StartPortForwardingSession --parameters '{"portNumber":["22"],"localPortNumber":["2222"]}'

The tunnel will operate as long as this command remains running.

From here, you can use SSH to do anything you could if you had a direct connection. For example, SCP files to or from the device.

export SSH_KEY_PATH=""
export SOURCE_PATH=""
export TARGET_PATH=""
scp -P 2222 -I $SSH_KEY_PATH -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no $SOURCE_PATH username@localhost:$TARGET_PATH

Note

If you’ve already got a record for localhost:2222 in your known hosts file this would normally fail.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.

We get around this by setting UserKnownHostsFile and StrictHostKeyChecking. You normally wouldn’t want to do this, however in our situation we re-use the same hostname/port combo for different hosts. The alternative is to either remove the entries in the known hosts file, or to dedicate one port per SSM host so you don’t run into collisions.

HTTP Proxy

This is really useful to connect to remote devices over HTTP. With the tunnel still open run the following.

export SSH_KEY_PATH=""
sudo ssh -p 2222 -i $SSH_KEY_PATH -ND 9999 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no username@localhost

Set your browser proxy config

Host: 127.0.0.1
Port: 9999
Protocol: SOCKS v5

Now just browse to any page on the remote network and your web browser will proxy the request through the SSM agent.

With the frames getting stored in S3 the next step is to process them. In this demo we’re going to run a Sobel Operator over them before saving in another S3 bucket.

Now my default GoTo for Lambda functions is Python. It’s a popular enough language you can easily find help for. However, any image processing library I know of includes C libraries that need compiling on install. This causes problems because (as far as I’m aware) Pip doesn’t support cross-compiling during library install. Which makes packaging up Lambda functions from my Macbook more pain than I want to deal with for this solution.

So I start thinking “you’ve been meaning to try out Rust for a while, why not now?” Rust has a great Lambda tool. You can use it on your local machine and have it build Lambda compatible binaries.

After getting a simple Hello World working in Lambda, I went on the hunt for a Sobel Operator library. I came across Edgy by a guy named Dan Greco. Edgy is a command line app rather than a library, but I could pull what I needed from it.

From there it was a simple case of implementing the infrastructure to trigger the Lambda on new items to the frame bucket, having the function process them, then save them into the output bucket.

Output Bucket

Before we do anything else we need somewhere to store processed frames.

resource "aws_s3_bucket" "processed_frames" {
  bucket = "mediaconvert-test-processed-frames-${random_string.suffix.result}"
}

Modify Lambda Module

We’re going to run our frame processor on an arm64 Lambda. Add the following variable to the module.

variable "architectures" {
  description = "CPU architectures to run the Lambda function on"
  default     = ["x86_64"]
  type        = list(string)
}

And update the aws_lambda_function resource

resource "aws_lambda_function" "lambda_function" {
  ...
  architectures = var.architectures
  ...
}

Lambda Code

In the Lambda functions directory run this to create the function template

cargo lambda new frame-processor

It’ll ask you some questions:

Is this function an HTTP function? N
AWS Event type that this function receives: s3::S3Event (scroll down to it)

Add the following dependencies to the Cargo.toml

anyhow = "1.0"
aws-config = "1.1.4"
aws-sdk-s3 =  "1.14.0"
image = "0.21.0"
serde = "1.0.196"
serde_json = "1.0.113"

In main.rs we want to import a few libraries. Most of this is standard Lambda stuff. image is the image library we’re going to use (duh). tracing is for logging. mod sobel refers to the sobel.rs we’re going to create further on.

extern crate aws_config;
extern crate aws_lambda_events;
extern crate aws_sdk_s3;
extern crate image;

use aws_config::{BehaviorVersion, load_defaults};
use aws_lambda_events::event::s3::S3Event;
use aws_sdk_s3::{Client, primitives::ByteStream};
use lambda_runtime::{run, service_fn, Error, LambdaEvent};
use std::{fs::File, io::Write, path::Path, env::var, str::FromStr};
use tracing::Level;

mod sobel;

A couple of static variables

static TMP_SOURCE_FILE: &str = "/tmp/source.jpg";
static TMP_OUTPUT_FILE: &str = "/tmp/output.jpg";

A function to process files. This will download the file from S3, run the Sobel Operator on it, then upload it to the output bucket using the same key as the ingress object.

async fn handle_record(client: Client, bucket: &str, key: &str) -> Result<(), Error> {
    tracing::debug!({%bucket, %key}, "Handling record");

    if !key.ends_with(".jpg") {
        tracing::info!({%key}, "File is not a JPEG image, exiting");
        return Ok(())
    }

    let output_bucket = var("OUTPUT_BUCKET").unwrap();
    let blur_modifier = var("BLUR_MODIFIER").unwrap().parse::<i32>().unwrap();

    let mut source_file = File::create(&TMP_SOURCE_FILE)?;
    let mut object = client
        .get_object()
        .bucket(bucket)
        .key(key)
        .send()
        .await?;

    while let Some(bytes) = object.body.try_next().await? {
        source_file.write_all(&bytes)?;
    }
    tracing::debug!({%key, %TMP_SOURCE_FILE}, "Downloaded file from S3 to local disk");

    sobel::process_image(TMP_SOURCE_FILE, TMP_OUTPUT_FILE, blur_modifier);
    tracing::debug!({%TMP_SOURCE_FILE, %TMP_OUTPUT_FILE, %blur_modifier}, "Finished processing image");

    let body = ByteStream::from_path(Path::new(TMP_OUTPUT_FILE)).await;
    let _ = client
        .put_object()
        .bucket(&output_bucket)
        .key(key)
        .body(body.unwrap())
        .send()
        .await;
    tracing::debug!({%TMP_OUTPUT_FILE, %output_bucket, %key}, "Uploaded output file to S3");

    Ok(())
}

We need to update the function_handler() to process the incoming S3Event.

#[tracing::instrument(skip(event), fields(req_id = %event.context.request_id))]
async fn function_handler(event: LambdaEvent<S3Event>) -> Result<(), Error> {
    let aws_config = load_defaults(BehaviorVersion::v2023_11_09()).await;
    let client = Client::new(&aws_config);

    for record in event.payload.records.iter() {
        let bucket = record.s3.bucket.name.clone().expect("Could not get bucket name from record");
        let key = record.s3.object.key.clone().expect("Could not get key from object record");

        tracing::debug!({%bucket, %key}, "Received new file");

        match handle_record(client.clone(), &bucket, &key).await {
            Ok(()) => {
                tracing::info!({%bucket, %key}, "Processed file");
            }
            Err(err) => {
                tracing::error!({%bucket, %key, %err}, "Failed to process file");
            }
        }
    }
    Ok(())
}

These events contain one or more records. Each record is a file that has been uploaded to S3. We pull the object key and bucket name from the record and make a call to the handle_record() function.

In the main() function, we’re going to update the logging functionality.

#[tokio::main]
async fn main() -> Result<(), Error> {
    tracing_subscriber::fmt().json()
        .with_max_level(Level::from_str(&std::env::var("LOG_LEVEL").unwrap()).unwrap())
        .with_current_span(false)   // remove duplicate information from the logs
        .without_time()             // disabling time is handy because CloudWatch will add the ingestion time.
        .with_target(false)         // disable printing the name of the module in every log line.
        .init();
    run(service_fn(function_handler)).await
}

What we’ve got here will take the event details from the S3 trigger, iterate over ever record in the event, and process each image using the process_image() function from the sobel module.

For the module we need to create sobel.rs in the src directory. In here we’re going to refactor some of the Edgy app.

use image::{GenericImageView, ImageBuffer, Luma};

pub fn process_image(source_filename: &str, output_filename: &str, blur_modifier: i32) {
    tracing::debug!({%source_filename, %output_filename, %blur_modifier}, "Starting image processing");
    let source = image::open(source_filename).unwrap();
    let (width, height) = source.dimensions();
    let sigma = (((width * height) as f32) / 3630000.0) * blur_modifier as f32;
    let gaussed = source.blur(sigma);
    let gray = gaussed.to_luma();

    let sobel_width:u32 = gray.width()-2;
    let sobel_height:u32 = gray.height()-2;
    let mut buff:ImageBuffer<Luma<u8>, Vec<u8>> = ImageBuffer::new(sobel_width, sobel_height);

    for i in 0..sobel_width {
        for j in 0..sobel_height {
            let val0 = gray.get_pixel(i, j).data[0] as i32;
            let val1 = gray.get_pixel(i+1, j).data[0] as i32;
            let val2 = gray.get_pixel(i+2, j).data[0] as i32;
            let val3 = gray.get_pixel(i, j+1).data[0] as i32;
            let val5 = gray.get_pixel(i+2, j+1).data[0] as i32;
            let val6 = gray.get_pixel(i, j+2).data[0] as i32;
            let val7 = gray.get_pixel(i+1, j+2).data[0] as i32;
            let val8 = gray.get_pixel(i+2, j+2).data[0] as i32;

            let gx = (-1*val0) + (-2*val3) + (-1*val6) + val2 + (2*val5) + val8;
            let gy = (-1*val0) + (-2*val1) + (-1*val2) + val6 + (2*val7) + val8;

            let mut mag = ((gx as f64).powi(2) + (gy as f64).powi(2)).sqrt();

            if mag > 255.0 {
                mag = 255.0;
            }

            buff.put_pixel(i, j, Luma([mag as u8]));
        }
    }


    buff.save(output_filename).unwrap();
}

You can compile the function with

cargo lambda build --release --arm64 --output-format binary

This will create a target/lambda/frame-processor directory that contains everything we need to upload to AWS.

Lambda Function Infrastructure

Now that we’ve built our executable, we need to create the function in AWS. First we define the access our function gets.

data "aws_iam_policy_document" "frame_processor" {
  statement {
    sid = "WriteToCloudWatchLogs"

    actions = [
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents",
    ]

    resources = ["*"]
  }

  statement {
    sid = "ReadFilesFromRawFramesBucket"

    actions = [
      "s3:GetObject"
    ]

    resources = [
      aws_s3_bucket.raw_frames.arn,
      "${aws_s3_bucket.raw_frames.arn}/*",
    ]
  }

  statement {
    sid = "WriteFilesToProcessedFramesBucket"

    actions = [
      "s3:PutObject"
    ]

    resources = [
      aws_s3_bucket.processed_frames.arn,
      "${aws_s3_bucket.processed_frames.arn}/*"
    ]
  }
}

This lets it pull images from the MediaConvert output and write to our Processed Frames bucket. It also grants permissions to write logs.

For the function itself we’re going to make use of our lambda_function module.

module "frame_processor" {
  source = "./modules/lambda_function"

  function_name = "mediaconvert-frame-processor-${random_string.suffix.result}"
  iam_policy    = data.aws_iam_policy_document.frame_processor.json
  runtime       = "provided.al2"
  architectures = ["arm64"]
  handler       = "bootstrap"
  source_dir    = "${path.root}/lambda_functions/frame_processor/target/lambda/frame-processor"
  timeout       = 120

  environment_variables = {
    "OUTPUT_BUCKET" = aws_s3_bucket.processed_frames.bucket
    "BLUR_MODIFIER" = 8
    "LOG_LEVEL"     = "INFO"
  }
}

The BLUR_MODIFIER variable adjusts how much blur is used when processing the image. Adjust this until you find the sweet spot.

Next we need to connect the function to the S3 bucket. This is done with a aws_s3_bucket_notification resource. We want to filter for ObjectCreated events so the function gets run every time a new image gets uploaded to the bucket.

To allow the bucket to trigger the Lambda function we need an aws_lambda_function resource. This grants the bucket the InvokeFunction permission.

resource "aws_lambda_permission" "frame_processor_trigger" {
  statement_id  = "AllowExecutionFromS3Bucket"
  action        = "lambda:InvokeFunction"
  function_name = module.frame_processor.function_arn
  principal     = "s3.amazonaws.com"
  source_arn    = aws_s3_bucket.raw_frames.arn
}

resource "aws_s3_bucket_notification" "frame_processor_trigger" {
  bucket = aws_s3_bucket.raw_frames.id

  lambda_function {
    lambda_function_arn = module.frame_processor.function_arn
    events              = ["s3:ObjectCreated:*"]
  }

  depends_on = [aws_lambda_permission.frame_processor_trigger]
}

Testing

I’m sure once you’ve got all this deploy you’re going to want to test it. The complete method would be to upload a video to the ingress bucket. However, if you can’t be bothered waiting for MediaConvert, you can simply upload a JPEG to the frames bucket. It needs to have a .jpg file extension as we filter for it. If everything’s working correctly you should see a file show up in the ProcessedFrames bucket.

If you’re having trouble, check out the complete diff for all changes made.

Wrapping Up

With this we’ve got a pipeline that rips videos into frames and processes them. Next steps would be to update the FrameProcessor to track the number of frames we’ve processed. We currently store the total number of frames in DynamoDB but we may want to reconsider that due to the number of IO requests per video.

Once that’s done we’ll be able to tell when we’ve processed all frames for a particular video and we can stitch it back together again. I’ll aim to get that sorted sooner than the 10 months this post took.

There was a mixture of in-house stuff written in Python, Bash, and Golang that would either be written directly to files on the device, or binaries copied from from S3 and placed where needed. There were also externally sourced packages that needed to be installed. Mostly in Deb format. These were also housed in S3. Installation and configuration would then be controlled using Ansible.

Different pieces of software were managed by different teams. As you can imagine, this meant keeping track of what’s getting installed where, and keeping versioning and dependency management straight started to become difficult.

To help clean all this up and ensure a uniform method of distributing and installing software it was decided that we would package everything up into Deb packages and create our own Apt repository.

This provided the benefet of having multiple versions of software available to us. Each package listed its own dependencies and Aptitude will handle finding and installing them for us. The team managing the IoT devices didn’t need to know what runtime would be required for a certain piece of software or what would be required to get it working; all of that was the responsibility of the package maintainer.

When it came to actually creating the repo, we wanted to run this out of S3. There are a few tools to help us build a repository, but most of them wanted to create the entire respository locally, then sync it up with your target bucket. Since we were building and releasing packages individually, we wanted to avoid this. The best tool for our usecase we found was deb-s3. This would let us create the repository and manage packages on a per-package basis.

Now that we’ve got the bones of a repository, we need to ensure we can access it from the IoT devices. The quickest way would be to enable a static website on the S3 bucket. The problem with this is that it opens the repo up to the world. We wouldn’t be able to control who can access it. Additionally, since our IoT devices didn’t have internet access, they wouldn’t be able to access it themselves. What we needed was some way to make Apt talk S3 via a VPC Endpoint.

Thankfully we also didn’t have to create this from scratch either. The apt-transport-s3 Apt method provided much of the functionality we were looking for. We took that, replaced the credentials part with something that was compatible with our IoT devices, and deployed it via Ansible.

With all that sorted we now had a uniform way we could release software and updates to our IoT devices and anything else we desired.

Building

Here I’m going to go over a quick demo of how to set something like this up. I’m not going to set up a network that’s an exact copy of ours. This one will have internet access and allow the EC2 to have access to the standard S3 Endpoint.

Set up the repository

1. Set up the Terraform environment located in this repository: terraform apply -var="ssh_key=name-of-my-ssh-key-uploaded-to-ec2"
2. Use SCP to copy the test HelloWorld package to the EC2 instance
3. Connect to the EC2 instance and run the following to build our test package: bash helloworld/build.sh. Note that you need to be in the parent directory of the package.
4. You can use dpkg to install this package to test. Run it with helloworld. Don’t forget to remove it afterwards
5. Install Ruby: sudo apt install -y ruby
6. Install deb-s3: sudo gem install deb-s3
7. Publish the package to S3:

   deb-s3 upload \
     --bucket apt-repo-test-blah-blah-blah \
     --s3-region my-aws-region \
     --visibility private \
     --preserve-versions \
     helloworld_0.1_all.deb 
   

Configure Apt to pull from the repo
This one doesn’t need to be on the same instance as the first task

1. Install apt-transport-s3
2. Add the following to your /etc/apt/sources.list

   deb [trusted=yes] s3://apt-repo-test-blah-blah-blah/ stable main 
   

3. Create /etc/apt/s3auth.conf with

   Region = 'ap-southeast-2'
   

4. Test

   sudo apt update
   sudo apt install helloworld
   helloworld
   

Follow Up

This builds a basic setup with a single EC2 instance pointing to a bucket. Some things you probably want to do afterwards include configuring apt-transport-s3 to use a VPC Endpoint and setting up signing for the Deb packages

Known Issues

Version v2.1.0 of the apt-transport-s3 package didn’t work out the door. I received the following error when running apt update

Traceback (most recent call last):
  File "/usr/lib/apt/methods/s3", line 639, in <module>
    method = S3_method(config)
  File "/usr/lib/apt/methods/s3", line 426, in __init__
    self.iam.get_credentials()
  File "/usr/lib/apt/methods/s3", line 234, in get_credentials
    str(self.iamrole, 'utf-8')))
TypeError: decoding str is not supported

This was resolved by updating line 234 from str(self.iamrole, 'utf-8'))) to self.iamrole))

He takes me across the carpark to another building where I get introduced to one of our developers. We have a chat about a project he’s been working on. It takes a GoPro video, runs each frame through a machine learning algorithm, and spits out a JSON file at the end.

Some pretty awesome shit.

Unfortunately going frame-by-frame each video ends up taking a while. The dev will load up a queue on Friday and let it run over the weekend.

Bossman starts turning starfish shaped and says “we should take this project, and push it somewhere else.” Naturally, by Somewhere Else he meant AWS. And by We he meant me.

So I did.

The goal here was to reduce the time it takes to process a video, likely by processing frames in parallel. Ideally we’d toss them in Lambda triggered off EventBridge when the frame shows up in S3. The alternative would be to just chuck it in EC2, but that would run into the same compute limitations we had with the physical hardware.

Now don’t get me wrong, this wasn’t easy. First thing was to get the processing into Lambda. Of course the whole thing was way too large. Tensorflow itself wouldn’t fit in Lambda at the time. Even once we got that sorted, the issue relevant to this post was how to split the video into frames.

Splitting a video into frames isn’t actually too hard. You could do this in Python with OpenCV, or from the CLI with FFMPEG. Our real problem was getting enough ephemeral storage in Lambda to stash not only the source video, but also each of the 65k frames that make it up, and getting it all done in the before the timeout.

With the 500MB of disk Lambda gave us at the time we could either stash the video or the frames but not both. Sweet as, what if we skip splitting the video and do it straight in the frame processing. OK, so you want to pull the video from S3 65k times each video? Hell no. Even leaving the video in memory for warm Lambdas, the S3 costs would blow out the bill.

OK, how about stashing it on an EFS disk? This worked fine for our 2 second test video, but once we tossed a couple of full ones at it, the disk started throttling. While we could increase the IO, we kept running into the same issue.

Sweet as, Lambda’s out. ECS is the next obvious choice. Aaanndd… same issues as Lambda. Lack of enough ephemeral storage.

At this point I’d given up and had started working on getting an EC2 AMI created.

In walks an AWS SA, here for an Immersion Day the boss was running. Noticing the roughly head shaped dent in my desk, he asks how the project was going.

After unloading in a way that would make a therapist take up day drinking, he asks me “have you looked at MediaConvert?” Now, I had actually come across MediaConvert in my research. While it won’t split the video into frames, it will take stills at a set rate. All the documentation indicated that you’d be looking to only grab a frame every few seconds, with the intention of using them for promotional material or something. His response? Do it anyway. Hard to argue with that. The result was bloody epic.

This had one problem. We can only grab frames x times per second. The video that gets uploaded is recorded at 29.38 FPS (or something equally not easily divisible by a round number) and I’d previously been informed that the telemetry is at the same rate. It’s at this point the dev on the project pipes up “actually, the telemetry is at 30 FPS, it’s just that it’s close enough to the video that I’ve been treating it as one-for-one.”

Office supplies may or may not have been thrown at this point. Reports vary.

Hacking away at it for a bit longer, we managed to get the whole process down to about five minutes. Fuck yeah. Even more fine tuning, ripping out unnecessary functionality, and decreasing the processed frame rate to 5 FPS (we didn’t actually need to process every frame) we were able to get a video through at 1:40ish at the fastest.

Now of course as additional functionality got added, that time’s blown out again. But still nowhere near the original 6 hours it used to take.

If you’re interested in reading more checkout the boss’s write up. There’s also a This Is My Architecture on Youtube. Otherwise continue on for how to get something similar up and running.

Something Similar

First thing we’re going to do is create a Terraform module to handle Lambda function creation. This is because we’re going to be using a few of them and they get real repetitive. In modules/lambda_function/main.tf

variable "environment_variables" {
  description = "Dict of environment variables to attach to the Lambda function"
  default     = null
}

variable "function_name" {
  description = "Name of the Lambda function and related resources"
}

variable "handler" {
  description = "Function entrypoint"
}

variable "iam_policy" {
  description = "IAM policy to attach to the Lambda Function in JSON"
}

variable "runtime" {
  description = "Lambda Runtime to deploy"
}

variable "source_dir" {
  description = "Directory for the Lambda Functions source code"
}

variable "timeout" {
  description = "Time in seconds before the Lambda function will end with a failed state"
  default     = 3
}

resource "aws_iam_role" "lambda_function" {
  name               = var.function_name
  assume_role_policy = data.aws_iam_policy_document.role.json
}

data "aws_iam_policy_document" "role" {
  statement {
    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }

    actions = ["sts:AssumeRole"]
  }
}

resource "aws_iam_role_policy" "lambda_function" {
  name   = var.function_name
  role   = aws_iam_role.lambda_function.id
  policy = var.iam_policy
}

resource "random_uuid" "suffix" {
  keepers = {
    for filename in fileset(var.source_dir, "**/*") :
    filename => filemd5("${var.source_dir}/${filename}")
  }
}

data "archive_file" "lambda_function" {
  source_dir  = var.source_dir
  output_path = "${path.module}/lambda-function-${var.function_name}-${random_uuid.suffix.result}.zip"
  type        = "zip"
}

resource "aws_lambda_function" "lambda_function" {
  function_name = var.function_name
  role          = aws_iam_role.lambda_function.arn
  filename      = data.archive_file.lambda_function.output_path
  runtime       = var.runtime
  handler       = var.handler
  timeout       = var.timeout

  dynamic "environment" {
    for_each = var.environment_variables == null ? [] : [1]
    content {
      variables = var.environment_variables
    }
  }
}

output "function_arn" {
  value = aws_lambda_function.lambda_function.arn
}

output "iam_role_arn" {
  value = aws_iam_role.lambda_function.arn
}

This will zip the function code up, create an IAM role, and deploy the function in AWS.

Uploading a Video

Moving along to the actual ingress, we want an S3 bucket to old our videos. Back in the root of the repo:

resource "random_string" "suffix" {
  length  = 16
  special = false
  upper   = false
}

resource "aws_s3_bucket" "ingress" {
  bucket = "mediaconvert-test-ingress-${random_string.suffix.result}"
}

output "ingress_bucket" {
  value = aws_s3_bucket.ingress.bucket
}

The random_string will be appended to required resource names to ensure uniqueness.

Let’s create the the function and link it to the S3 bucket:

data "aws_iam_policy_document" "ingress_processor_policy" {
  statement {
    actions = [
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]

    resources = ["*"]
  }
}

module "ingress_processor" {
  source = "./modules/lambda_function"

  function_name = "ingress-processor-${random_string.suffix.result}"
  handler       = "lambda_function.lambda_handler"
  iam_policy    = data.aws_iam_policy_document.ingress_processor_policy.json
  runtime       = "python3.7"
  source_dir    = "${path.root}/lambda_functions/ingress_processor"
}

resource "aws_lambda_permission" "ingress_trigger" {
  statement_id  = "AllowExecutionFromS3Bucket"
  action        = "lambda:InvokeFunction"
  function_name = module.ingress_processor.function_arn
  principal     = "s3.amazonaws.com"
  source_arn    = aws_s3_bucket.ingress.arn
}

resource "aws_s3_bucket_notification" "ingress_trigger" {
  bucket = aws_s3_bucket.ingress.id

  lambda_function {
    lambda_function_arn = module.ingress_processor.function_arn
    events              = ["s3:ObjectCreated:*"]
  }

  depends_on = [aws_lambda_permission.ingress_trigger]
}

And our Lambda function, lambda_functions/ingress_processor/lambda_function.py:

def lambda_handler(event, context):
    print(event)

At the moment our function doesn’t do much other than write the trigger event to CloudWatch so that we can see what we’re working with.

If you run terraform apply you’ll get a bucket name in the output. This is the bucket you want to upload videos to. Go ahead now and upload a test file. It doesn’t have to be anything special.

If you leave it sit for a while you’ll eventually see a message in the Lambda functions CloudWatch Logs.

➜  aws s3 cp ingress.tf s3://mediaconvert-test-ingress-XXXXXX/video-id-here/video.mp4
upload: ./ingress.tf to s3://mediaconvert-test-ingress-XXXXXX/video-id-here/video.mp4

If you leave it sit for a while you’ll eventually see a message in the Lambda functions CloudWatch Logs.

{
  "Records": [
    {
      "eventVersion": "2.1",
      "eventSource": "aws:s3",
      "awsRegion": "ap-southeast-2",
      "eventTime": "2022-06-13T04:25:08.424Z",
      "eventName": "ObjectCreated:Put",
      "userIdentity": {
        "principalId": "AWS:XXXXXXX"
      },
      "requestParameters": {
        "sourceIPAddress": "119.224.56.196"
      },
      "responseElements": {
        "x-amz-request-id": "XXXXX",
        "x-amz-id-2": "XXXXXX"
      },
      "s3": {
        "s3SchemaVersion": "1.0",
        "configurationId": "XXXXXXXX",
        "bucket": {
          "name": "mediaconvert-test-ingress-XXXXXX",
          "ownerIdentity": {
            "principalId": "XXXXXX"
          },
          "arn": "arn:aws:s3:::mediaconvert-test-ingress-XXXXXX"
        },
        "object": {
          "key": "video-id-here/video.mp4",
          "size": 1418,
          "eTag": "433ea94e91c11fae124bda110fab213b",
          "sequencer": "0062A6BC2452A2857F"
        }
      }
    }
  ]
}

The part we’re looking for is Records[].s3.object.key. This will tell us our filename.

Storing Video State

Now that we can ingress a video we need to actually process it. But before that we’re going to need somewhere to store details about individual videos.

resource "aws_dynamodb_table" "jobs" {
  name         = "jobs-${random_string.suffix.result}"
  billing_mode = "PAY_PER_REQUEST"

  hash_key = "VideoId"

  attribute {
    name = "VideoId"
    type = "S"
  }
}

Add the following to data.aws_iam_policy_document.ingress_processor_policy to allow the function to read and write to the DB.

statement {
  actions = [
    "dynamodb:GetItem",
    "dynamodb:BatchGetItem",
    "dynamodb:Query",
    "dynamodb:PutItem",
    "dynamodb:UpdateItem",
    "dynamodb:DeleteItem"
  ]

  resources = [aws_dynamodb_table.jobs.arn]
}

Set an environment variable in the aws_lambda_function.ingress_processor

module "ingress_processor" {
  source = "./modules/lambda_function"

  function_name = "ingress-processor-${random_string.suffix.result}"
  handler       = "lambda_function.lambda_handler"
  iam_policy    = data.aws_iam_policy_document.ingress_processor_policy.json
  runtime       = "python3.7"
  source_dir    = "${path.root}/lambda_functions/ingress_processor"

  environment_variables = {
    "JOBS_TABLE" = aws_dynamodb_table.jobs.name
  }
}

Finally update the Ingress Lambda Function to save the job.

import boto3 
import os


dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table(os.environ.get('JOBS_TABLE'))

def lambda_handler(event, context):
    for record in event['Records']:
        filename = record['s3']['object']['key']
        video_id = '/'.join(filename.split('/')[:-1])

        table.update_item(
                Key={'VideoId': video_id},
                UpdateExpression='SET ' +
                'IngressVideoBucket= :ingress_video_bucket,' +
                'IngressVideoKey= :ingress_video_key',
                ExpressionAttributeValues={
                    ':ingress_video_bucket': record['s3']['bucket']['name'],
                    ':ingress_video_key': filename
                }
        )

And if we upload a new file, same as before, we can see it in Dynamo:

Working Directories

We need some S3 buckets before we can continue. A couple to store frames in and one to stash the output.

resource "aws_s3_bucket" "raw_frames" {
  bucket = "mediaconvert-test-raw-frames-${random_string.suffix.result}"
}

resource "aws_s3_bucket" "processed_frames" {
  bucket = "mediaconvert-test-processed-frames-${random_string.suffix.result}"
}

resource "aws_s3_bucket" "egress" {
  bucket = "mediaconvert-test-egress-${random_string.suffix.result}"
}

MediaConvert IAM Role

Before we can kick off a MediaConvert job, we need an IAM role to grant it access to our resources.

resource "aws_iam_role" "mediaconvert" {
  name               = "mediaconvert-${random_string.suffix.result}"
  assume_role_policy = data.aws_iam_policy_document.mediaconvert_role.json
}

resource "aws_iam_role_policy" "mediaconvert" {
  name   = aws_iam_role.mediaconvert.name
  role   = aws_iam_role.mediaconvert.id
  policy = data.aws_iam_policy_document.mediaconvert_policy.json
}

data "aws_iam_policy_document" "mediaconvert_role" {
  statement {
    principals {
      type        = "Service"
      identifiers = ["mediaconvert.amazonaws.com"]
    }

    actions = ["sts:AssumeRole"]
  }
}

data "aws_iam_policy_document" "mediaconvert_policy" {
  statement {
    actions = [
      "s3:Get*",
      "s3:List*"
    ]

    resources = ["${aws_s3_bucket.ingress.arn}/*"]
  }

  statement {
    actions = [
      "s3:Put*"
    ]

    resources = ["${aws_s3_bucket.raw_frames.arn}/*"]
  }
}

Creating a MediaConvert Job

MediaConvert jobs are submitted by a very long JSON file. Writing this file from scratch will suck, so we’re going to submit a job via the console and export the JSON.

Browse to the MediaConvert console and click on GetStarted.

Toss the URL of your test video sitting in S3.

On the left add an Output Group. This should be a File Group. Set the output S3 URL, this’ll be our raw frames bucket.

Open up the H.264 output and enter 5000000 in the Max bitrate.

Go back up to the File Group and click on Add Output. Open up Output 2, Audio 1, and click on Remove Audio. Change the Container to “No container” and the Video Codec to “Frame Capture to JPEG” Enter 30/1 in the Frame Rate.

Back to the left, click on AWS Integration. Ensure “Use an existing service role” is selected and paste in the ARN or the IAM role created in the previous step.

You should now be good to click on Create in the bottom right.

Once the job is complete, open it up and click on View JSON in the top right. We’re going to edit this a bit and save it into lambda_functions/ingress_processor/mediaconvert_job.json.tpl

Remove everything wrapping around the Settings value until it is the root of the file. Replace the value of OutputGroups.OutputGroupSettings.FileGroupSettings.Destination with ${output_path} and the value of Inputs[0].FileInput with ${input_path}.

Back in our Ingress Lambda Function

import boto3
import json
import os
from string import Template


dynamodb = boto3.resource('dynamodb')
mediaconvert = boto3.client('mediaconvert')

mediaconvert_endpoints = mediaconvert.describe_endpoints()
mediaconvert = boto3.client('mediaconvert', endpoint_url=mediaconvert_endpoints['Endpoints'][0]['Url'])

table = dynamodb.Table(os.environ.get('JOBS_TABLE'))

frames_bucket = os.environ.get('FRAMES_BUCKET')
mediaconvert_role = os.environ.get('MEDIACONVERT_ROLE')


def lambda_handler(event, context):
    for record in event['Records']:
        filepath = record['s3']['object']['key']
        video_id = '/'.join(filepath.split('/')[:-1])
        input_bucket = record['s3']['bucket']['name']

        input_path = f's3://{input_bucket}/{filepath}'
        output_path = f's3://{frames_bucket}/{video_id}/'        

        job_config = {
            'input_path': input_path,
            'output_path': output_path
        }
        
        with open('./mediaconvert_job.json.tpl', 'r') as f:
            src = Template(f.read()).substitute(job_config)
            job = json.loads(src)
        
        res = mediaconvert.create_job(
            Role=mediaconvert_role, 
            Settings=job,
            UserMetadata={
                'videoId': video_id,
                'task': 'GENERATE_FRAMES'
            }
        )
        mediaconvert_job_id = res['Job']['Id']

        table.update_item(
                Key={'VideoId': video_id},
                UpdateExpression='SET ' +
                'IngressVideoBucket= :ingress_video_bucket,' +
                'IngressVideoKey= :ingress_video_key,' +
                'MediaConvertJobId= :mediaconvert_job_id',
                ExpressionAttributeValues={
                    ':ingress_video_bucket': record['s3']['bucket']['name'],
                    ':ingress_video_key': filepath,
                    ':mediaconvert_job_id': mediaconvert_job_id
                }
        )

And finally update our Terraform to add the additional environment variables and permissions.

data "aws_iam_policy_document" "ingress_processor_policy" {
  ...

  statement {
    actions = [
      "mediaconvert:CreateJob",
      "mediaconvert:DescribeEndpoints"
    ]

    resources = ["*"]
  }

  statement {
    actions = [
      "iam:PassRole"
    ]

    resources = [aws_iam_role.mediaconvert.arn]
  }
}

module "ingress_processor" {
  ...

  environment_variables = {
    "FRAMES_BUCKET"     = aws_s3_bucket.raw_frames.bucket
    "JOBS_TABLE"        = aws_dynamodb_table.jobs.name
    "MEDIACONVERT_ROLE" = aws_iam_role.mediaconvert.arn
  }
}

If you kick off a video now, you’ll see some output getting spit out into the Frames bucket.

Handling MediaConvert Success

One thing we need to now is how manu frames we’re going to end up with so we can keep track of progress. Unfortunately, due to how we’re extracting them, there’s no easy way to get this information out of the video in advance. So what we need to do is count the number of frames in the S3 bucket after MediaConvert has finished. This’ll be done via a Lambda Function we’re going to Trigger off an EventBridge event.

resource "aws_cloudwatch_event_rule" "mediaconvert_success" {
  name        = "mediaconvert-success-${random_string.suffix.result}"
  description = "Trigger for when a MediaConvert job finishes"

  event_pattern = <<-EOF
    {
      "source": [
        "aws.mediaconvert"
      ],
      "detail-type": [
        "MediaConvert Job State Change"
      ],
      "detail": {
        "status": [
          "COMPLETE"
        ]
      }
    }
  EOF 
}

resource "aws_cloudwatch_event_target" "mediaconvert_success_handler" {
  rule      = aws_cloudwatch_event_rule.mediaconvert_success.name
  target_id = "LambdaFunction"
  arn       = module.mediaconvert_success_handler.function_arn
}

resource "aws_lambda_permission" "mediaconvert_success_handler" {
  statement_id  = "AllowExecutionFromEventBridge"
  action        = "lambda:InvokeFunction"
  function_name = module.mediaconvert_success_handler.function_arn
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.mediaconvert_success.arn
}

module "mediaconvert_success_handler" {
  source = "./modules/lambda_function"

  function_name = "mediaconvert-success-handler-${random_string.suffix.result}"
  handler       = "lambda_function.lambda_handler"
  iam_policy    = data.aws_iam_policy_document.mediaconvert_success_handler.json
  runtime       = "python3.8"
  source_dir    = "${path.root}/lambda_functions/mediaconvert_success_handler"
  timeout       = 60 # This may need to be adjusted depending on max video length and framerate

  environment_variables = {
    "JOBS_TABLE"              = aws_dynamodb_table.jobs.name
    "PROCESSED_FRAMES_BUCKET" = aws_s3_bucket.processed_frames.bucket
    "RAW_FRAMES_BUCKET"       = aws_s3_bucket.raw_frames.bucket
  }
}

data "aws_iam_policy_document" "mediaconvert_success_handler" {
  statement {
    actions = [
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]

    resources = ["*"]
  }

  statement {
    actions = [
      "s3:ListBucket"
    ]

    resources = [
      aws_s3_bucket.raw_frames.arn,
      aws_s3_bucket.processed_frames.arn
    ]
  }

  statement {
    actions = [
      "dynamodb:GetItem",
      "dynamodb:UpdateItem"
    ]

    resources = [
      aws_dynamodb_table.jobs.arn
    ]
  }
}

Our input message looks like this

{
  "version": "0",
  "id": "8e35a40e-b5f3-cfac-4688-72345559362d",
  "detail-type": "MediaConvert Job State Change",
  "source": "aws.mediaconvert",
  "account": "XXXX",
  "time": "2022-07-03T04:30:29Z",
  "region": "ap-southeast-2",
  "resources": [
    "arn:aws:mediaconvert:ap-southeast-2:XXXX:jobs/1656822614416-ocyvn0"
  ],
  "detail": {
    "timestamp": 1656822629927,
    "accountId": "XXXX",
    "queue": "arn:aws:mediaconvert:ap-southeast-2:XXXX:queues/Default",
    "jobId": "1656822614416-ocyvn0",
    "status": "COMPLETE",
    "userMetadata": {
      "videoId": "41becbca-1fd9-460b-b2b4-9c9470dc6c21",
      "task": "GENERATE_FRAMES"
    },
    "outputGroupDetails": [
      {
        "outputDetails": [
          {
            "outputFilePaths": [
              "s3://mediaconvert-test-raw-frames-8m3yofu4o8fhw145/41becbca-1fd9-460b-b2b4-9c9470dc6c21/video.mp4"
            ],
            "durationInMs": 31664,
            "videoDetails": {
              "widthInPx": 1280,
              "heightInPx": 720
            }
          },
          {
            "outputFilePaths": [
              "s3://mediaconvert-test-raw-frames-8m3yofu4o8fhw145/41becbca-1fd9-460b-b2b4-9c9470dc6c21/video.0000949.jpg"
            ],
            "durationInMs": 31666,
            "videoDetails": {
              "widthInPx": 1280,
              "heightInPx": 720
            }
          }
        ],
        "type": "FILE_GROUP"
      }
    ]
  }
}

We can get the Video ID from one of the output paths. In lambda_functions/mediaconvert_success_handler/lambda_function.py

import boto3
import os

dynamodb = boto3.resource('dynamodb')
s3 = boto3.client('s3')

table = dynamodb.Table(os.environ.get('JOBS_TABLE'))
raw_frames_bucket = os.environ.get('RAW_FRAMES_BUCKET')
processed_frames_bucket = os.environ.get('PROCESSED_FRAMES_BUCKET')

def get_s3_key_count(bucket_name, path, continue_token=None):
    if continue_token != None:
        s3_res = s3.list_objects_v2(
                Bucket=bucket_name,
                Prefix=path,
                ContinuationToken=continue_token)
    else:
        s3_res = s3.list_objects_v2(
                Bucket=bucket_name,
                Prefix=path)

    count = s3_res['KeyCount']

    if s3_res['IsTruncated']:
        count += get_s3_key_count(bucket_name, path, s3_res['NextContinuationToken'])

    return count


def video_split_handler(event):
    video_id = event['detail']['userMetadata']['videoId']
    mediaconvert_status = event['detail']['status']

    total_frames = get_s3_key_count(raw_frames_bucket, video_id) - 1 # subtracting the video file 

    table.update_item(
            Key={'VideoId': video_id},
            UpdateExpression='SET MediaConvertStatus= :mediaconvertStatus, TotalFrames= :totalFrames',
            ExpressionAttributeValues={':mediaconvertStatus': mediaconvert_status, ':totalFrames': total_frames}
    )


def lambda_handler(event, context):
    if event.get('detail', {}).get('userMetadata', {}).get('task') == 'GENERATE_FRAMES':
            return video_split_handler(event)

From here, toss a video into the ingress bucket (remember to include a video ID directory) and watch it get ripped to pieces.

Wrapping Up

With that we have an S3 bucket we can toss videos into. They’ll automatically get picked up and run through MediaConvert. Individual frames will be dumped out into another S3 bucket, from which they can be processed further.

The biggest takeaway for me was to never just skip over a potential solution simply because it doesn’t look to be designed for your purpose. Everything is a hammer if you’re game enough.

If you’re having difficulty following allow, you can find a working copy on my GitHub.

For as long as I’ve been working for him, my boss has been repeatedly (read once or twice) mentioned that he’s been harrassing AWS to provide an Immersion Day for Direct Connect.

Direct Connect is a direct network connect between your office or data center and an AWS region. This lets you get dedicated bandwidth on a private connection that exists outside of the internet.

As you can imagine creating a lab environment where you need to be able to create and destroy dedicated WAN connections on a whim is a bit hard. Regardless, Amazon have come through and it was awesome. It was great being able to get some hands-on experience and having an instructor walk you through the process was invaluable.

Immersion Day lab environments tend to hang around for a few hours after the days has finished and this one was no exception. So after everything’s wrapped up, I tore it all down and hit up the Terraform docs.

It’s Business Time

To start with, I really recommend reading through the Provision Direct Connect Connectivity part of the Immersion Day guide. This is what we’re going to be recreating with Terraform and it’ll help provide a visual overview of what you’re doing.

If you’ve read the guide, you’ll have seend that, our existing environment cosists of multiple VPCs and a Transit Gateway. Our example has 10.0.0.0/24 as the on-prem network and 172.16.0.0./16 in AWS.

First thing you’re going to do is go to your Direct Connect provider and request a connection. The specifics of this are outside the scope of this post. Eventually you’ll see the connection in your AWS console where you can accept it.

Next we want a Direct Connect Gateway. This requires a name and an ASN for the Amazon side. We’re going to use 64512 and stash it in a variable.

variable "aws_side_bgp_asn" {
  description = "BPG ASN used on the AWS side"
  default     = 64512
}

resource "aws_dx_gateway" "dx" {
  name            = "direct-connect-lab"
  amazon_side_asn = var.aws_side_bgp_asn
}

We now need to associate the DX Gateway with our pre-existing Transit Gateway. Here we’ll need the ID of the Transit Gateway and the CIDR ranges for all VPCs attached to it. This can be one large, all encompassing block if desired.

variable "transit_gateway_id" {
  description = "ID of the pre-existing Transit Gateway to attach the DX to"
}

variable "transit_gateway_ip_prefixes" {
  description = "Subnets attached to the Transit Gateway to allow to route through the DX conn"
  default = [
    "172.16.0.0/16"
  ]
}

data "aws_ec2_transit_gateway" "example" {
  id = var.transit_gateway_id
}

resource "aws_dx_gateway_association" "dx" {
  dx_gateway_id         = aws_dx_gateway.dx.id
  associated_gateway_id = data.aws_ec2_transit_gateway.example.id

  allowed_prefixes = var.transit_gateway_ip_prefixes
}

The transit_gateway_ip_prefixes is a list of subnets attached to the Trasnit Gateway. These get advertised to the DX Gateway.

Next we create a Transit Virtual Interface and provide it the IP addresses of the Amazon and customer sides of the connection. It will also need a BGP auth key.

variable "direct_connect_connection_id" {
  description = "ID of the DX Connection created by the DX provider"
  default     = "dxcon-fg5dryvu"
}

variable "vlan" {
  description = "The VLAN used by the DX Connection. Provided by DX provider"
  default     = 222
}

variable "dx_gateway_bgp_asn" {
  description = "BGP ASN used by the DX Gateway"
  default     = 64513
}

variable "dx_amazon_address" {
  description = "IP Address of the Amazon side of the DX connection"
  default     = "169.254.100.2/30"
}

variable "dx_customer_address" {
  description = "IP Address of the customer side of the DX connection"
  default     = "169.254.100.1/30"
}

variable "dx_bgp_auth_key" {
  description = "Key used for the BGP authentication"
  default     = "badPassword"
}

resource "aws_dx_transit_virtual_interface" "dx" {
  name = "direct-connect-lab"

  connection_id = var.direct_connect_connection_id
  dx_gateway_id = aws_dx_gateway.dx.id

  vlan           = var.vlan
  address_family = "ipv4"
  bgp_asn        = var.dx_gateway_bgp_asn

  amazon_address   = var.dx_amazon_address
  customer_address = var.dx_customer_address
  bgp_auth_key     = var.dx_bgp_auth_key
}

Your router will need configuring. The following is an example for the virtual router provided in the lab. The specifics for yours will vary.

Setting the Router IP:

conf t
interface eth1
ip address 169.254.100.1/30
exit
exit
show interace brief

Configuring BGP:

conf t
route bgp 64513
neighbor 169.254.100.2 remote-as 64512
neighbor 169.254.100.2 password badPassword
address-family ipv4 unicast
network 10.0.0.0/24
exit
exit
exit
show bgp detail

With all that we should be good to deploy the Terraform.

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_dx_gateway.dx will be created
  + resource "aws_dx_gateway" "dx" {
      + amazon_side_asn  = "64512"
      + id               = (known after apply)
      + name             = "direct-connect-lab"
      + owner_account_id = (known after apply)
    }

  # aws_dx_gateway_association.dx will be created
  + resource "aws_dx_gateway_association" "dx" {
      + allowed_prefixes                    = [
          + "172.16.0.0/16",
        ]
      + associated_gateway_id               = "tgw-0dffb943a3221b366"
      + associated_gateway_owner_account_id = (known after apply)
      + associated_gateway_type             = (known after apply)
      + dx_gateway_association_id           = (known after apply)
      + dx_gateway_id                       = (known after apply)
      + dx_gateway_owner_account_id         = (known after apply)
      + id                                  = (known after apply)
    }

  # aws_dx_transit_virtual_interface.dx will be created
  + resource "aws_dx_transit_virtual_interface" "dx" {
      + address_family      = "ipv4"
      + amazon_address      = "169.254.100.2/30"
      + amazon_side_asn     = (known after apply)
      + arn                 = (known after apply)
      + aws_device          = (known after apply)
      + bgp_asn             = 64513
      + bgp_auth_key        = "badPassword"
      + connection_id       = "dxcon-fg5dryvu"
      + customer_address    = "169.254.100.1/30"
      + dx_gateway_id       = (known after apply)
      + id                  = (known after apply)
      + jumbo_frame_capable = (known after apply)
      + mtu                 = 1500
      + name                = "direct-connect-lab"
      + tags_all            = (known after apply)
      + vlan                = 222
    }

Plan: 3 to add, 0 to change, 0 to destroy.

Once that’s deployed we can connect to an EC2 instance and hit the on-prem network. In this case we’ll log into the server at 10.0.0.253.

sh-4.2$ ssh labuser@10.0.0.253
labuser@10.0.0.253's password:
Last login: Wed May 18 08:12:07 2022 from 172.16.94.57

[labuser@dns ~]$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
991: eth0@if992: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f2:e1:12:74:d9:d8 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.0.0.253/32 brd 10.0.0.253 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f0e1:12ff:fe74:d9d8/64 scope link
       valid_lft forever preferred_lft forever
993: eth1@if994: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 9a:30:45:10:60:12 brd ff:ff:ff:ff:ff:ff link-netnsid 0
995: eth2@if996: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 96:18:e7:75:7f:f6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::9418:e7ff:fe75:7ff6/64 scope link
       valid_lft forever preferred_lft forever

Random, Unstructured Closing Thoughts

The Terraform stack deploys a lot faster than you’d think. This was especially helpful as the primary on-prem server I was testing the connection with had crashed. I tore the connection down and stood it up again a few times until I noticed that it was the on-prem side reporting that the ping failed, and switching to a dffirent on-prem server proved that the connection worked.

You can accept the DX connection from the vendor using a aws_dx_connection_confirmation TF resource, however this one feels a bit iffy. Personally, I’d confirm the connection manually, then create the other required resources with Terraform.

The Immersion Day was really fun. Having the lab around afterwards was great for being able to test different deployments.

Click here to get to business.

Earlier this week I was having a chat with an AWS Architect I work with. He was talking about some of the things he’s done with a WiFi enabled Arduino. In turn I mentioned some of the sensors I picked up with the intention of getting them connected to AWS’s IoT Core service. During this discussion I may have mentioned that I’ve become a bit lazy with my personal projects and that it was probably gonna take me a while to actually get anything set up. Now, while his reaction probably didn’t have any ill intentions, it miffed me a little. So this weekend I got off my ass, pulled out an ESP32 board, and attempted to get it hooked up to IoT Core.

This didn’t go well.

I ended up having a bit more difficulty that I’d like to admit. First up, you need to determine what board you have. I grabbed the ESP32 board package from Esppresif, but then had to figure out what the Duinotech ESP32 Main Board was based off. It’s the ESP32 Dev Module by the way. Once I confirmed I was able to program the device, I ran a couple of the included examples to confirm I was able to connect to my WiFi.

With the device connected to the WiFi, I went looking for a pre-existing example of connecting it to AWS IoT. Naturally, the first result in Google was from AWS’s own blog. Following through the start of this tutorial I was able to manually set up IoT Core and get the ESP32 sending and receiving messages. However, as I’ve said previously “manual’s for chumps.” The next move was to get everything set up via Terraform.