Separation of Concerns is a key principle in software engineering. When we used to deploy applications to physical hardware, the two separate concerns of infrastructure and software would often become blurred as an application would often need to tailor to particular hardware, or more likely the operating system running on that hardware.
In the modern world of Infrastructure as Code (IaC), where everything is software (almost), it’s potentially even more difficult to separate the concerns of infrastructure and software. Organisations often have two teams. A platform team to look after the infrastructure and a development team to develop the software. Once the infrastructure is created and the software is written there is an overlap between the teams which is deployment. This is where the teams must work together.
Consider AWS Lambdas. There are two separate components which are often considered as a whole, the Lambda itself within the AWS infrastructure and the code which runs within the Lambda. The Lambda itself changes infrequently, while the code changes frequently. Therefore it makes sense to separate the way both are deployed.
Terraform is a tool for creating infrastructure from code and is usually the responsibility of the platform team. As we’ll see shortly, Terraform can be used to create Lambdas in AWS, deploy and redeploy the code. However, if the development team must rely on the platform team every time they need to deploy the Lambda, delays and friction between the two teams may occur. Given the relatively stable nature of the Lambda function itself and the frequent changes to the code, it is sensible to adopt a separate deployment approach for each.
Let’s have a look at how we might do this, by first creating a basic Lambda with Terraform and then automating the deployment of iterations of the code running in the Lambda with GitHub actions.
Terraform & AWS
I am assuming that you have a working knowledge of Terraform and AWS, including configuring the AWS provider, somewhere to store Terraform state, AWS VPCs, AWS security groups, etc. The Terraform we develop here together is also available as part of a GitHub repo:https://github.com/pjgrenyer/email-sender-lambda-platform
and was developed as part of a Lambda based email sender I’m creating.
Assuming you have a Terraform project configured and ready to go, let’s start with where to store the code.
Amazon Simple Storage Service (S3)
As we want to store the Lambda code separately from the creation of the Lambda itself, we need somewhere to put it. Amazon Simple Storage Service, more commonly known as S3, is the perfect place. It’s cheap, scalable and simple to use. There are a number of ways of deploying Lambda code, but putting it in a zip file is quick and easy. This makes S3 even more ideal as we can simply upload a new version of the zip file and tell AWS to redeploy the code to the Lambda any time we want to.Let’s start by creating an S3 bucket. Remember that a S3 bucket name must be unique across all of AWS:
// s3.tf
resource "aws_s3_bucket" "email_sender_lambda" {
bucket = "email-sender-lambda"
}
A quick
terraform apply
should create the new bucket. Next we need to have some simple Lambda function code which we can zip up and use, just once, to initialise the Lambda when it’s first created. For example:
// index.js
exports.handler = async (event) => {
console.log(event);
const response = {
statusCode: 200,
body: JSON.stringify(event),
};
return response;
};
This short function sends the message received by the Lambda to the console, so that we can see it working in CloudWatch, converts the message to a string and passes it back with a status code in a new object. This means that when we invoke the Lambda, we’ll get back a message which demonstrates it’s working.
Use your favourite zip tool to zip up the index.js file into a zip file called email-sender-lambda.zip. The zip file will need to be available to Terraform. I like to put it in a subdirectory called lambdas. Then we can use the zip file to create an S3 object which, when the terraform is applied, will upload the zip file into the S3 bucket:
// lambda.tf
resource "aws_s3_object" "email_sender_lambda" {
bucket = aws_s3_bucket.email_sender_lambda.id
key = "email-sender-lambda.zip"
source = "lambdas/email-sender-lambda.zip"
}
Go ahead and apply the Terraform code to upload the zip file.
Lambda Permissions
Before we can create the Lambda itself, we need to create a role for it and give the role permissions to write logs to CloudWatch; so that we can see that it is working and debug if it doesn’t:// iam.tf
resource "aws_iam_role" "lambda" {
name = "lambda"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}
resource "aws_iam_role_policy" "lambda_role_logs_policy" {
name = "LambdaLogsPolicy"
role = aws_iam_role.lambda.id
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
EOF
}
Now it’s time to create the Lambda.
Lambda
With the zip file containing the code in S3 and the Lambda permissions in place, we have everything we need to go ahead and create the Lambda:// lambda.tf
resource "aws_lambda_function" "email_sender_lambda" {
s3_bucket = aws_s3_bucket.email_sender_lambda.id
s3_key = aws_s3_object.email_sender_lambda.key
function_name = "email-sender-lambda"
role = aws_iam_role.lambda.arn
handler = "index.handler"
publish = true
runtime = "nodejs16.x"
layers = []
}
The s3_bucket and s3_key properties refer to the S3 bucket and the name of the object to use to get the code for the Lambda. We also give the Lambda a name and its role and tell it where to find the function in the code. We want creating the Lambda to publish a new version of the code, so we set publish to true. Finally we set the nodejs version to run the code in and specify that there aren’t any AWS Lambda layers we want to use.
To prove it works we can invoke the lambda with a payload. Remember, our code should return an object with a status code and a string version of the message it receives. In this context, payload, message and event refer to much the same thing. So we need a payload. Create a file called payload.json and put it somewhere that Terraform can access it, I favour the lambdas directory I created before:
// payload.json
{
"key": "value"
}
Hopefully, as a developer using AWS, you have the AWS command line tool installed. If not, install and configure it before moving to the next step.
To execute the Lambda, we can use the AWS command line tool:
aws lambda invoke --function-name email-sender-lambda --payload file://lambdas/payload.json --cli-binary-format raw-in-base64-out response.json && more response.json
lambda invoke says that we want to execute a Lambda. Then we specify the name of the Lambda with, --function-name. Then the payload file and the format and file we want the response to go into. Finally we print the response from the file to the console.
When you execute the Lambda you should get the following response:
{
"StatusCode": 200,
"ExecutedVersion": "$LATEST"
}
{"statusCode":200,"body":"{\"key\":\"value\"}"}
Change the payload and execute the Lambda again a few times to satisfy yourself that the Lamba is working as expected. Finally, go to CloudWatch, in the AWS console, and find the log group aws/lambda/email-sender-lambda, find the most recent log stream and see that the Lambda is executing.
You’re done with your Terraform code for now, so if you’ve created a git repository for it, commit and push the latest code and then put it to one side. You’ll need it again later.
New Code
Now that we’ve got a Lambda running in AWS we need to demonstrate that we can update the code it’s running without the need to run any Terraform. First we need a suitable nodejs project. I’ve created one with the code in here:https://github.com/pjgrenyer/email-sender-lambda-code
Create the node project in the usual way using:
npm init
and copy the index.js from above in. Make a slight change to the code so that it brings in the project version from package.json and adds it to the response:
const package = require('./package.json');
exports.handler = async (event) => {
console.log(event);
const response = {
statusCode: 200,
body: JSON.stringify(event),
version: package.version
};
return response;
};
Now we have a simple way of versioning the code and seeing that the new version is deployed.
Package the Code
Now that we have code, we need to package it. We need a zip file which contains both the index.js file and now the package.json file. We also want the version number to be part of the zip file’s name, so that we know which version of the code it contains. And we’re developers and that means we’re lazy, so we want to make it as simple to do again and again as possible!There are a number of zip tools. I find the standard zip which comes with the Linux platform I’m using works well, but for this I think we need something we can bundle with our project. repack-zip is such a tool and is simple to use. You can install it with:
npm i --save-dev repack-zip
but it does need a little configuration. When you run repack-zip, you pass it a directory to zip the contents of and a name for the zip file. There are a few files we want to exclude from the zip, so add the following to your package.json:
"repackZipConfig": {
"excludes": [
"LICENSE",
"README.md",
"package-lock.json"
]
}
Then add a script to package.json, so that we can to execute repack-zip repeatedly without having to remember the full command, and use the name and version from package.json to name the zip file:
"scripts": {
…
"package": "repack-zip . ${npm_package_name}-${npm_package_version}.zip"
},
Run the script with:
npm run package
And you should find you get a zip file, with a name something like:
email-sender-lambda-0.0.1.zip
containing just the index.js and package.json files, which is what we want.
You may also notice that the zip file does not contain a node_modules directory, even though we didn’t exclude it. This is because repack-zip is clever and knows there are only dev dependencies. If there were non-dev dependencies, the node_modules directory would be included in the zip, but the dev dependencies would be excluded.
Push and Publish
Now that we have a zip file containing an update of the code for our Lambda, it’s time to publish it. First we must push the zip file into S3. We’re still feeling lazy and we need the project name and version from package.json, so we’ll create a script for that too using the AWS command line tool. Add the following script to the package.json:"upload": "aws s3 cp ${npm_package_name}-${npm_package_version}.zip s3://email-sender-lambda/${npm_package_name}-${npm_package_version}.zip",
It’s quite straightforward. We’re telling the tool to copy our zip file into the bucket and what to call it in the bucket. Run it:
npm run upload
and see what happens:
> email-sender-lambda-code@0.0.1 upload
> aws s3 cp ${npm_package_name}-${npm_package_version}.zip s3://email-sender-lambda/${npm_package_name}-${npm_package_version}.zip
upload: ./email-sender-lambda-code-0.0.1.zip to s3://email-sender-lambda/email-sender-lambda-code-0.0.1.zip
Hopefully your output is similar and the zip file is now in the S3 bucket alongside the original zip file. Log into the AWS console and check.
Next we need to tell the Lambda to use the new code. Again this can be done with the AWS command line tool and a package.json script:
"publish": "aws lambda update-function-code --function-name email-sender-lambda --s3-bucket email-sender-lambda --s3-key ${npm_package_name}-${npm_package_version}.zip"
This is quite straightforward too. We’re telling the tool that we want to update the Lambda’s code with update-function-code. We’re telling it which lambda function with --function-name, which S3 bucket the code is in with --s3-bucket and what the zip file containing the code is called with --s3-key. Run it:
npm run publish
You should get quite a lot of output, but it should indicate that it was successful.
Now, all you should need to do is execute the Lambda and see the new response which includes the version number, don’t forget to copy or recreate the payload.json file from the platform project, add it to the repack-zip excludes, and create another package.json script:
"invoke": "aws lambda invoke --function-name email-sender-lambda --payload file://payload.json --cli-binary-format raw-in-base64-out response.json && more response.json"
When you run it:
npm run invoke
You should see some output along the times of:
{
"StatusCode": 200,
"ExecutedVersion": "$LATEST"
}
{"statusCode":200,"body":"{\"key\":\"value\"}","version":"0.0.1"}
Fantastic! Success! There’s the new version number, but are we sure? Let’s check by increasing the version number:
"version": "0.0.2"
Republishing, which we can do via a further script which packages, pushes and publishes in one go:
"package-upload-publish": "npm run package && npm run upload && npm run publish",
npm run package-upload-publish
And then execute the Lambda once more.
npm run invoke
And you should see:
{
"StatusCode": 200,
"ExecutedVersion": "$LATEST"
}
{"statusCode":200,"body":"{\"key\":\"value\"}","version":"0.0.2"}
I’m convinced and I hope you are too! You should now be able to easily publish new code to an existing Lambda.
Continuous Deployment
Continuous Deployment is the process of automatically deploying software without a manual step. For example creating and pushing a zip file of code to an S3 bucket and publishing the code to a Lambda when code is pushed to a repository.GitHub Actions are a great way to create pipelines which implement Continuous Deployment and, as we’re lazy developers, Continuous Deployment is exactly what we want.
GitHub Actions
When we check in some code which is ready to be deployed, GitHub Actions can take that code and deploy it to AWS without manual intervention. There are a number of ways to identify code which is ready to be deployed. I favour using a particular branch. I usually use the git flow process and when code is pushed to the "develop" branch, it is deployed to my Dev environment and when a release is created, merged and pushed to the "main" branch, it is deployed to my production environment. For our Lambda though, we’ll get it to deploy every time we push new code.Let’s create a continuous delivery pipeline!
Make sure you have your Lambda code pushed to a GitHub repository. GitHub Actions are configured via a YML file and placed in the workflows directory in the .github directory, which is in the root of the project. Start off by creating a YML file called publish.yml in the workflows directory. You’ll most likely need to create both the .github directory and the workflows directory.
# .github/workflows/publish.yml
on:
push
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: '18.x'
- run: npm install -g npm
- run: npm ci
- run: npm run package-upload-publish
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: 'eu-west-2'
The on element tells GitHub Actions when to run the job. In this case, when code is pushed to any branch. The job runs inside an Ubuntu container so that we’ve got all the tools we need. The code is checked out and node 18 is installed. Then npm is installed, followed by the Lambda project’s dependencies. Then the package-upload-publish script we created is executed to package and publish the Lambda. We know the script uses the AWS command line which, fortunately, is available to us in this environment. However we need to give it the AWS keys and region, which is done via environment variables.
Go on, add, commit and push the file to GitHub. Then go to the Actions tab of the repository and you’ll see the action fail at the point it tries to push the zip file to S3. This is because we haven’t added the values for the AWS environment variables. You could go ahead and add the key and secret you’ve been using for development, but the chances are this either has full admin permissions or more permissions than the action needs. In either case this is an unnecessary security risk.
AWS Lambda Deployment User
When accessing AWS from a third party system, such as GitHub, it’s good practice to use a user with only the minimum necessary permissions so that if the user’s credentials are compromised the amount of damage which can be done with it is minimised.With Terraform we can create a new user, the necessary keys and some restricted roles. Go back to your platform project and lets add a new user called, email-sender-lambda-deploy:
resource "aws_iam_user" "email_sender_lambda_deploy" {
name = "email-sender-lambda-deploy"
force_destroy = true
}
By setting force_destroy to true, we ensure that the user is recreated with new keys if its permissions are changed. Next we need a user role which will allow the user to access S3:
resource "aws_iam_user_policy" "lambda_s3_deploy_policy" {
name = "EmailSenderLambdaS3DeployPolicy"
user = aws_iam_user.email_sender_lambda_deploy.id
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject"
],
"Resource": "arn:aws:s3:::email-sender-lambda/email-sender-lambda*"
}
]
}
EOF
}
This role permits only the put object and get object actions on the email-sender-lambda bucket; and files beginning with email-sender-lambda only. No other roles, buckets or file names are permitted. Put object is required to push the zip file into the S3 bucket and get object is required to allow the code to be published to the Lambda.
We also need a user role to allow us to publish the code to the Lambda:
resource "aws_iam_user_policy" "lambda_function_deploy_policy" {
name = "EmailSenderLambdaFunctionDeployPolicy"
user = aws_iam_user.email_sender_lambda_deploy.id
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "lambda:UpdateFunctionCode",
"Resource": "arn:aws:lambda:eu-west-2:100241228786:function:email-sender-lambda"
}
]
}
EOF
}
Here the only action is UpdateLambdaFunction and it’s restricted to our Lambda.
We also need some keys to give to GitHub so that it can use the user:
resource "aws_iam_access_key" "email_sender_lambda_deploy" {
user = aws_iam_user.email_sender_lambda_deploy.name
pgp_key = var.pgp_key
}
output "secret" {
value = aws_iam_access_key.email_sender_lambda_deploy.encrypted_secret
}
output "id" {
value = aws_iam_access_key.email_sender_lambda_deploy.id
}
This is quite straightforward, generate some keys for the user and output the key and the secret when the Terraform is applied. The interesting bit is the pgp_key, which I’ve put into a variable:
variable "pgp_key" {}
The PGP key is used to encrypt the secret before it’s printed to the console. There’s more about it in the Terraform documentation:
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_access_key
I opted for the Keybase solution, which requires setting up an account and installing Keybase locally. Once the terraform is applied:
terraform apply
...
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
Outputs:
id = "AKI..."
secret = "wcFMAwY4M..."
the secret can be decrypted with:
terraform output -raw secret | base64 --decode | keybase pgp decrypt
Now that you have a key and a secret, go to the settings tab in the GitHub repository which has the Lambda code in, “Secrets and variables” and then “Actions” and add the key as a secret called AWS_ACCESS_KEY_ID and the secret as a secret called AWS_SECRET_ACCESS_KEY. Then go to the Actions tab, find the failed job and rerun it. You should find it works this time.
Go back to your Lambda code, up the version number in package.json, add commit and push the change. Watch the GitHub Action deploy the Lambda and then execute from the command line, as before, to prove that the new version of Lambda has deployed automatically.
Finally
Here we’ve looked at how to create a basic AWS Lambda with Terraform, build, upload and publish new lambda code independently and automate deployments of new versions of the code.
This is sustainable as long as you’re always publishing new code versions and don’t need to roll back. If you have to rebuild your infrastructure from scratch or roll a version back, then a manual intervention is required to get the right version of code deployed. There are ways to do this, including a hybrid approach using Terraform, but that’s for another time.
The AWS roles we created need to be tightened up to and bound to specific resources. Giving AWS keys to GitHub actions isn’t necessarily the most secure configuration either and you could consider: https://aws.amazon.com/blogs/security/use-iam-roles-to-connect-github-actions-to-actions-in-aws/
I’m sure you get the idea though and can see this is one way of separating the concern of infrastructure from the concern of code.
Thank you to Sam Pennington and Steve Cresswell for inspiration and review.
Comments
Post a Comment