Application Deployment at Deliveroo
Our services are rapidly growing in number and we need a scalable way of managing the mayhem. Over the course of a year, our site was served as a single monolith and is now composed of over 50 services.
Breaking up the monolith and new feature initiatives means that we are adding (or maybe I’m just discovering) two new services a week. Over the past nine months, we have been making strides to take ownership of our application infrastructure.
Making it easy to deploy new apps, with consistency, is key to preventing this devolving into mayhem. With an infrastructure description language, like Hashicorp’s Terraform, we can describe applications programatically. With Infrastructure as Code, maintaining the service platform itself can be done with GitHub PRs.
Deploying new applications
module "avocado" {
source = "../templates/roo_app"
...
# Somewhere in GitHub, assume github.com/deliveroo/%repo_name%
repo_name = "avocados-service"
}
module "avocado-web-service" {
source = "../templates/roo_service_public_web"
...
process_name = "web"
container_port = 3000
}
With a few lines, we can define and instantiate new applications with very little effort.
These templates are quite powerful and hides a lot of our infrastructure decisions.
- Set up the Elastic Container Repository, ECR, to store deployable Docker Images.
- Set up the CircleCI environment for building (and pushing) Docker Images to ECR.
- Create AWS resources, such as S3, IAM and Cloudwatch as needed to configure this application.
- Register the application in Hopper, our release manager.
- Add an ALB to Load Balance HTTP requests for the web service.
- Set up (optional) Autoscaling for the workers.
- Add a basic newrelic dashboard for the service.
- Create an initial ECS placeholder task.
- Create DNS records in route53 and Cloudflare.
The end result is that all of our services can now be developed and deployed with consistency. There is no playbook for “adding new apps”. Teams can create a PR and we all wait for Terraform to do its thing.
Adding more resources
My team maintains a bunch of extra templates for common resources that applications can use. For example, adding a postgresql database.
module "avocado-db" {
source = "../templates/postgresql"
application_name = "${module.avocado.app_name}"
backup_retention_period = "30"
storage_encrypted = true
ebs_size = 100
master_instance_class = {
production = "db.m4.large"
staging = "db.t2.small"
}
replica_instance_class = {
production = "db.m4.large"
staging = "db.t2.small"
}
replica_count = 2
...
}
# This is injected into the application environment at runtime
resource "hopper_variable" "avocado-env-database-url" {
app_name = "${module.avocado.app_name}"
name = "DATABASE_URL"
value = "${module.avocado-db.url}"
write_only = true
}
This can be extended to any Terraformable resource, not just AWS. The template modules can abstract common details and let us pick which parameters we can allow diverge across our organisation.
Beyond Infrastructure as Code
We don’t control everything about our applications in Terraform. While keeping a git log of all changes can be useful, not all changes need to be serialised through my team.
Higher frequency changes such as deploying a commit or updating variables are tasks that are controlled by dashboards and control panels. With an CI/CD workflow, a team may choose to have the system autodeploy whenever the GitHub Merge button is clicked.
We think we have this abstraction at the right level. The particular details about the platform can be concealed in the templates, while application choices are established for each instance.
About Ben Cordero
I’m a platform engineer building tools and provisioning infrastructure for everyone else.
Ping me about running containers in clouds, cluster schedulers and TCP corner cases.
You can find me as @bencord0 on the internet