Stories

The requirements for fast and reliable data pipelines are growing quickly at Deliveroo as the business continues to grow and innovate. We have delivered an event streaming platform which gives strong guarantees on data quality, using Apache Kafka® and Protocol Buffers.

What do you do when your JVM-based application gets OOM killed or exhibits wild memory usage pattern? In this blog post, I will explore some tools and concepts we can use to start debugging this problem.

Take advantage of computing power and empirical data to use Monte Carlo simulation to perform experiment power analysis.

Charles Proxy is an application that sits between your computer and the Internet to record HTTP(S) traffic. Most importantly for me, it has the ability to also modify a server’s HTTP response. I’ve been using Charles over the past few weeks to rewrite responses from our APIs and I’ve been slowly learning what Charles is capable of but also what it isn’t. Needless to say, there’s still a lot to learn, but the defining moment for me was when I was able to recreate a production bug, by replaying the JSON payloads to my local development server causing the same error to be raised.

As data scientists at Deliveroo we evaluate our work via robust experimentation, and we take a frequentist hypothesis testing approach. The standard method places a lot of importance on deciding upfront how large an impact we believe any experiment might have. If the estimated impact is very different to the actual impact, we might waste a lot of time running an experiment where we could have obtained a result sooner. This means that we can’t iterate and innovate as quickly as we would like. Our preferred solution to this problem is sequential experiment designs.

Taking advantage of Golang’s duck typed interfaces and net/http/httptest to test third party dependencies in web applications.

While integrating with a new payment provider, we needed to sync merchant ids via SFTP. We built an AWS Lambda function with Terraform to do this. I’ll walk through our Terraform configuration and the hurdles we overcame around accessing the S3 bucket and retrieving sensitive credentials.

Last week a seemingly simple ActiveRecord query was causing problems on production, by being incredibly slow. Together with my colleague Marty I debugged the issue and as a result we decreased the query time from about 80 seconds to about 100 milliseconds.

One of the best things about working in Deliveroo Engineering is that we have the opportunity to work on a great ever-changing product, which means we are constantly evolving, improving and facing new challenges. One of the main problems we have right now, which is a great problem to have, is the fact that we have grown so much that our monolithic application can’t hold everything in just one place anymore.