Learn from our experience developing the Deliveroo product.
Take advantage of computing power and empirical data to use Monte Carlo simulation to perform experiment power analysis.
Charles Proxy is an application that sits between your computer and the Internet to record HTTP(S) traffic. Most importantly for me, it has the ability to also modify a server’s HTTP response. I’ve been using Charles over the past few weeks to rewrite responses from our APIs and I’ve been slowly learning what Charles is capable of but also what it isn’t. Needless to say, there’s still a lot to learn, but the defining moment for me was when I was able to recreate a production bug, by replaying the JSON payloads to my local development server causing the same error to be raised.
As data scientists at Deliveroo we evaluate our work via robust experimentation, and we take a frequentist hypothesis testing approach. The standard method places a lot of importance on deciding upfront how large an impact we believe any experiment might have. If the estimated impact is very different to the actual impact, we might waste a lot of time running an experiment where we could have obtained a result sooner. This means that we can’t iterate and innovate as quickly as we would like. Our preferred solution to this problem is sequential experiment designs.