Craft

Deliveroo Summary

Overview

Deliveroo is a technology company focused on marketing, selling and delivering restaurant meals to the household or office. Its technology platform optimizes food ordering and delivery by integrating web and mobile consumers with restaurant tablet-based point-of-sale order management terminals and logistics optimization algorithm via its delivery driver smartphone software.

TypePrivate
Founded2012
HQLondon, GBMap
Websitedeliveroo.co.uk
Employee Ratings
3.7
More
Overall CultureC-More

Locations

Deliveroo is headquartered in
London, United Kingdom

Location Map

Latest Updates

Company Growth (employees)

Employees (est.) (Jul 2021)6,692(+2%)
Job Openings273
Website Visits (Apr 2021)10 m
Revenue (FY, 2017)£277.1 M(+116%)
Cybersecurity ratingBMore

Key People/Management at Deliveroo

Deliveroo Office Locations

Deliveroo has offices in London, Los Angeles, Balaclava, Brussel and in 8 other locations

Deliveroo Financials and Metrics

Deliveroo Revenue

Summary Metrics

Founding Date

2012

Deliveroo total Funding

$1.7 b

Deliveroo latest funding size

$180 m

Time since last funding

6 months ago

Deliveroo investors

Index Ventures, Accel Partners, DST Global, T. Rowe Price, General Catalyst, Amazon, Fidelity, 14W, JamJar Investments, Hoxton Ventures, Hummingbird Ventures, Greenoaks Capital, Fidelity Management and Research Company, Felix Capital, Greg Marsh, Bridgepoint, Rancilio Cube, GR Capital, NGP Capital, H14, Entrée Capital, GC Capital, Greyhound Capital, Angel Capital Management, Future Fifty, Greenoaks, Arnaud Bertrand, Durable Capital

Deliveroo's latest funding round in January 2021 was reported to be $180 m. In total, Deliveroo has raised $1.7 b. Deliveroo's latest valuation is reported to be $800 m.

Deliveroo's revenue was reported to be £277.14 m in FY, 2017 which is a 115.6% increase from the previous period.

GBP

Revenue (FY, 2017)£277.14 m
Revenue growth (FY, 2017 - FY, 2016), %115.6%
Gross profit (FY, 2017)£64.31 m
Gross profit margin (FY, 2017), %23.2%
Net income (FY, 2017)(£183.53 m)
EBITDA (FY, 2017) (£165.06 m)
EBIT (FY, 2017) (£177.99 m)
Cash (31-Dec-2018)£184.56 m
EV $6.86 b

GBPFY, 2015FY, 2016FY, 2017FY, 2018
Revenue18.1 m128.6 m277.1 m
Revenue growth, %611%116%
Cost of goods sold19.5 m127.5 m212.8 m476 m
Gross profit(1.4 m)1.1 m64.3 m
GBPSep, 2018
Cost of goods sold227 m
Operating expense total106 m
Pre tax profit(185 m)
show all

GBPFY, 2013FY, 2014FY, 2015FY, 2016FY, 2017FY, 2018
Cash132.3 k16.5 m90.9 m179.8 m380 m184.6 m
Accounts Receivable72.8 k1.5 m10.1 m19.7 m35.2 m
Inventories5.3 m7.2 m
Current Assets132.3 k16.6 m95.7 m197 m409.4 m243.1 m
show all

GBPFY, 2014FY, 2015FY, 2016FY, 2017
Net Income(30.1 m)(129.1 m)(183.5 m)
Depreciation and Amortization705.4 k5.1 m
Cash From Operating Activities(967.7 k)(24 m)(111.1 m)(126 m)
Purchases of PP&E(4.2 m)(14.9 m)
show all

GBP
FY, 2018
Financial Leverage1.7 x

Revenue Breakdown

Deliveroo revenue breakdown by business segment: 99.8% from Provision of services and 0.2% from Other

Deliveroo Operating Metrics

Deliveroo's Active Cities was reported to be 500 in May, 2019.

Apr, 2015Aug, 2016FY, 2016May, 2017Aug, 2017Sep, 2017Nov, 2017Feb, 2018May, 2018Jun, 2018May, 2019
Customers74 k
Drivers45020 k30 k35 k35 k60 k
Restaurants16 k25 k20 k25 k80 k
Active Cities84120140200268500

Deliveroo Acquisitions / Subsidiaries

Company NameDateDeal Size
CultivateJuly 31, 2019
MapleMay 08, 2017
Deliveroo Australia
Deliveroo Belgium SPRL
Deliveroo DMCC
Deliveroo France SAS
Deliveroo Germany GMBH
Deliveroo Hong Kong Limeted
Deliveroo Ireland Limeted
Deliveroo Italy SRL

Deliveroo Hiring Categories

Deliveroo Cybersecurity Score

Cybersecurity ratingPremium dataset

B

86/100

SecurityScorecard logo

Deliveroo Website Traffic

Alexa Website Rank

Total Visits per monthSimilarWeb

Deliveroo Online and Social Media Presence

Twitter followers

68.17 k Twitter followers

6 Months

Deliveroo has 68.17 k Twitter Followers. The number of followers has increased 1.10% month over month and increased 2.21% quarter over quarter.

Deliveroo's Trends

Search term - Deliveroo

Twitter Engagement Stats for @deliveroo

  • 33.43 k

    Tweets

  • 138

    Following

  • 67.8 k

    Followers

  • 275

    Tweets last 30 days

  • 12.2

    Avg. likes per Tweet

  • 67.6%

    Tweets with engagement

Deliveroo Technology StackBuildWith Logo

  • ads

    19 products used

    • Advertising.com
    • AppNexus
    • Atlas
      • Bizo
      • DoubleClick.Net
      • Facebook Custom Audiences
      • Google Floodlight Counter
      • Google Floodlight Sales
      • Google Remarketing
      • Index Exchange
      • LinkedIn Ads
      • Nanigans
      • Rubicon Project
      • SkimLinks
      • Snap Pixel
      • Tapad
      • The Trade Desk
      • Twitter Ads
      • Yahoo Small Business
  • analytics

    25 products used

    • Bizo Insights
    • DoubleClick Floodlight
    • Facebook Pixel
      • Facebook Signal
      • Fastly
      • Global Site Tag
      • Google AdWords Conversion
      • Google Analytics
      • Google Analytics Classic
      • Google Analytics Ecommerce
      • Google Conversion Linker
      • Google Conversion Tracking
      • Google Universal Analytics
      • Hubspot
      • LinkedIn Insights
      • Mixpanel
      • New Relic
      • Rapleaf
      • Salesforce
      • Segment
      • Twitter Analytics
      • Twitter Conversion Tracking
      • Twitter Website Universal Tag
      • Yahoo Dot
      • Yahoo Web Analytics
  • cdn

    6 products used

    • AJAX Libraries API
    • CDN JS
    • Cloudflare
      • Content Delivery Network
      • GStatic Google Static Content
      • Yahoo Image CDN
  • cdns

    2 products used

    • Amazon CloudFront
    • Fastly Verified CDN
Learn more on BuiltWith

Deliveroo Company CultureCultureAndCompensation Logo

  • Overall Culture

    C-

    65/100

  • CEO Rating

    F

    40/100

  • Compensation

    B

    74/100

Learn more on Comparably

Deliveroo News and Updates

Jul 08, 2021
Deliveroo lifts guidance as orders soar in second quarter
_3xOCqDeliveroo this morning raised its guidance for the first year after a quarter in which the company said that orders The post Deliveroo lifts guidance as orders soar in second quarter appeared first on CityAM.
Jul 06, 2021
Deliveroo to create 400 new tech jobs after London stock market float
_3xOCqDeliveroo will create 400 new tech jobs over the next year as the company aims to improve its service after The post Deliveroo to create 400 new tech jobs after London stock market float appeared first on CityAM.
Jun 24, 2021
Deliveroo defeats another workers’ rights challenge in UK courts
Deliveroo has had another win in the UK courts, beating back an appeal by the IWGB union which has sought for years to challenge the gig platform over couriers’ rights but has continued to fail to overturn the company’s classification of riders as self-employed. The latest appeals court ruling is the fourth judgment in the […]
Jun 24, 2021
Deliveroo Jumps After Court Rules its Drivers are Self-Employed
_3xOCqDeliveroo Drivers Are Self-Employed, U.K. Appeal Judges Rule
Jun 24, 2021
UK Court of Appeal rules that Deliveroo riders are self-employed
_3xOCqBritain’s Court of Appeal today ruled that Deliveroo riders were self-employed by dismissing an appeal by the IWGB union against The post UK Court of Appeal rules that Deliveroo riders are self-employed appeared first on CityAM.
May 19, 2021
THE GIG’S UP: Fair Work Commission’s finds Deliveroo driver was unfairly dismissed because he was an employee, not a contractor
_3xOCqThe ruling by Australia’s Fair Work Commission that online food delivery platform Deliveroo unfairly dismissed driver Diego Franco marks a major shift in the Australian “gig” economy. What’s most significant is the commission has ruled Franco was, in fact, an employee of Deliveroo, not an independent contractor – the legal strategy platform companies use to... Read more »

Deliveroo Blogs

Jun 18, 2021
Diversity in Data Science with Ela Osterberger, DS Director
Ela, thanks for taking the time out. It would be great if you could start by introducing yourself and your role here. I am one of two Data Science directors - both of us are women which is rare to find in this discipline! I have been working at Deliveroo since it was a small-ish start-up, 4 years ago. I am no longer doing IC work, so my role is to help the team succeed and ensure the impressive work of the individuals in the group shines. Thanks, let’s get stuck in. Diversity. It is a very topical issue and can mean different things in different spheres. What does it mean for you here at Deliveroo? It means taking a critical look at our teams and acknowledging how representative or otherwise we are of our customers. Given that we want to build a product for everyone, we need to ensure we are an inclusive and diverse workplace that reflects this ambition. Diversity at Deliveroo also means celebrating our individual differences and appreciating people’s uniqueness. More concretely, we want to make sure we attract ethnic, sexual orientation and gender diversity - but also people with diverse skill sets and educational backgrounds. Anything really that allows us to view a problem from different perspectives and come to conclusions that we wouldn’t normally arrive at if we were surrounded by people with the same backgrounds. Diversity of skill sets is interesting because still to this day the definition of a Data Scientist varies from company to company. How do you define data science, and what kind of skill sets could make for an impactful Data Scientist? For me, a Data Scientist is someone who has the technical skills to solve analytical problems to help the company make better decisions. However, in order to have impact, these technical skills need to be paired with a business understanding and an ability to build relationships. A person’s educational background typically plays a big part in shaping their early career. What do you think about people coming from unconventional educational backgrounds, take humanities for example, that want to become a Data Scientist? Technical skills are less important to me in hiring than having critical thinking, analytical skills and a passion for delivering a great product and service. Data Science is a relatively new discipline. Many people on our team didn’t specifically study what they are doing now at university. This makes Data Science teams really interesting - people come from diverse educational backgrounds with their strengths and weaknesses - together we are more than the sum of our parts. Any tips for aspiring Data Scientists from such backgrounds? My recommendation for everyone, independent of their background, is to think about the needs of the people you are building products or services for first. If you do that and you focus on your strengths while improving on your weaker areas, you will succeed. I’ll end with a short but difficult question; in your opinion, what should we be doing to help bridge the gender gap in the tech space? We need to do our best in the workplace - this includes hiring for and retaining diversity - and have regular unconscious bias training. I also believe we should look beyond our workplace to make a difference. I, for example, worked with a school in one of the UK’s most deprived areas to encourage girls to follow a STEM educational path. It’s really important to get girls and young women interested in technology and science and allow them to make confident and well-informed educational decisions.
May 13, 2021
Metrics matter - how we set OKRs at Deliveroo
OKRs stand for Objectives and Key Results - objectives are relatively high level things that we wish to focus on as a company - for example “reducing fraud and abuse”, whilst key results are the way that we measure success against these objectives. OKRs are fairly well established in the literature - see for example this post about some of the history behind them. This blog post details Deliveroo’s specific experience and guidance to how we set OKRs internally - namely the “key results” part, as these are typically the most difficult to get right. At Deliveroo we use two main types of key results - ship goals, and metric goals. An example of a ship goal might be “launch Deliveroo Plus in France”, whilst a metric goal might be “increase Plus profitability by 10%”. Metric goals In setting metric goals, we consider three main factors: Choosing the right metric Ensuring the metric can be measured Setting the right movement of that metric Choosing the right metric This is often the most difficult part. Below I explore some things to consider when trying to choose a metric in the first place: Ability to influence within the team If a metric is set for a team, the team should have the full ability to influence all parts of that metric - otherwise movements in the metric (and therefore hitting their goals) are outside of their control. An example here is around negative order experiences (e.g. when a customer orders food but it never turns up) - at Deliveroo we have a broadly defined metric for these which we use to measure improvements to customer service levels. These negative experiences can be caused by multiple issues - for example due to our delivery network algorithms, or due to customer fraud. If the delivery network algorithm team is goaled on reducing these experiences but it turns out that an increase in consumer fraud causes the metric to rise, then the team should either have their scope widened, or the goal should be spread across multiple teams - one responsible for the delivery network element, one responsible for the consumer fraud element, etc. Ability to influence within the timeframe set Some metrics might measure exactly what we want, however take a long time to influence. An example here could be something that relates to the signing of legal contracts (e.g. partnerships). If an engineering team is goaled on increasing our product offering to improve likelihood of signing these partnerships, but contracts are typically only revisited on a yearly basis, then a Q3 goal to increase this number may not make sense, as contract negotiations will likely take too long. In this example, setting a longer term goal might make more sense, or alternatively using other proxy metrics for Q3 goals, with the overall metric measured but not goaled against in the short term. Ability to influence - and realistically take credit for If OKRs are set for a specific team, but the overall metric is ultimately influenced by factors outside of that team (e.g. commercial or legal) - should this metric be used? The above example again works here - signing legal contracts obviously has a large reliance on teams other than engineering, meaning that meeting the goal may be entirely down to another team, or alternatively not meeting the goal also may be no fault of the team in question. Another example here is around overall profitability. One metric may be “increase profitability from X to Y” - whilst this makes sense for the whole company to be goaled on, it is not the responsibility of one single team. Instead, this metric could be changed to be e.g. “increase Plus profitability from X to Y” - if measured experimentally, we can calculate the cumulative impact of one team on this metric, and therefore realistically the team can take credit. Cannibalisation is important to take into account here - see later section. Gameability - don’t move the goalposts Don’t make goals that are easily gameable by the team that is responsible for them - if you set a goal of “increase metric X by 10%” and then simply change the metric definition at the end of the timeframe, any team can easily achieve the goal. Instead, doing some work upfront to set metric definitions will help to keep teams accountable. Metrics can change definition if new information arises and is relevant, so long as this is recorded and goals are updated accordingly if needed. Give metrics meaning Related to gameability, metrics should be defined in such a way that we actually care about their outcome. An example is the difference between “number of users” and “number of active users’’ - the former could be gamed by promoting low-intent users to sign up with no concern for whether or not they actually place any orders. “Number of active users” however includes the “…and placing orders” assumption on the metric, so in this case better measures the actual business outcomes that we want (new users lead to more orders), whilst still being relevant to e.g. the growth team. Decomposability If we see a movement in a metric, we ideally want to be able to tell why the move happened - which means we will likely need to decompose it into its constituent parts. An example is order growth - if we see order growth changing, we can break this down into geographic regions, types of users, new vs old customers, order types. This of course all relies on the relevant data being logged - the metadata used to decompose the metric needs to be readily available, such as the geographic region that an order was placed in. Consider cannibalisation In the above example of increasing profitability of one part of the business, the downside here is that if one team pursues this goal, it could have negative consequences elsewhere in the business - i.e. one product cannibalising another. To counter this at Deliveroo, our experimentation platform has a suite of “do no harm” metrics which measure the impact of any experiments on one product across other products or areas of the business. This way, we can catch any possible degradations early, and make decisions on whether prioritising hitting one goal over another makes sense for the wider business. Internal-facing products and proxy metrics At Deliveroo we have a number of internal facing teams - for example our production engineering team, and our experimentation platform team who work on facilitating data scientists’ ability to run experiments. The ultimate goal of these teams is to level up the ability of the rest of the organisation, ensuring stability and consistency. As a result, their impact on any outward facing metrics such as profitability or growth is likely to happen only via second order effects. Typically for these teams we will use a number of proxy metrics instead for goaling purposes. Using the experimentation platform team as an example, we have historically goaled this team on the number of active users of the platform. The logic here is that by increasing the number of active users of the platform, we will free up more time for data scientists to work on more impactful projects or allow them to increase their scope, ultimately saving the company money and ensuring we make more consistent and accurate business decisions. Ensuring the metric can be measured To measure success against metrics, we need to be able to calculate the metric. Does the data exist to calculate the metric Firstly - you cannot measure a metric if the data doesn’t exist. It is likely you already record all the relevant data for e.g. orders, or users, however more complex metrics there may be upfront work to be done to either add new data logging, or data pipelining work. An example here is the “negative order experience” metric - at Deliveroo we use a version of this which aggregates a number of possible factors, combining it into one headline metric that we can use as an OKR. Before we were able to goal on this metric, the metric had first be well defined, and pipelines had to be created so that we could report on our progress against it. Consider the tradeoff of effort required to measure a metric For some metrics, we cannot get the data for them even if we tried with reasonable effort. In these scenarios, we either need to calculate some sort of proxy metric, or change the metric entirely. An example here could be reducing false positives due to an unsupervised algorithm - if we cannot measure false positives without manually labelling data, which would require a labour-intensive effort, then perhaps this is not the right metric. Setting the right movement of a metric Once you’ve defined a metric, you need to define what the goal of the metric is - e.g. increasing adoption of a product from X% to Y% of our active user base. Goals have to be within the realms of possibility, but also stretch the team to achieve them. At Deliveroo we typically do not expect the company to hit all of its internal product team goals, as this would likely imply that targets were too easy to hit. To combat this, our leadership team reviews past OKRs each quarter to assess whether our goal-setting process was too aggressive or not, and uses this to feed back into the next round. Note that externally-facing goals or commitments to the Board and investors are very different - this blog post solely focusses on internal goals. Factors to consider when aiming to choose the “X” that a metric needs to hit are: Do we have other comparisons that we can look at? An example here is that if we saw a previous year’s work achieving a 10% reduction in some metric, do we think that another 10% reduction is achievable (given likely diminished marginal returns)? What about if past analyses or experiments have given us a figure that we believe is achievable? Is the movement realisable? We shouldn’t set movements that are out of the realms of possibility. For example, increasing the share of wallet of payment method X to 50%, even though less than 50% of our users are capable of using this payment method. At Deliveroo we usually perform this sort of due diligence by getting all OKRs reviewed by multiple other people, who can additionally sense-check whether the metrics are realistic or not. Can market research or external data help? An example here is where third parties might publish market data on fraud solution false positive rates - if this data is available, we can use it to benchmark our own goal setting in this area. Does the goal tie in to some other wider company objective? An example here might be related to order growth - if we have a company-level goal to achieve growth of X%, then our bottoms-up aggregation of all goals should equal that X%. Of course the “realms of possibility” argument is relevant here, but it is useless setting a company-wide goal of growth of X when the sum of each individual team’s goals is less than that X. Ship goals Broadly speaking, ship goals are less preferable to metric goals - before deciding to choose a ship goal, ideally the above “how to choose a good metric” approach should have been exhausted. Ship goals reduce the autonomy of teams - as the goal is essentially “do this thing”. But sometimes you really do just need to do this thing. Below are some examples of when ship goals may be necessary. 0 to 1 products For some features or products, we simply cannot know the impact before we launch them, as they are in entirely unknown territory. Here, if we have solid business reasons for the product, then a ship goal can be relevant. An example here could be a new feature where we might have market research and competitor intelligence that tells us that this is the correct feature to launch, however we do not know in advance whether we can really expect 5%, 10%, 20% of users to use the feature - so setting a “usage” goal would be a complete stab in the dark anyway. External factors - contractual agreements, or legal obligations Sometimes we simply have to do something - for example be GDPR compliant. Other times, there is a big contractual reason why we should do something - for example if we believe that launching a large new partner in the UK requires a certain product feature, then we could argue that implementation of that product feature should be a ship goal. When a metric just doesn’t make sense or work for the timeframe Sometimes setting up proper metrics might take a considerable amount of time, which doesn’t help if it’s already March and you’re setting Q2 goals. In this case, ship goals may suffice for the time being. Similarly, some goals may not really make sense to have proper metrics for, or they may be impossible to measure via a metric. An example here is Deliveroo’s recent IPO - for this, we had a set list of things to achieve to be “IPO ready”, and therefore set ship goals against milestones we needed to achieve to get there. Tracking goals once they’ve been set Once goals have been set, we want to track our progress against them. Dashboards vs static analyses For some goals, setting up pipelines and continually monitoring progress against goals makes sense - for example if our goal is to get take-up rates of some product to 50% of our active user base, then setting up a dashboard that tracks this usage can be a helpful motivational tool for the whole team to see how their work is influencing the metric. However, for others, offline static analyses will suffice. Some metrics may rely on survey data rather than some constant data feed, in which case given that surveys can only be ran at distinct time intervals, a fixed cadence of reporting results makes more sense. Measuring cumulative experimental impact If your goal is something like “increase profitability from a certain product from X to Y”, likely individual product feature launches will be run as an experiment, to see their impact on profitability. However, at the end of the time period, we will need to calculate what the cumulative impact is of all the product changes. There are two common approaches to this. The first approach is to have a fixed holdout group which remains in place for the entire time period (e.g. a quarter, half, or year). All changes to the product can be calculated relative to this holdout group, to see the cumulative impact of product launches. The pro of this approach is that interaction effects are taken into account - if two separate product feature launches are not independent, the combined impact of them launched together is likely not equal to the sum of their individual experiment results. An overall holdout takes this into account. The downside, however, is that a set of users will have a different app experience for a long period of time. If our population is small, this holdout might need to be high in order to get statistical significance - e.g. 10% of all users. Alternatively, a “sum of experiments” approach can be taken. This is where each individual product feature launch is measured in isolation, and the sum of their impact is assumed to be the cumulative impact had. The main downside here is the interaction effects problem explained above - likely the total impact will be less (or possibly even higher) than the individual sums. However, the benefit is that holdouts only exist during experimentation, and features can be launched more widely. Accuracy of results is traded for a better, more consistent user experience. Don’t go overboard Setting goals takes time - particularly data scientists’ time. There is always more work that can be done on a set of goals - could we improve our forecast model? Could we define our metric better? Could we build some beautiful dashboards to track our progress? The answer is always yes, but we don’t have infinite data scientists. OKRs, at a high level, are intended to ensure that we are on the right course and making progress. We will likely continue to iterate on our OKR setting process as we mature as an org, so no doubt our earlier attempts at OKRs will not be perfect - however the above guide should hopefully ensure that they are at least “directionally correct”. A motivating example of OKRs in action at Deliveroo Greg Beech (former principal engineer at Deliveroo) tweeted about his previous positive experience with OKRs whilst working on the Care team: https://twitter.com/gregbeech/status/1236707894588039170
Sept 07, 2020
Increase the Reliability of a Go Codebase with Object Constructors
Coming from a heavy production experience with languages such as C# and TypeScript, I must admit that my journey with Go has been a bumpy ride so far, but it’s for sure a positive one overall. Go certainly shines in some parts such as its runtime efficiency, built-in tooling support and its simplicity which allows you to get up to speed with it so quickly! However, there are some areas where it limits your ability to express and model your software in code in a robust way, especially in a codebase where you get to work on as a team such as lack of sum types and generics support (luckily, generics support seems to be on its way). One of these limitations I have come across is not having any built-in constructor support. I stumbled upon this limitation while learning Go, but I was mostly being open-minded. After seeing a few of the problems which lack of constructors caused, I can see the value of constructors to be adopted in most Go codebases. In this post, I will share a solution that worked for our team, and the advantages of adopting such solution. I must give credit to John Arundel. Thanks to the discussion we have had on Twitter, I am able to express a solution to this problem here which is based on what John made me aware of first. Now, when I say constructors in the title of the post here, I must confess that it’s a bit of a overstatement because I don’t see a way of having pure object constructors like we have with C# or Java in Go without changes in the language itself. However, we can work around the lack of constructors in Go by leveraging some other aspects of the language such as package scoping and interfaces and essentially adopt the factory method pattern. Let’s first touch on these two aspects of Go, and see how we can use them to our advantage to make our code more robust and protect against unexpected consumptions in the feature. Package Scoping Go doesn’t have access modifiers such as private, internal or public per se. However, you can influence whether a type should be internal to a package or should be exposed through naming in Go respectively by “unexporting” or “exporting” them. When your type is named by starting with a lowercase letter, it will only be available within the package itself. This rule also applies to the functions, and members of the types such as fields and methods. For example, the following code sample does not compile: singers/jazzsinger.go file: package singers type jazzSinger struct { } func (jazzSinger) Sing() string { return "Des yeux qui font baisser les miens" } main.go file: package main import ( "fmt" "github.com/tugberkugurlu/go-package-scope/singers" ) func main() { s := singers.jazzSinger{} fmt.Println(s.Sing()) } If we were to run this code, we would get the following error: ➜ go-package-scope go run main.go # command-line-arguments ./main.go:9:7: cannot refer to unexported name singers.jazzSinger ./main.go:9:7: undefined: singers.jazzSinger This sort of demonstrates how package scoping works in Go. You can learn more about packages in Go from Uday’s great article on this topic but this should be enough for us to get going for our example. Interfaces Let’s now look at interfaces in Go, which act very similar to what you would expect them to be. However, the way you “implement” (in Go “satisfy”) interfaces is very different to how you would do in C#, Java or TypeScript. The main difference is that you don’t explicitly declare that a struct implements an interface in Go. A struct is considered to be satisfying an interface by the compiler as long as it provides all the methods within it with matching signatures, or in the Go terminology, as long as the “method set” of the type can satisfy the interface requirements. Let’s look at the following example: package main import ( "fmt" ) type Singer interface { Sing() string } type jazzSinger struct { } func (jazzSinger) Sing() string { return "Des yeux qui font baisser les miens" } func main() { s := jazzSinger{} singToConsole(s) } func singToConsole(singer Singer) { fmt.Println(singer.Sing()) } This code happily executes. Notice that jazzSinger struct doesn’t say anything about implementing the Singer interface. This is what’s called structural typing, as opposed to nominal typing like one of C#’s characteristics (see the diff here). We can understand from this that Go has a way to abstract away the implementation and this fact will hugely help us when it comes to work around the lack of constructors in Go. Bringing All These Together These two aspects of the language can be brought together to allow us to hide the implementation from the contract by only exposing what we need. The challenge here is to be able to provide a way to construct the implementation. Fortunately, there is a workaround for this in Go: we can define an exported function within the package, which has access to the internal implementation, but also exposes it through the interface, as shown in the example below: package singers type Singer interface { Sing() string } type jazzSinger struct { } func (jazzSinger) Sing() string { return "Des yeux qui font baisser les miens" } func NewJazzSinger() Singer { return jazzSinger{} } NewJazzSinger function here can be accessed by the package consumer but jazzSinger struct is still hidden. package main import ( "fmt" "github.com/tugberkugurlu/go-package-scope/singers" ) func main() { s := singers.NewJazzSinger() singToConsole(s) } func singToConsole(singer singers.Singer) { fmt.Println(singer.Sing()) } Why is this good and how does this make our code more reliable? Let’s go over the main advantages of this technique, and how they make our code more reliable. Changes in the struct’s fields would make our code fail at compile time, rather than runtime Unlike other languages (such as TypeScript), Go doesn’t have a way to enforce assigning fields directly (omitted fields default to the zero value, which may not always be what you want) - so the compiler would not help us here - we would need to track all updates to the struct’s fields manually, which is tedious and error prone (specially in large codebases). Best case scenario, the code would be well tested and the unit tests would break. Worst case scenario, the code would blow up at Runtime, which would require a rollback of this release. To make matters worse, your application could be happily working without any crashes, but the its behaviour could be wrong due to the way the implementation might work. This one is the hardest and potentially most harmful bug to catch as it could have a larger impact on your efforts and the outcome you wanted to achieve in the first place. Let’s imagine our jazzSinger would start getting lyrics from an external resource. You would structure this by providing an interface and allowing jazzSinger to call into that, which would look like the following snippet/example: package singers // Lyrics type LyricsProvider interface { GetRandom() string } type jazzLyricsProvider struct { } func (jazzLyricsProvider) GetRandom() string { return "Des yeux qui font baisser les miens" } func NewJazzLyricsProvider() LyricsProvider { return jazzLyricsProvider{} } // Singer type Singer interface { Sing() string } type jazzSinger struct { lyrics LyricsProvider } func (js jazzSinger) Sing() string { return js.lyrics.GetRandom() } func NewJazzSinger(lyrics LyricsProvider) Singer { return jazzSinger{ lyrics: lyrics, } } If we were to build our application directly without modifying the main package (which is the consumer of the singers package), we would see the following error: ➜ go-package-scope go build main.go # command-line-arguments ./main.go:9:28: not enough arguments in call to singers.NewJazzSinger have () want (singers.LyricsProvider) We wouldn’t get this level of feedback if we were to initialize the struct directly. What we would get instead is a failure: ➜ go-package-scope go run main.go panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1091512] goroutine 1 [running]: github.com/tugberkugurlu/go-package-scope/singers.JazzSinger.Sing(0x0, 0x0, 0x1010095, 0xc00000e1e0) /Users/tugberkugurlu/go/src/github.com/tugberkugurlu/go-package-scope/singers/jazzsinger.go:31 +0x22 main.singToConsole(0x10d7520, 0xc00000e1e0) /Users/tugberkugurlu/go/src/github.com/tugberkugurlu/go-package-scope/main.go:14 +0x35 main.main() /Users/tugberkugurlu/go/src/github.com/tugberkugurlu/go-package-scope/main.go:10 +0x57 exit status 2 Allows you to provide parameter validation as early as possible Enforcing parameter validation also allows the consumer to explicitly act on potential errors. I must be honest here, we mostly need this level of validation due to Go’s inability to enforce nil pointer check before accessing the value, which is provided in languages like TypeScript. My post on TypeScript demonstrates what I mean by this. However, there are genuinely other cases where a compiler cannot guard against your own business logic. In our example above, we can still make our code compile successfully with the constructor implementation but get a runtime error: package main import ( "fmt" "github.com/tugberkugurlu/go-package-scope/singers" ) func main() { s := singers.NewJazzSinger(nil) singToConsole(s) } func singToConsole(singer singers.Singer) { fmt.Println(singer.Sing()) } When we run we see the error below - even though the code compiled successfully: ➜ go-package-scope go build main.go ➜ go-package-scope ./main panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1091512] goroutine 1 [running]: github.com/tugberkugurlu/go-package-scope/singers.jazzSinger.Sing(0x0, 0x0, 0x1010095, 0xc00008e030) /Users/tugberkugurlu/go/src/github.com/tugberkugurlu/go-package-scope/singers/jazzsinger.go:31 +0x22 main.singToConsole(0x10d75a0, 0xc00008e030) /Users/tugberkugurlu/go/src/github.com/tugberkugurlu/go-package-scope/main.go:14 +0x35 main.main() /Users/tugberkugurlu/go/src/github.com/tugberkugurlu/go-package-scope/main.go:10 +0x5c There isn’t a solution available in Go as far as I am aware which would allow us to fail for these cases during compilation. However, thanks to the dedicated constructor for this object, we can explicitly signal potential construction errors by returning multiple values from the function call: func NewJazzSinger(lyrics LyricsProvider) (Singer, error) { if lyrics == nil { return nil, errors.New("lyrics cannot be nil") } return jazzSinger{ lyrics: lyrics, }, nil } At the time of consumption, it becomes very explicit to deal with returned result: s, err := singers.NewJazzSinger(nil) if err != nil { log.Fatal(err) } // ... Allows you to control the flow of your implementation The code below is a simplified and intended-use scenario of an interesting bug we had in production a while ago: package main import ( "fmt" ) type JazzSinger struct { count int } func (j *JazzSinger) Sing() string { j.count++ return "Des yeux qui font baisser les miens" } func (j *JazzSinger) Count() int { return j.count } func main() { s := &JazzSinger{} singToConsole(s) fmt.Println(s.Count()) singToConsole(s) fmt.Println(s.Count()) } func singToConsole(singer *JazzSinger) { fmt.Println(singer.Sing()) } This code works as expected: the singer sings, and the count is incremented. All great! Des yeux qui font baisser les miens 1 Des yeux qui font baisser les miens 2 This works because our method signature on the JazzSinger struct accepts a pointer to an instance of JazzSinger which means that the count will be incremented as expected even if the type is passed around, and that’s what’s happening with the above scenario. However, can we guess what will happen if we change our usage as below: func main() { s := JazzSinger{} singToConsole(s) fmt.Println(s.Count()) singToConsole(s) fmt.Println(s.Count()) } func singToConsole(singer JazzSinger) { fmt.Println(singer.Sing()) } My first guess was that compiler would fail here, and this is a perfectly reasonable assumption to make since we are not passing a pointer to Sing method call. If you made the same assumption as I did, you would be wrong. This compiles perfectly but it won’t work as expected: Des yeux qui font baisser les miens 0 Des yeux qui font baisser les miens 0 The worst part is that this would actually work if we were to get rid of the singToConsole function and embed its implementation: func main() { s := JazzSinger{} s.Sing() fmt.Println(s.Count()) s.Sing() fmt.Println(s.Count()) } Des yeux qui font baisser les miens 1 Des yeux qui font baisser les miens 2 This is the exact reason why your tests will pass even if they have the wrong usage! package main import ( "github.com/deliveroo/assert-go" "testing" ) func TestJazzSinger(t *testing.T) { t.Run("count increments as expected", func(t *testing.T) { singer := JazzSinger{} singer.Sing() singer.Sing() assert.Equal(t, singer.Count(), 2) }) } ➜ jazz-singer git:(master) ✗ go test -v === RUN TestJazzSinger === RUN TestJazzSinger/count_increments_as_expected --- PASS: TestJazzSinger (0.00s) --- PASS: TestJazzSinger/count_increments_as_expected (0.00s) PASS ok github.com/tugberkugurlu/algos-go/jazz-singer 0.549s After a bit more digging, it turned out that this is actually the intended behavior of Go, and it’s even documented in its spec: A method call x.m() is valid if the method set of (the type of) x contains m and the argument list can be assigned to the parameter list of m. If x is addressable and &x’s method set contains m, x.m() is shorthand for (&x).m(). I am still unsure why this could be useful, but it is what it is, and it’s so easy to make the same mistake since you can ensure how the consumer will flow the type as the creator of the type if it can be constructed freely. In fact, the decision of how the type should be flowed should be the decision of the owner (i.e. its package) of the type, not the consumer, and I have never found a case where I needed to flow a type both as a pointer or value. Languages like C# puts the burden of this choice onto the author of the type by forcing them to choose between a class and struct. In Go, you can make this safer through the use of the constructor pattern as well, by ensuring that your struct is not allowed to be constructed directly and you controlling how the initialized value should be flowed. package singers type Singer interface { Sing() string Count() int } type jazzSinger struct { count int } func (j *jazzSinger) Sing() string { j.count++ return "Des yeux qui font baisser les miens" } func (j *jazzSinger) Count() string { return j.count } func NewJazzSinger() Singer { return &jazzSinger{} } The consumer of this type needs to construct it through NewJazzSinger function here, which is making the decision to flow the type as a pointer because it needs to be able to mutate its own state as it’s being used. package main import ( "fmt" "github.com/tugberkugurlu/go-package-scope/singers" ) func main() { s := singers.NewJazzSinger() singToConsole(s) fmt.Println(s.Count()) singToConsole(s) fmt.Println(s.Count()) } func singToConsole(singer singers.Singer) { fmt.Println(singer.Sing()) } Drawbacks of using Interfaces Returning an Interface from your constructor function allows you to encapsulate the implementation details fully, and disallow uncontrolled changes in your state. However, this comes with its own trade-off that since as the consumer of a specific package/library, you need to do further digging to understand the intent, and whether the underlying type is being returned as a value or as a pointer. This can be significant for some use cases where, for example, you want to reduce the pressure from the garbage collector by reducing the pointer usage. It’s actually possible to eliminate to use of interface here, and just returning the raw struct. Howevever, that also comes with its own drawback: If you return the exported struct, the consumer of your package can initialize a new struct without using the constructor, e.g. JazzSinger{}. Allowing the consumers to bypass constructor usage will come with its own problems as we have seen in this post. If you return an unexported struct, you will make it hard for the consumers of your package to accumulate the results from the constructor. This Go Playground example shows where this might be critical. This can be worked around by owning the interface that matches at least the partial signature of the unexported struct at the consumption level. This Go Playground example shows how to achieve that. In any case, it’s best to be informed about this drawback, and go with the right option which will fit into your use case. Conclusion Modelling your domain is hard and it’s even harder if you have rich models which hold a mutable state along with explicit behaviours. Go programming language may may not give you all the tools to directly model your domain in a rich way as some other programming languages provide. However, it’s still possible to make it work for some cases by adopting some usage principles. Constructor pattern is one of them, and it has been one of the most useful ones for me since I can confidently encapsulate the initialisation logic of my model by enforcing state validity within a package scope.
Jun 16, 2020
Using AWS EC2 and ECS to host hundreds of services
One of my goals of moving internally to the Production Engineering team was to help demystify the concepts that are commonplace within our Platform teams. My first internal blog post to do this was to share how we use EC2 (AWS Elastic Compute Cloud) and ECS (AWS Elastic Container Service) to host the hundreds of services that our Software Engineers build and improve every day. What is an EC2 host? “Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides secure, resizable compute capacity in the cloud. It is designed to make web-scale cloud computing easier for developers. Amazon EC2’s simple web service interface allows you to obtain and configure capacity with minimal friction. It provides you with complete control of your computing resources and lets you run on Amazon’s proven computing environment.” Amazon’s description of EC2 I would say this; an EC2 host is a server. More simply, it is a computer. It is (in most cases) not an actual physical server in a rack, and Amazon abstracts that detail away from us, but I find I get my head around the concept easier by thinking of them as physical machines anyway. The machines we generally use have 16 vCPUs and 64 GiB of Memory (RAM). It comes preinstalled with the software required to make it a usable computer; like an operating system (you can just assume Linux for now - others are available though), so it can be booted up and can run processes - more on that later… What do we use EC2 hosts for? A few different uses, but the most common use is in an ECS Cluster, a grouping of EC2 machines used as a home for ECS Tasks - these are the Dockerized containers of our applications that are running with a command specified by the engineer in a config file. ECS? What’s that, and how is it related to EC2? ECS is AWS’s Elastic Container Service. It is an orchestration service that makes sure the correct number of each service is running in the environment. What it is actually running are the Docker containers that our continuous deployment provider built when a PR was last merged to master. When an engineer tells Hopper, our application release manager, to first scale one of their app’s services up from 0 to 1 task, Hopper makes a call to ECS to ask it to make sure that at all times there is one healthy instance of their Docker container running on any one of the EC2 hosts. This is the desired number of tasks - if the number of running tasks is less than this, ECS will start more containers to reach the desired number, if there are more than desired, ECS will safely terminate running containers to reach the desired number. Where does ECS start running this one container I’ve asked for? This takes us back to our cluster of EC2 machines. ECS will find an EC2 machine in the cluster that has enough spare capacity on it to hold and run your task (i.e. it has enough spare reserved CPU and memory - which is specified in the config file). There are some other rules in place regarding which Availability Zone your task is running in (we don’t want all your eggs in one basket), but for the most part, we leave it to ECS to decide. What happens if the cluster is full? We are constantly monitoring the ECS cluster, and autoscale EC2 instances based on how much spare capacity there is. If there’s not enough spare capacity to immediately run another 40 large docker containers, we bump up the desired count of EC2 instances in the cluster, and EC2 spins up new machines (the number of machines we start up depends on how much below approximately 40 large container capacity we are). New EC2 instances can take a few minutes before they’re ready to be used by ECS, so we need to have a buffer to deal with unexpected spikes in demand. How do we change or upgrade the machine image used? Circling back to the software that is preinstalled on these EC2 servers. When booted up, an Amazon Machine Image (AMI) is used, which has some basic tools installed on it to make the machine usable. Amazon provides a base AMI which we have built upon, using Packer and Ansible, to create our own Amazon Linux-derived machine image. This, and some initialization scripts, give all our running ECS tasks access to things that all Deliveroo’s services will need, such as software (like the Datadog agent which sends metrics to Datadog, and AWS Inspector, AWS’s automated security assessment service), roles, security policies, and environment variables that we need to apply to the containers. The process of rolling out a new machine image when an update is available, or when we make changes to our custom machine image, is not as straightforward as I’m used to as an application developer (I have a new-found appreciation for release management software). Only newly created EC2 machines will be built using this new image, and so the process of rolling out is one of the following on each of our AWS environments (sandbox, staging, production): Disabling some of the cluster autoscaling rules, as we only want EC2 instances using the old image being terminated when the cluster gets too big. Slowly scaling up the number of desired EC2 instances using the new AMI and observing whether the change looks to be applied correctly, or if there are issues occurring, or alerts triggering. Slowly reducing the desired number of old EC2 instances - terminated instances will send a message to ECS to safely end all the tasks being run on the instance. Without doing this, very few new services will actually be placed on the new EC2 instances to test the changes in an incremental fashion. Once the cluster is fully on the new EC2 instances, adjust and re-enable the autoscaling rules so that the old AMI is no longer used, and we continue to autoscale instances using only the new AMI. Repeat until fully rolled out, on all environments. We use an A/B system to deploy - the old AMI and configurations remained as the B option, while any changes are only applied to the A track. On the first attempt we noticed some issues with the new machine image after starting a relatively small number of EC2 machines; it was as simple as scaling B back up to an appropriate level, and A down to 0. As disappointing as it was to fail the first time, I learnt so much more about the process by having to undo it halfway through than I would have done if it had gone perfectly.
Jun 16, 2020
Using AWS EC2 and ECS to host hundreds of services
One of my goals of moving internally to the Production Engineering team was to help demystify the concepts that are commonplace within our Platform teams. My first internal blog post to do this was to share how we use EC2 (AWS Elastic Compute Cloud) and ECS (AWS Elastic Container Service) to host the hundreds of services that our Software Engineers build and improve every day. What is an EC2 host? “Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides secure, resizable compute capacity in the cloud. It is designed to make web-scale cloud computing easier for developers. Amazon EC2’s simple web service interface allows you to obtain and configure capacity with minimal friction. It provides you with complete control of your computing resources and lets you run on Amazon’s proven computing environment.” Amazon’s description of EC2 I would say this; an EC2 host is a server. More simply, it is a computer. It is (in most cases) not an actual physical server in a rack, and Amazon abstracts that detail away from us, but I find I get my head around the concept easier by thinking of them as physical machines anyway. The machines we generally use have 16 vCPUs and 64 GiB of Memory (RAM). It comes preinstalled with the software required to make it a usable computer; like an operating system (you can just assume Linux for now - others are available though), so it can be booted up and can run processes - more on that later… What do we use EC2 hosts for? A few different uses, but the most common use is in an ECS Cluster, a grouping of EC2 machines used as a home for ECS Tasks - these are the Dockerized containers of our applications that are running with a command specified by the engineer in a config file. ECS? What’s that, and how is it related to EC2? ECS is AWS’s Elastic Container Service. It is an orchestration service that makes sure the correct number of each service is running in the environment. What it is actually running are the Docker containers that our continuous deployment provider built when a PR was last merged to master. When an engineer tells Hopper, our application release manager, to first scale one of their app’s services up from 0 to 1 task, Hopper makes a call to ECS to ask it to make sure that at all times there is one healthy instance of their Docker container running on any one of the EC2 hosts. This is the desired number of tasks - if the number of running tasks is less than this, ECS will start more containers to reach the desired number, if there are more than desired, ECS will safely terminate running containers to reach the desired number. Where does ECS start running this one container I’ve asked for? This takes us back to our cluster of EC2 machines. ECS will find an EC2 machine in the cluster that has enough spare capacity on it to hold and run your task (i.e. it has enough spare reserved CPU and memory - which is specified in the config file). There are some other rules in place regarding which Availability Zone your task is running in (we don’t want all your eggs in one basket), but for the most part, we leave it to ECS to decide. What happens if the cluster is full? We are constantly monitoring the ECS cluster, and autoscale EC2 instances based on how much spare capacity there is. If there’s not enough spare capacity to immediately run another 40 large docker containers, we bump up the desired count of EC2 instances in the cluster, and EC2 spins up new machines (the number of machines we start up depends on how much below approximately 40 large container capacity we are). New EC2 instances can take a few minutes before they’re ready to be used by ECS, so we need to have a buffer to deal with unexpected spikes in demand. How do we change or upgrade the machine image used? Circling back to the software that is preinstalled on these EC2 servers. When booted up, an Amazon Machine Image (AMI) is used, which has some basic tools installed on it to make the machine usable. Amazon provides a base AMI which we have built upon, using Packer and Ansible, to create our own Amazon Linux-derived machine image. This, and some initialization scripts, give all our running ECS tasks access to things that all Deliveroo’s services will need, such as software (like the Datadog agent which sends metrics to Datadog, and AWS Inspector, AWS’s automated security assessment service), roles, security policies, and environment variables that we need to apply to the containers. The process of rolling out a new machine image when an update is available, or when we make changes to our custom machine image, is not as straightforward as I’m used to as an application developer (I have a new-found appreciation for release management software). Only newly created EC2 machines will be built using this new image, and so the process of rolling out is one of the following on each of our AWS environments (sandbox, staging, production): Disabling some of the cluster autoscaling rules, as we only want EC2 instances using the old image being terminated when the cluster gets too big. Slowly scaling up the number of desired EC2 instances using the new AMI and observing whether the change looks to be applied correctly, or if there are issues occurring, or alerts triggering. Slowly reducing the desired number of old EC2 instances - terminated instances will send a message to ECS to safely end all the tasks being run on the instance. Without doing this, very few new services will actually be placed on the new EC2 instances to test the changes in an incremental fashion. Once the cluster is fully on the new EC2 instances, adjust and re-enable the autoscaling rules so that the old AMI is no longer used, and we continue to autoscale instances using only the new AMI. Repeat until fully rolled out, on all environments. We use an A/B system to deploy - the old AMI and configurations remained as the B option, while any changes are only applied to the A track. On the first attempt we noticed some issues with the new machine image after starting a relatively small number of EC2 machines; it was as simple as scaling B back up to an appropriate level, and A down to 0. As disappointing as it was to fail the first time, I learnt so much more about the process by having to undo it halfway through than I would have done if it had gone perfectly.
Jan 02, 2020
CloudFormation To Terraform
For those starting with either Terraform or CloudFormation this guide is a good way to understand the differences between the two. I found myself a little bit stuck because I needed to find/create code (in this case) that would help me in Benchmarking our compliance status in AWS. I found a solution in CloudFormation, so I wondered if there was some sort of translator tool (there wasn’t), and if not, how and where would I start translating this code? Would it be worth me building it from scratch in Terraform? How to convert CloudFormation (CF) to Terraform (TF): CIS Foundations Quickstart First lets state the differences and how each syntax is built: Terraform Terraform language uses HCL (Hashicorp Configuration Language). Terraform code is built around 2 key syntax constructs: Arguments: Arguments assigns a value to a particular name: image_id = "blabla" Blocks: A block is a container for other content resource "aws_instance" "example" { ami = "abc123" network_interface { # ... } } Terraform consists of modules, which is really up to the builder on what it does. Each module has blocks and along with the configuration, it tells terraform how/when to use/build that module. Configuration files in Terraform are written in JSON. CloudFormation CloudFormation is all about templates. If you want to build a configuration for an application or service in AWS, in CF, you would create a template, these templates will quickly provision the services or applications (called stacks) needed. The most important top-level properties of a CloudFormation template are: Resources: This would be where we define the services used in the stack. For example, we could define an EC2 instance, its type, security group etc. EC2Instance: Type: AWS::EC2::Instance Properties: InstanceType: Ref: InstanceType SecurityGroups: - Ref: InstanceSecurityGroup Parameters: If we define an instance, with its type, this is where that “parameter type” would be passed in: Parameters: InstanceType: Description: WebServer EC2 instance type Type: String Default: t2.small Configuration files for CF are written either in YAML or JSON. Converting CF to TF In this document, I’ll take you through the steps I went through on how to convert CF to TF. In particular, a recent project I worked on. In case you haven’t heard about it, CIS is the Center for Internet Security, and they provide cyber security standards and best practices. Recently, AWS launched a new service called AWS Security Hub, which analyses security findings from various supported AWS and third-party products. Security hub supports the CIS AWS Foundations Benchmark, (read more here) which, quoting CIS is “An objective, consensus-driven security guideline for the AWS Cloud Providers”. To jump straight into it, AWS Security Architects partnered up with Accenture and created a CIS-Foundations Quickstart written in CloudFormation. So, after looking around, realised there wasn’t any versions written in Terraform, and also no guides on how to translate it. Or automated translation tools for the matter (future work? hit me up for a collab) I decided to do it manually, as I felt this was a bit of a sensitive project to be testing automated tools on. But fear not, I did not do it as manually as you think. Simplicity above everything! Part 1: Understand the structure, state the stack Lets take a look at how the CloudFormation CIS Benchmark Quickstart works. The stack can be described as follows: Cloudtrail AWS Config S3 Templates are the following: Pre-requisites template: makes sure CloudTrail, config and S3 are created or exist and meet the preconditions for CIS Benchmarking: Config must have an active recorder running. CloudTrail must be delivering logs to CloudWatch Logs Config Setup template: sets the configurations needed for AWS config CloudTrail-setup template: sets the configurations needed for CloudTrail CIS-benchmark template: this is the tricky one, it contains all 42 objectives the account should meet to be CIS foundations compliant. Main template: this is the main template, and it nests the stacks created from the previous templates so it can deploy the CIS AWS Foundations benchmark. Part 2: design TF Now that we stated how this CF project works, lets see how we can transform them into the likes of Terraform. The templates can be transformed into modules. Pre-requisites can be part of the config and Circle CI checks (we will take a look at that in the end) Main template will be the main.tf, contains all the callable modules. Lets see how a CF template would look like: AWSTemplateFormatVersion: 2010-09-09 Description: (stuff) Metadata Labels: (Stuff) Parameters: (Stuff) Conditions: (more Stuff) Resource: (This is where all the cheesy stuff happens) Now, lets see how we can use that to “translate” into Terraform. Part 3: translation Now, apart from tedious, translating line by line, especially in a big project, is a bit of science fiction (for me). So I dug around: 1) Terraform accepts CF stack templates: By Stating Resource: aws_cloudformation_stack_set, you can manage a CloudFormation stack set, so this functionality allows you to deploy CloudFormation templates. It only accepts JSON templates. Possible challenge: templates built in YAML instead of JSON No problem! I had this myself, after a bit of googling, there is actually a tool called cfn-flip explicitly for the translation of YAML to JSON in CF: So for example, if you want to create the template in json: $ cfn-flip main.template > main.json or, just copy the output: $ cfn-flip main.template | pbcopy 2) But, what if it’s a giant template? This is my case too, the cis-benchmark template is quite big. Luckily for us again, you can reference the json template, by uploading it to a S3 bucket. It would look like this: resource "aws_cloudformation_stack" "cis-benchmark" { name = "cis-benchmark-stack" template_url = "https://cis-compliance-json.s3-eu-west-1.amazonaws.com/cis-benchmark.json" } Note: never reference config files and templates that have hardcoded variables (also never hard code sensitive data ) that are hosted publicly. In my case think of that template as skeletal, it doesn’t have any sort of compromising info. And done, we just created a part of the module in just 3 lines of code! Challenges 3) Nested Stacks In our case, the Quickstart uses nested stacks. Now aws_cloudformation_stack terraform functionality doesn’t have a “nested stacks” option. But creating the resources in the same module works fine. Parameters If you need to pass Parameters, you can do it as you would normally do, state the vars in the resource where you create the stack and should be good to go too! Example code: (This is an extract of the module) #root module resource "aws_cloudformation_stack" "pre-requisites" { name = "CIS-Compliance-Benchmark-PreRequisitesForCISBenchmark" template_url = "https://{bucket-name}.s3-(region-here).amazonaws.com/cis-pre-requisites.json" parameters = { QSS3BucketName = "${var.QSS3BucketName}" QSS3KeyPrefix = "${var.QSS3KeyPrefix}" ConfigureConfig = "${var.ConfigureConfig}" ConfigureCloudtrail = "${var.ConfigureCloudtrail}" } } resource "aws_cloudformation_stack" "cloudtrail-setup" { name = "CIS-Compliance-Benchmark-cloudtrail-stack" template_url = "https://{bucket-name-here}.s3-(region-here).amazonaws.com/cis-cloudtrail-setup.json" capabilities = ["CAPABILITY_IAM"] } [...] 4) Done! Now you can simply run and manage your stacks using Terraform. I suggest to always be careful with sensitive data and parameters and follow best practices. You can read more about it here Conclusion When it comes to features, CF and TF are not equivalent. It is not possible to express what CF is able to deploy in TF. Which is why I aimed at this solution, translating line by line would be very tedious, so if that is your case i’d suggest rewriting the entire module in TF. However writing a translator would be complex but very useful, still would have to figure out how it would work when CF uses intrinsic functions (please contact me for ideas!) but i’d guess that’ll be for future work. I hope this quick workaround helped you out!

You may also be interested in Related companies

logo
Swiggy
HQ
Bengaluru, IN
Employees
11086  -9decrease
Swiggy is a company which offers an on-demand food delivery platform designed to provide food from neighborhood restaurants to the customers.
View company
logo
Thriver
HQ
Toronto, CA
Employees
11 
Thriver (formerly known as Platterz) owns and operates an online meal ordering marketplace for ordering large and shareable meals for offices.
View company
logo
GrubHub
HQ
Chicago, US
Employees
2722  28increase
GrubHub provides an online and mobile platform for restaurant pick-up and delivery orders.
View company
logo
CMEOW
HQ
Richmond Hill, CA
Employees
11 
CMEOW is a company which provides consumer food services.
View company
When was Deliveroo founded?
Deliveroo was founded in 2012.
Who are Deliveroo key executives?
Deliveroo's key executives are Adam Miller, Will Shu and Akshay Navle.
How many employees does Deliveroo have?
Deliveroo has 6,692 employees.
What is Deliveroo revenue?
Latest Deliveroo annual revenue is £277.14 m.
What is Deliveroo revenue per employee?
Latest Deliveroo revenue per employee is £102.49 k.
Who are Deliveroo competitors?
Competitors of Deliveroo include touch2success, Uber Eats and GrubHub.
Where is Deliveroo headquarters?
Deliveroo headquarters is located at 1 Cousin Lane , London.
Where are Deliveroo offices?
Deliveroo has offices in London, Los Angeles, Balaclava, Brussel and 9 other locations
How many offices does Deliveroo have?
Deliveroo has 15 offices.

Craft real-time company insights

Learn about Craft real-time company insights

Receive alerts for 300+ data fields across thousands of companies

Footer menu