API design

Designing APIs in a resource-oriented architecture Edit

Table of Contents

  1. Introduction
  2. General principles
    1. RESTful
    2. Hypermedia / HATEOAS
    3. Fine-grained
  3. API and domain modelling
    1. Listing intrinsic properties
    2. Listing relations
    3. Normalising concepts
  4. Documenting APIs
  5. Conventions on requests
    1. Content type negotiation
    2. Resource lifecycle
    3. Path segments
    4. Naming
    5. Parameters
    6. Security
    7. Versioning
    8. Internationalisation (i18n)
  6. Conventions on responses
    1. Single-resource representation
    2. Single-entity GET endpoints
    3. Collection GET endpoints
    4. POST, creating entities
    5. PATCH, mutating entities
    6. DELETE, destroying entities
    7. Return codes and errors
    8. Query parameters
    9. Caching
    10. Mutable resources
    11. Compression
  7. External-facing APIs
    1. Mobile-friendly APIs
    2. Public-friendly APIs
  8. Tools of the trade
  9. Further reading
    1. Principles
    2. Examples of decently well-thought-out APIs and guidelines
    3. Context and debate
    4. More on JSON APIs

Introduction

This set of guidelines and conventions outline how to design APIs that are reusable and match with our Service design guidelines.

These guidelines mostly apply to internal APIs, meant to be consumed by software we build and maintain.

APIs that face the public, or 3rd-party integrators, or simply our own apps outside the datacenter, have very different constraints.

The section external-facing APIs has details on how to handle those cases.

Note to readers: Many responses in this document will be represented as equivalent Yaml instead of JSON for conciseness; actual responses should still be JSON.

In our examples, as a use case we’ll generally assume we’re building APIs for a hotel booking website - concepts will include hotels, rooms, bookings for instance.

General principles

We choose to adopt three general principles. Here’s a shortcut to remember:

RESTful, Hypermedia, Fine-grained

RESTful

We decide that our APIs will let consumers perform Representational State Transfer, as opposed to Remote Procedure Call. In particular, this means that:

  1. The top-level concepts of the APIs are always nouns, i.e. paths contain nouns which refer to the domain concepts.

  2. The only verbs are HTTP verbs: GET to read, POST to create, PATCH to modify, DELETE to destroy, and HEAD to obtain metadata.

  3. Read methods (GET, HEAD) have no side effects, and write methods (PATCH) are idempotent.

  4. DELETE is not idempotent and should return 404 or 410 when the resource does not exist (or not any longer).

Example of verb vs. noun usage:

# Good
POST /bookings { hotel: { id: 1234 } }

# Bad
POST /hotel/1234/book

Example of proper method usage:

# Good
PATCH /bookings/432 { state: "requested", payment_id: 111 }

# Bad
POST  /bookings/432 { state: "requested", payment_id: 111 }

Note that the PUT verb, which is fairly ambiguous (can both create or update a resource) should generally not be used.

Hypermedia / HATEOAS

The principle of HATEOAS is that “a client interacts with a network application entirely through hypermedia provided dynamically by application servers. (…) The HATEOAS constraint decouples client and server in a way that allows the server functionality to evolve independently.”

In practice, this means that interacting with the API should generally rely on URLs, not IDs (like our internal, numeric identifiers for resources). In responses, associations are specified using their URL.

More importantly, consumers should not need to construct URLs, instead using only URLs dynamically discovered in responses.

Ideally the domain can be discovered by calling GET on the root:

#> GET /api
#> Accept: application/json

#< HTTP/1.0 200 OK
#< Content-Type: application/json

_links:
  hotels:
    href: /api/hotels
  hotel:
    href: /api/hotels/{id}
    templated: true
  bookings:
    href: /api/bookings
  booking:
    href: /api/bookings/{id}
    templated: true

This lowers coupling as consumers no longer need to maintain a copy of the routing table of the services they consume.

HATEOAS is difficult to achieve in practice on large APIs, but is a very valuable target to aim for - it significantly improves maintainability and allows for high-level clients that can “walk” relationships transparently.

Fine-grained

A fine-grained API should provide

The purpose is to honour the “principle of least surprise” and minimise confusion with developers consuming the API; we aim to make the answers to “how do I get information about a {thing}” or “what’s this field for again” as obvious as possible.

In practice, this means that:

A given entity has a single, canonical route.

… although there may be more than one route for its concept.

Good:

GET   /users/{id}              # single user
GET   /users                   # user index
GET   /hotels/{id}/guests      # hotel's user index

Bad:

GET   /users/{id}              # single user
GET   /hotels/{id}/guest/{id}  # duplicate!

Embedding entities should be avoided

If an entity’s representation contains a representations of its relations,

In practice, embedded documents should be avoided as they make caching horribly difficult.

Good:

#> GET /hotels
#< HTTP/1.0 200 OK
_links:
  hotel:
    - href: /hotels/123
    - href: /hotels/124
#> GET /hotels/123
#< HTTP/1.0 200 OK
id: 123
name: "Luxury resort in Marylebone"
_links:
  manager:
    href: /users/111

Bad:

Embedding on index requests.

#> GET /hotels
#< HTTP/1.0 200 OK
hotel:
  - id: 123
    _links:
      manager:
        href: /users/111
  - id: 124
    _links:
      manager:
        href: /users/112

Embedding on resource requests.

#> GET /hotels/123
#< HTTP/1.0 200 OK
id: 123
_embedded:
  manager:
    id:   111
    name: "John O'Foobar"

Exceptions on embedding can be made on a case-by-case basis, see the “Domain modelling” section below.

Few fields should be returned

Few fields mean the response payloads will be small and be more cacheable, both good characteristics of an API.

If a representation has many fields, it’s usually a symptom of poor domain modelling; a classic cause being that the representation is just a dump of the underlying storage columns.

Look out for implicitly embedded relations as a possible API design issue, and normalise/decouple the API. Also note that a service does not necessarily need to expose all it knows about a resource; and definitely should not expose anything only relevant to how it persists it.

Many calls may be required

A consequence of a well-normalised API is that many calls may be required to render anything significant.

For instance, take a listing page for a product catalog: you’ll probably need to make

For those coming from coupled applications, you’ll typically make one call per database row you’d ordinarily fetch. This may sound dire, but isn’t normally a problem with a good use of caches:

An important corner case is when building mobile-friendly APIs as opposed to inter-service APIs. Here, it’s often important to limit the number of requests, mainly because the client cost is very high (HTTP connections are not reusable, slow to establish, and cannot be parallelised) and scalability is poor (caching space is limited, bandwidth is limited).

The recommended pattern is not to disregard these guidelines, but instead to build a facade service which:

Such a facade service can be considered a “view service” which pre-renders to JSON.

See also the External-facing APIs for generics on non-internal APIs; this article also has a more elaborate explanation and example.

API and domain modelling

Defining good APIs (with respect to the principles outlined above) relies on domain-driven design.

This, in turn, requires one to abstract out any implementation details (particularly, how “things” will be stored in a database), and instead reflect on what the domain is, how it can be split down into concepts and operations on those. Clarity on naming is crucial.

We recommend reading about Domain driven design, although in many cases common sense can be enough.

An entity of the domain is an object that is not defined by its attributes, but rather by a thread of continuity and its identity. A given user, a given hotel are entities; their name may change without breaking the “thread of identity”. We refer to a given identity by a (unique) identifier, its URL. For instance, User 1234 can solely referred to by the URL /users/1234.

A concept of a domain is the set of entities that have a similar representation and lifecycle; users or hotels are concepts.

An entity can have any number of representations. The canonical one is obtained by requesting its URL, and is composed of

Note that intrinsic properties are not “database fields”; the worst possible way to represent an entity is by dumping the way it’s been stored in a legacy system.

Listing intrinsic properties

Listing intrinsic properties is a difficult task, as it’s usually a grey area with no hard answers. We can, however, provide a number of hints that a property is intrinsic (and therefore should be part of the representation) or extrinsic (and should probably be part of a linked entity’s representation, instead).

No single hint can lead to the conclusion that a given property is intrinsic or extrinsic; it’s generally the addition that matters.

Hint towards extrinsic: is a user’s avatar a property, or a separate entity?

Hints towards intrinsic:

A classic trap is the “physical inclusion” trap. For instance, rooms are inside hotels does not imply that the representation of rooms must be properties of the representation of hotels. They can, but that’s a modelling decision; one can, for instance

Listing relations

Typically, when exposing a concept with an API, the database will contain a number of thing_id columns.

These are relations, not properties; the payload can contain a number of links to the corresponding resources, but should not (ever) contain thing_id properties.

Good:

#> GET /hotels/123
#< HTTP/1.0 200 OK
id: 123
_links:
  self:
    href: /hotels/123
  city:
    href: /cities/456

Bad:

#> GET /hotels/123
#< HTTP/1.0 200 OK
id: 123
city_id: 456

Normalising concepts

Elaborating on the example above, it’s not uncommon for an entity to refer to multiple, similar others. A hotel’s record can for instance contain a city_id, region_id, and country_id.

The naive transformation into an API would be to entities of the city, region, and country concepts;

One could argue this is a lack of normalisation; and that cities, regions, and countries are actually entities of a broader places concept; hotels would then relate to a number of places with varied kind properties, and which relate to each other as a tree (or digraph) — but depending on the use case, this might be cumbersome over-normalisation.

Documenting APIs

API users are both developers and machines; therefore, you should:

Conventions on requests

Content type negotiation

All requests should include the Accept: application/json headers.

Requests may use the application/json MIME type instead for backwards compatibility reasons.

The Accept header may include the v parameter to specify the API version requested; see “Versioning” below.

Server may react to the Accept-Language header, see “i18n” below.

Resource lifecycle

All GET requests for a single resource may specify If-* headers to avoid fetching payloads when revelant (thus expecting a possible 304 response).

All PATCH requests must include at least one If-* header (either If-Unmodified-Since or If-Match) to avoid editing conflicts.

Path segments

There should not be more than 3 path segments, API root (typically /, /api, or /api/{tenant}) excluded.

In practice:

As a rule of thumb, there should not be more than one (numeric) identifier per URL.

Naming

All path segments which refer to a domain concept should be plurals, except if there is only zero or one entity in the concept (singleton relations).

Note that relation endpoints must link to a toplevel endpoint.

Example:

# Singleton
/manager_profiles/{id}
/users/{id}/manager_profile

# Normal case
/photos/{id}
/hotels/{id}/photos

Parameters

Endpoints returning single entities should not accept any parameters. They may return an error if parameters are passed.

Collection endpoints may accept parameters (e.g. for filtering). If they do, those must be specified in the root document’s link relations.

Example:

#> GET /api
#< HTTP/1.0 200 OK
_links:
  hotel:
    href:      "/hotels/{id}"
    templated: true
  hotels:
    href:      "/hotels{?published}"
    templated: true
  hotel_photos:
    href:      "/hotels/{id}/photos{?default}"
    templated: true

Note: in a root document, the href fields will typically be URI templates as per RFC 6570.

Security

A service must accept connections over HTTPS. It should not respond over plain HTTP, and in particular, it should not redirect from HTTP to HTTPS. It should respond to plain HTTP requests with status 426, Upgrade Required.

A service should require HTTP Basic authentication. It should ignore the username and use the password as a token. It may accept unauthenticated requests for some endpoints.

Rationale: why not HTTP Digest?

Rationale: why not an X-Token header?

Rationale: why not ?token=abcd in the query string?

Versioning

A service may provide different APIs (endpoints and representations) in the form of API versions.

Clients may specify a desired version as the v parameter of the Accept header, for instance:

Accept: application/json;v=2

The service should respond with status 406, Not Acceptable if the version is unavailable.

If a version was specified by the client, and is available, the service must respond with the same version:

# Request:
Accept: application/json;v=2

# Response:
Content-Type: application/json;v=2

If the version was unspecified, the server should use the latest available version, and specify the Vary header, as future request may yield a different response:

# Request:
Accept: application/json

# Response:
Content-Type: application/json;v=2
Vary: Accept

Finally, a service’s root endpoint should list the available versions:

#> GET /api
#< HTTP/1.0 200 OK
_links:
  ...
_versions:
  - 1
  - 2

Note: Another Approach is to version APIs through path segments (e.g. /api/v1/things/123). We choose not to follow it. The major issue is that entities may have multiple URLs which risk being misinterpreted as referencing different entities.

Internationalisation (i18n)

A service may provide internationalised representations of entities.

A client may specify their desired locale using the Accept-Language header as per RFC 2616.

If representation is localised, the service should include the Content-Language header in the response.

If the locale requested is not available, the service should respond with status 406, Not Acceptable.

If the response does not match the requested locale exactly (either more than one locale options were requested, or none), the service should include the Vary: Accept-Language header in the response.

Rationale:

Conventions on responses

Responses should be valid JSON-HAL documents.

In addition, embedded entities (using _embedded) should be avoided when possible, and only introduced:

Single-resource representation

A single resource representation should have a numeric id field. It must have a link to self. It may contain a number of intrinsic properties of the entity (see the “domain modelling” discussion above.

Example:

#> GET /hotels/1234
#< HTTP/1.0 200 OK
id:   1337
name: "The Four Seasons*****"
lat:  1.2345
lng:  45.678
_links:
  self:
    href:   "/hotels/1337"
  reviews:
    href:   "/hotels/1337/reviews"
  manager:
    href:   "/users/8008"
    type:   "user"
  photos:
    href:   "/hotels/1337/photos"

Note that intrinsic-ness of a given property is a gray area: think hard and have a debate whenever considering adding another property to a representation.

There may occasionally be arguments outside the domain, e.g. performance considerations. For instance, one may decide to model hotel descriptions (which are lengthy text blobs) as a relation to hotels, instead of as an intrinsic field, because (a) the payload would become very large, and (b) it is seldom needed by consumers.

As a particular case, note that fields that count relations (eg. photos_count in the example above) are not intrinsic and should not be made part of the representation.

In exceptional cases, counts (which are a property of the relation) may be mentioned as a part of the link metadata. Consumers should not expect the value to be authoritative, and should refer to the relation URL if consistency is required.

Example:

#> GET /hotels/1234
#< HTTP/1.0 200 OK
id:   1337
_links:
  self:     "/hotels/1337"
  photos:
    href:   "/hotels/1337/photos"
    count:  27

Single-entity GET endpoints

A single-entity GET endpoint should always be of one of the forms

Such endpoints must return the representation of a single entity, and any links, as described in the previous section.

Partial responses (e.g. with field query param) should not be returned.

Responses should include a Last-Modified or ETag header, and may include both. The ETag should be based on a hash of the response payload, not on timestamps.

Collection GET endpoints

A collection GET endpoint should be of one of the forms:

Such endpoints must return a representation of the collection. They must link to a (possibly empty) list of entities.

Rationale: In domain terms, an index endpoint actually returns a view on the collection of resources; ie. the resource returned is the view. The current page, links, and page size are data (intrinsics) of that view. The number of pages and the total number of resources depend (if your view can filter it’s data; if it can only order it’s metadata).

A collection representation

Example:

#> GET /hotels?checkin=2016-01-02&checkout=2016-01-09
#< HTTP/1.0 200 OK
page:     1
per_page: 10
total:    153277
_links:
  self:   
    href:   "/hotels?checkin=2016-01-02&checkout=2016-01-09&page=1"
  prev:     null
  next:   
    href:   "/hotels?checkin=2016-01-02&checkout=2016-01-09&page=2"
  hotels:
    - href: "/hotels/1"
    - href: "/hotels/2"

Exceptionally, a collection representation may embed representations of the linked resources, which may be incomplete, but must include at least a mandatory link to self.

Note that as for other use cases of _embedded, there should be a very robust reason to do so as it makes using the API more complex (partial representations, caching issues, etc).

Example:

#> GET /hotels?checkin=2016-01-02&checkout=2016-01-09
#< HTTP/1.0 200 OK
page:     1
per_page: 10
total:    153277
_links:
  self:   
    href:   "/hotels?checkin=2016-01-02&checkout=2016-01-09&page=1"
  prev:     null
  next:   
    href:   "/hotels?checkin=2016-01-02&checkout=2016-01-09&page=2"
  hotels:
    - href: "/hotels/1"
    - href: "/hotels/2"
_embedded:
  hotels:
    - id: 1
      _links:
        self: 
          href:   "/hotels/1"
    ...
    - id: 10
      _links:
        self: 
          href:   "/hotels/2"

POST, creating entities

A collection GET endpoint may respond to the POST method to create new entities.

If it exists, it should return status:

Additional 4xx response codes may be used:

The response must be a valid single resource representation, although it may be partial, including at least the numeric id and the mandatory link to self.

Example:

#> POST /hotels
name: "Castle by the lake"
lat:  1.2345
lng:  45.678
#< HTTP/1.0 201 Created
id: 1337
_links:
  self: "/hotels/1337"

PATCH, mutating entities

A single-resource GET endpoint may respond to the PATCH method to modify existing entities.

The response status should be

The response must be a valid single resource representation, although it may be partial, including at least the numeric id and the mandatory link to self.

Example:

#> GET /hotels/1337
#< HTTP/1.0 200 OK
#< Last-Modified: Tue, 15 Nov 1994 12:45:26 GMT
#< Etag: "e04c6ca4-6ac9-11e6-ab5a-cf7dd1791cc9"
id:       1337
name:     "Castle by the lake"
_links:
  self:   "/hotels/1337"

#> PATCH /hotels/1337
#> If-Match: "e04c6ca4-6ac9-11e6-ab5a-cf7dd1791cc9
name:     "Manor by the lake"
#< HTTP/1.0 200 OK
id:       1337
name:     "Manor by the lake"
_links:
  self:   "/hotels/1337"

DELETE, destroying entities

A resource GET endpoint may respond to the DELETE method to permanently destroy an existing entity.

If it exists, it should return status:

Additional 4xx response codes may be used:

The response should be empty on success.

Example:

#> DELETE /hotels/1234
#< HTTP/1.0 204 No Content

Return codes and errors

In the case of client or server errors (i.e. when the return code is 400+), the content-type should be application/json.

The results are not just intended to be acted on by machines, but rather presented to users.

The response should be an errors object whose keys are either keys present in the original request, parameter names, or the general key for failures not attributable to a request key.

Each value should be a human readable message, localised according to the Accept-Language header.

Example: index with out-of-bounds page:

#> GET /hotels?page=52196
#< HTTP/1.0 404 Not Found
errors:
  page: "Page is out of bounds"

Example of a PATCH version fail:

#> PATCH /hotels/1337
#> If-Match: "e04c6ca4-6ac9-11e6-ab5a-cf7dd1791cc9"
name:       "Manor by the Lake"
#< HTTP/1.0 412 Precondition failed
errors:
  version:  "Resource was updated since you read it."

Example of bad/missing values in POST:

#> POST /hotels/1337
lat: "foobar"
#< HTTP/1.0 400 Bad Request
errors:
  name:  "Name is required."
  lat:   "Latitude must be a floating-point number."
  lng:   "Latitude is required."

Example of bad/missing values in POST:

#> POST /hotels/1337
name: "Luxury resort"
#< HTTP/1.0 409 Conflict
errors:
  name:  "Name must be unique."

HTTP status codes should be used as possible to being semantic where these guidelines are unclear. In particular, syntactic and semantic failures should not be confused:

Likewise, for success codes:

Query parameters

Single-entity endpoints should not accept query parameters (for any HTTP method).

Those endpoints may return 400 Bad Request if parameters are specified.

Collection GET endpoints are the only endpoints that usually accept query parameters. Those should accept the page and per_page parameters. They may accept parameters that match property names of the corresponding concept; if they do, they should

They may accept the order parameter; if they do,

Example:

GET /hotels?page=1&order=-updated_at,name

They may also respond to other parameters, although it is not recommended. If they do those should be mentioned in the root document and the behaviour is unspecified.

Caching

Caching efficiency is a critical aim of well-designed APIs, as it is influential on service performance; cache consistency is as important.

These guidelines only consider HTTP/1.1 and later. If the API is internal then you can make this a requirement. External APIs must always use TLS so only direct clients or trusted intermediaries who have our certificates (CDNs, typically) will be able to view the content; all CDNs support 1.1 or later and it’s not too much of a stretch to make this assumption for direct clients.

Responses with the following status codes should specify a Cache-Control header because without one the HTTP specification allows clients to cache them according to their own cache policy which is typically more lax than desirable:

The following status codes should also specify this header because many CDNs or intermediaries will choose to cache them even though they are not permitted to do so by the HTTP specification:

Other status codes should not specify a Cache-Control header

The HTTP Cache-Control header is somewhat confusing and some of the directives do not mean what you think they do. A basic summary of the confusing ones is:

For full details, and information about the other directives such as public, private and max-age, refer to RFC 7234 § 5.2.

Most of the time it is fine for clients to cache data and it’s often acceptable for the data to be at least somewhat stale (even if it’s just a minute or two) but rarely fine to use them after the expiration time, so in general your Cache-Control header should be:

Cache-Control: private, max-age={seconds}, must-revalidate

If the resource is immutable then {seconds} should be 31536000 which is one year, the maximum allowed. Statuses 301, 308 and 410 should be considered immutable as they are permanent conditions.

For resources that absolutely must be up-to-date when used you still normally want to allow the efficient return of 304 Not Modified so choose no-cache (note that this is typically the best choice for the 302, 307 and 404 status codes mentione above):

Cache-Control: private, no-cache

In the rare cases where data is extremely sensitive and must never be cached anywhere (for example, a password reset token) then use:

Cache-Control: no-store

Remember that because no-store prevents any kind of caching that clients cannot use conditional directives to get 304 Not Modified because they are not permitted to store the data between requests, so have no reference for the unmodified resource.

Responses should include an ETag header with a strong ETag; if this is not practical then they should include a Last-Modified header (ideally, include both). Strong ETags must be based on a hash of the response, not on timestamp information. Do not use weak ETags because they have confusing semantics, for example they cannot legally be used in preconditions on PUT, PATCH or DELETE requests.

Any GET requests may use either If-None-Match or If-Modified-Since, and all PUT/PATCH/DELETE requests should use either If-Match or If-Unmodified-Since. If the request provided a strong ETag then the “match” headers are better, otherwise use the “modified” headers.

Mutable resources

Single-entity endpoints should return an Etag header and a Last-Modified header.

They should accept If-None-Match and If-Modified-Since and return status 304 (and no payload) as appropriate.

Mutation endpoints should honour the If-Match and If-Unmodified-Since headers as appropriate.

Compression

Servers may support the Accept-Encoding header for compression purposes, but this is not mandatory.

Rationale: latency is more important than bandwidth savings for most internal APIs; therefore the overhead of compression is seldom justified.

External-facing APIs

We want to do our best to make out internal services use HATEOAS and we can try and catch any URL construction in PRs, but for anything exposed to third-party API consumers (integrators, developers in the general public) — it’s unlikely that everyone will stick to these ideals.

Performance constraints can also be quite different.

Mobile-friendly APIs

To build APIs that are friendly to mobile consumers, special attention is needed to limit the number of requests. This is because mobile connections are (relatively) high latency, and the cost of the roundtrip can result in bad user experience.

Our recommendation is to

  1. Still expose “pure”, RESTful, hypermedia APIs, but not to the app directly;
  2. Provide a “mobile adapter” service that uses the pure APIs to provide a less “chatty” interface.

The benefit of this approach is that the caching capabilities of the RESTful approach are preserved. The adapter service can aggressively cache representations, but has little logic beyond that — in particular, it owns no domain concept and should normally have no persistent storage.

In particular, the mobile adapter can take care of user-facing request authentication; whereas the internal services only need to care about service-to-service authentication.

Public-friendly APIs

For external services we should to stick to a somewhat different set of principles, because of the low incentive for 3rd-party consumers to support maintainability of our software.

  1. As above, public APIs should be implemented in terms or our private, “pure” APIs, in separate adapter services.
  2. API URLs should never change (because consumers risk constructing their own).
  3. While it is still recommended to include hypermedia links to encourage good practices, is it not mandated like for internal services.
  4. The recommended practice for versioning is DNS-based: a new (breaking/major) version of a set of public-facing APIS should be an entirely new domain (e.g. v2.my-api.example.com), with entirely segregated infrastructure.

Tools of the trade

We strongly recommend using Rails 5 to build API services, as per the service guidelines. Rails’s API-only mode has solid support for API building.

If the API service also has a user interface, it is suggested to make the API part a mounted Rails engine using API mode.

Using Sinatra is not recommended as there is no significante performance benefit over Rails, it lacks a router (which means all link URLs must be manually built), and most non-trivial Sinatra apps end up reinventing MVC.

Using Grape is not recommended as it lacks a router as well.

Further reading

Principles

Examples of decently well-thought-out APIs and guidelines

Context and debate

More on JSON APIs