REST Web API in Practice: Improving towards Perfection
Designing REST Web APIs in practice has many challenges. Usually, various questions are born for different parts of the project. Our decisions can determine, among others, the usability, extensibility, and performance of our APIs.
In our previous articles, we saw how the REST architectural style guides us to create Web APIs by defining six constraints (principles) and some key points that we can consider when designing APIs (.NET Nakama, 2021 September 4). Furthermore, we saw some practical suggestions for adopting consistent naming conventions in our URLs (API endpoints) and how to perform filtering, sorting, and pagination (.NET Nakama, 2021 October 4).
This article examines several topics that developers usually deal with when designing REST Web APIs. The topics include HTTP status codes, Media Types, HATEOAS, and Caching.
The catchy title about perfection is an excuse to remind us that there is no perfection in software development. There will always be bugs and space for improvement as the software and business requirements evolve. So, we can continuously improve our knowledge and code. However, our choices can take our Web APIs to the next level, but it requires always keeping in mind the following rules:
- Understand the needs of the API users.
- Do not follow the API best practices blindly.
- Examine and document each decision.
- Design APIs with consistent behavior.
The HTTP protocol defines a standard list of HTTP status codes that a server can respond to the client. The status codes indicate whether the client requests have been successfully received and processed by the server, an error has occurred, or some other action is needed. The HTTP status codes are grouped in the following five classes depending on the:
- 1xx (i.e., 100-199), Informational Response.
- 2xx (i.e., 200-299), Successful: The request was successfully received, understood, and accepted.
- 3xx (i.e., 300-399), Redirection: Further action is required to complete the request.
- 4xx (i.e., 400-499), Client Error: The client request contains a wrong syntax or cannot be fulfilled (i.e., the resource cannot be found).
- 5xx (i.e., 500-599), Server Error: The server failed to complete an apparently valid request (e.g., due to an internal error, timeout error, etc.).
System.Netnamespace contains the HttpStatusCode Enum, which includes the values of status codes defined for HTTP.
Any Web API needs to return consistent and meaningful HTTP status codes based on their standard definition. So, for example, we shouldn’t produce a
200 OK response when returning an error. In the following table, we can see the most appropriate (for most cases) used Success HTTP status codes per HTTP method.
|HTTP Method||Success HTTP Status Codes for Synchronous Operations||Success HTTP Status Codes for Asynchronous Operations|
We are returning a
202 Accepted status code for the asynchronous operations to inform the client that the requests have been received but not yet completed. In such case, we are also returning a
Location header with the URI, which the client should periodically call (using a GET method) to “ask” the server about the status of his original request (short polling). For more information about asynchronous operations, you can read the (Price E., et al., 2018).
Design APIs with consistent error handling and helpful information can eliminate confusion from the API clients when errors occur. I believe that we all had faced errors in APIs, which was almost impossible to understand the actual issue or the information was misleading. For that reason, we should use the Error HTTP Status Codes (4xx and 5xx) appropriately. In the following table, we can see some commonly used error status codes and how we should use them.
|Error HTTP Status Code||Description|
|400 Bad Request||The client’s request is not valid based on the server’s input validations.|
|401 Unauthorized||The user’s credentials are not valid (not authenticated), or the user did not provide credentials.|
|403 Forbidden||The user is authenticated (logged id) but doesn’t have the necessary permissions for this resource.|
|404 Not Found||The requested resource is not found on the server.|
|405 Method Not Allowed||An HTTP method is being requested that isn’t allowed for the authenticated user. For example, when a POST or PUT method is used on a read-only resource.|
|410 Gone||The requested resource is no longer available and will not be available again. This status code can be used for deprecated API calls (e.g., old API versions).|
|500 Internal Server Error||An unhandled exception occurs on the server, and a generic error message is returned.|
|502 Bad Gateway||The server acted as a gateway or proxy to another server from which it received an invalid response.|
|503 Service Unavailable||The server cannot handle the request (e.g., overloaded, down for maintenance, system failure, etc.).|
|504 Gateway Timeout||The server acted as a gateway or proxy to another server from which it didn’t receive a response in a specific timeframe.|
When implementing .NET Core applications, you could use the open-source Consistent API Response Errors (CARE) NuGet library to centralize the error handling (validation errors, application exceptions, and unhandled exceptions) and return consistent and valuable error information.
ASP.NET Core supports methods to initialize responses for the most common status codes at the ControllerBase class (of the
Microsoft.AspNetCore.Mvc namespace), such as the following.
|Initialize Response to Return||Produced HTTP Status Code|
||204 No Content|
||400 Bad Request|
||404 Not Found|
||Based on the provided Status Code.|
||Based on the provided Status Code with the objectresult.|
As we can understand, we should inform the server and the client about the format of the transferred data to transform it accordingly to their native object(s) (parsing).
In the HTTP protocol, formats are specified through media types (also known as a Multipurpose Internet Mail Extensions or MIME type). A media type is a standard (RFC 6838) that indicates the nature and format of a document, file, or bytes. The following table shows the most common formats that are used for Web APIs and their media type.
The clients include the
Accept HTTP header in their requests to determine the media type(s) that they can support. The
Content-Type header is used in the client requests or server responses to define the format of the transferred data. When the accepted or requested media types are not supported, the server responds with an appropriate HTTP status code (as shown in the following table) (Price E., et al., 2018).
|HTTP Header||Description||Example||Server HTTP Status Error Code|
|Accept||Determine the media type(s) that the client supports.||
||406 (Not Acceptable)|
|Content-Type||The format of the transferred data (in the request body).||
||415 (Unsupported Media Type).|
In .NET Core Web APIs, we specify which media types are supported in each controller using the
Produces attributes, as shown in the following example.
If you would like to support multiple response formats based on the client requests (e.g., from the URL), you can read the Andrew Lock (2017, August 01) article.
One of the REST architectural style constraints to provide a uniform interface between components is Hypermedia as the Engine of Application State (HATEOAS). Based on the HATEOAS constraint, the server should provide information for all available actions of each resource.
For example, when retrieving data of a bank account (e.g., balance information), the server could also return the URLs of possible actions, such as to deposit, withdraw, etc., as we can see in the following example (Wikipedia, 2021).
Currently, there are no general-purpose standards that define how to model the HATEOAS principle (Price E., et al., 2018). From my perspective, the most controversial and challenging to follow REST constrain is the HATEOAS. It adds a lot of complexity to the server project, and sometimes the clients do not use it. For that reason, several backend developers do not apply it. In such cases, the APIs are referred to as REST-Based (.NET Nakama, 2021 September).
Using HATEOAS offers several advantages, such as (SOA Bits and Ramblings, 2013):
- Explorable APIs,
- Inline documentation,
- Simpler client logic,
- Server ownership of URL structures,
- Easier versioning in the URI.
However, let’s keep in mind the key-point of “
Do Not Follow the API-Best-Practices Blindly” (.NET Nakama, 2021 September) and the “
Keep It Simple Stupid (KISS)” principle. Based on the needs of our clients and our project’s requirements, we should examine if the HATEOAS is worth implementing and then decide. Do not just try to make a RESTful API just for the name!
Caching is the process of temporarily storing data or files in storage to be accessed more quickly. This process is used when the computation or/and the retrieval of the data or files require significant time or/and resources (e.g., CPU, network bandwidth, etc.). However, it is essential to cache data or files for a duration related to each case (how frequently the specific data changes, its importance, etc.) because otherwise, they might be outdated (staled).
In client-server communication, we can perform caching on both sides for different reasons.
- In the server: We could cache (in Redis, MemoryCache, etc.) the data retrieved from a database or a third-party system to faster respond to the client.
- In the client: We could cache the retrieved data from the server to reduce the following requests.
The cache constraint in the REST architectural style is related to client caching guided by the server using HTTP headers to label the response data as cacheable or non-cacheable. In this way, the client can reuse the same data in later equivalent requests (usually for a limited timeframe), with partial or no interaction with the server (.NET Nakama, 2021 September). The HTTP client caching is generally performed for the
GET HTTP requests.
In the following sections, we will focus on the response HTTP headers that guide the clients caching.
Cache-Control header value (RFC-7234) can contain several directives (comma separated) to configure the caching policies of the requests and responses. The main idea is to cache the response until it becomes stale (based on the Cache-Control directives and the Expires HTTP Header). After that, stale resources can either be validated or fetched again (see Cache Validation section).
The following table presents the most common Cache-Control directives (Fielding R. et al., 2014).
|no-store||Cache-Control: no-store||A cache must not store any part of either the immediate request or response.|
|no-cache||Cache-Control: no-cache||The response must not be used to satisfy a subsequent request without successful validation on the server (it is always stale). In case it is used on a request, the server should regenerate the response for the client and store it in its cache.|
|public||Cache-Control: public||Any cache may store the response.|
|private||Cache-Control: private||The response message is intended for a single user and must not be stored by a shared cache. However, a private cache may store and reuse the response.|
|max-age||Cache-Control: max-age=5||Determine the number of seconds (e.g., five in the example) that a response can be cached. After that period, the cached value is considered stale.|
|must-revalidate||Cache-Control: must-revalidate||This indicates that once the cache has become stale, it cannot be used as a cached response to satisfy subsequent requests without successful validation on the origin server.|
Expires HTTP header contains the date/time after which the response is considered stale. In general, staled resources should not be used (Mozilla.org, 2021). If a
Cache-Control header exists with a
max-age directive in the response, then the
Expires header is ignored.
When a resource is staled (expired), it can either be validated or fetched again. The validation is performed to the server by providing some information in the request (as HTTP headers) about the specific resource (e.g., an identifier or the last modification date).
This information is provided by the server (in the original response) as HTTP headers, and specifically, it is the
ETag and the
The ETag HTTP response header is an identifier for a specific version of a resource. This identifier is usually a hash of the resource (see the following example). Thus, when the resource changes (different versions of the data), a new hash is generated.
Using the ETag value in an “If-None-Match” header to the following GET requests, the server can save network bandwidth by not resending the complete response when the content was not changed. In such cases, the server returns a
304 Not Modified HTTP status code which is used as a
200 OK response telling the client to use the cached resource.
Also, when transferring resources to the server, cache validation can be performed (e.g., POST, PUT, etc.) by sending the ETag value in an “If-Match” header. The server will compare the provided ETag value with the current resource ETag value, and if they match (i.e., it has not changed in the meantime), the update will be performed. On the other hand, if the ETag values do not match, the server will respond with a
412 Precondition Failed HTTP status code.
The Last-Modified response HTTP header contains a date and time when the server believes the resource was last modified. It is used similarly with the ETag HTTP header, but it is less accurate.
The Last-Modified value can be used in the same way in correspondence with the ETag value by using the following HTTP headers:
This article focused on HTTP status codes, Media Types, HATEOAS, and Caching to improve our REST Web APIs. Decisions on these topics can influence our Web APIs usability, extensibility, and performance.
The consistent and meaningful use of the HTTP status codes can make a huge difference in our API’s usability. For example, the server would inform better the API consumer applications and their developer(s) about what happened to perform the appropriate actions in case of errors.
Using HATEOAS offers several significant advantages regarding extensibility. However, it adds a lot of complexity, and sometimes the clients do not use it. Therefore, we should examine if the HATEOAS is worth implementing in each project separately and then decide. We should not just try to make a RESTful API just for the name!
Caching can improve the performance of our applications by temporarily storing data or files to be accessed more quickly. The cache constraint in the REST architectural style is related to client caching guided by the server. For that purpose, we examined the response HTTP headers that guide the clients caching.
There will always be space for improvement in our Web APIs and software development in general. However, we can continuously improve our technical skills, understand how things work, and gain experience, reflecting in our code and decisions.
- .NET Nakama (2021, September 4). Designing a RESTful Web API. https://www.dotnetnakama.com/blog/designing-a-restful-web-api/
- Fielding R., et al. (2014, June). Hypertext Transfer Protocol (HTTP/1.1): Caching. https://datatracker.ietf.org/doc/html/rfc7234#section-5.2.2
- Lock A. (2017, August 01). How to format response data as XML or JSON, based on the request URL in ASP.NET Core. https://andrewlock.net/formatting-response-data-as-xml-or-json-based-on-the-url-in-asp-net-core/
- Mozilla.org (2021, October 20). HTTP caching. https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching
- SOA Bits and Ramblings (2013, December 06). Selling the benefits of hypermedia in APIs. http://soabits.blogspot.com/2013/12/selling-benefits-of-hypermedia.html
- Price E., et al. (2018, December 1).RESTful web API design. https://docs.microsoft.com/en-us/azure/architecture/best-practices/api-design
- Wikipedia (2021, October 10). HATEOAS. https://en.wikipedia.org/wiki/HATEOAS#Example