Understanding the HTTP 431 Error: A Developer's Guide

The 431 HTTP status code, also known as "Request Header Fields Too Large," is a client-side error that indicates the server refuses to process a request because its header fields are too large. This error can be due to either the total size of the headers being too large or a single header field exceeding the allowed size.

The HTTP 431 status code was first introduced in RFC 6585, published in April 2012. This was part of an effort to standardize additional responses for HTTP/1.1, addressing issues that were not covered by the existing status codes at the time. Introducing this specific error code was crucial for enhancing server security and preventing malicious attacks like buffer overflow attacks, which could exploit large headers.

Understanding the HTTP 431 Error

When a 431 error occurs, it means that the server has blocked the client's request due to excessively large header fields. Each server has its own policies regarding the maximum allowed size for HTTP header fields, which helps protect against denial-of-service attacks.

Common Symptoms of the HTTP 431 Error

Total Size of Request Headers Too Large: When the error occurs due to the cumulative size of all request headers exceeding the limit.
Size of Individual Header Fields Too Large: In this case, a single header field is too large to process, leading to the 431 error.

Common Causes of HTTP 431 Errors

Too many cookies: Having excessively large or many cookies can cause the 431 error. This can happen if the same cookies are set multiple times or if multiple cookies are combined in a single request or response. One common scenario is reusing the same localhost port across multiple projects, and sending all of those project's cookies along with a request.
Improper Header Field Caching: Caching large headers incorrectly can lead to the 431 error. Improper caching can increase the header size, causing a buffer overflow.
Unnecessary or Poorly Formatted Request Headers: Overusing HTTP headers can result in requests that exceed the size limits set by the server, leading to a 431 response. Additionally, ensure your headers are properly formatted. Having a Referer URL that's too long can trigger a 431.

How to Avoid the 431 Error

Clear cookies: Regularly clear cookies on the client-side (chrome dev tool > Application > Cookies) and restrict the number of cookies used on the server.
Cache headers: Implement header caching correctly to ensure the cached headers reflect the actual request and compression response. Adjust ETags if you rely heavily on them for conditional requests, as large ETags can contribute to the error.
Minimize HTTP headers: Simplify your HTTP headers to include only necessary information. Avoid including redundant metadata, especially in headers like Authorization, Referer, and User-Agent.

How to Compress Header Size

Sometimes, you aren't sending too many headers and your headers are bloated due to constraints out of your control (ex. Your JWT token encodes a tonne of information). Header compression can significantly reduce the size of your request headers. Utilize protocols like HTTP/2 and HTTP/3, which support header compression by default. This can automatically reduce the size of your headers without requiring additional configuration.

Dealing with Server Max Header Size Limits

The HTTP spec does not define a limit on header size, however many servers do. Here's the limits for various popular web servers/hosts:

Apache: 8K
nginx: 4K-8K
IIS: 8K-16K
Tomcat: 8K-48K
Node: (\<13) - 8K; (\>13) - 16K
Cloudflare: 16K per header, 32K total

Most servers allow some form of configuration and different versions of the server software might have lower or higher limits. You should also check the latest documentation to see if these limits include other parts of the request (ex. cookies) or just the regular headers.

How to Increase the Request Header Size Limit

Sometimes, increasing the request header size limit may be necessary. You can typically do this from the web console or CLI for your server. If you are developing locally, there are typically CLI flags that can configure this value. Here's the flag to configure request header size limits in Node JS

--max-http-header-size=16384

Note: Increasing the header size limit should be done with caution, as larger headers can consume more memory and degrade performance.

How to Send HTTP 431 Error Responses

You might not want to rely on your server to determine your request header size limit, or maybe you want to enforce a custom header size limit per API endpoint. Follow the steps below to learn how to send your own 431 errors.

How to Build Request Header Size Limits into Your API

As mentioned above, your host + server combination will likely automatically enforce a limit. To build your own custom request header size limit capability in your API, you can use libraries like express in Node.js. Here is an example illustrating how to enable limits on request header sizes:

const express = require("express");
const app = express();
app.get("/", (req, res) => {
  const headerSizeInBytes = JSON.stringify(req.headers).length;
  const maxHeaderSizeInBytes = 2000; // example limit
  if (headerSizeInBytes > maxHeaderSizeInBytes) {
    res.status(431).send("Request Header Fields Too Large");
  } else {
    res.send("Hello World!");
  }
});
app.listen(3000, () => {
  console.log("Server is running on port 3000");
});

Differentiating Error Responses Between the Total Size of Request Headers and a Single Header Field

When handling the HTTP 431 error, make sure to differentiate responses between two scenarios:

Total Size of Request Headers Too Large: Return a response indicating that the cumulative size of the headers is too large.
Size of Individual Header Fields Too Large: In this case, provide an error response that states which specific header field exceeds the allowable size.

Modify the example from the previous section to achieve differentiated error responses:

const express = require("express");
const app = express();
app.get("/", (req, res) => {
  const headerSizeInBytes = JSON.stringify(req.headers).length;
  const maxHeaderSizeInBytes = 2000; // example limit
  const exceededHeaderField = Object.keys(req.headers).find(
    (key) => req.headers[key].length > maxHeaderSizeInBytes * 0.1, // example individual field limit
  );
   if (exceededHeaderField) {
    res
      .status(431)
      .send(
        `Size of Individual Header Field '${exceededHeaderField}' Too Large`,
      );
  } else if (headerSizeInBytes > maxHeaderSizeInBytes) {
    res.status(431).send("Total Size of Request Headers Too Large");
  } else else {
    res.send("Hello World!");
  }
});
app.listen(3000, () => {
  console.log("Server is running on port 3000");
});

What Does an HTTP 431 Response Look Like?

We recommend responding to your users with a Problem Details format API response.

HTTP/1.1 431 Request Header Fields Too Large
Content-Type: application/problem+json
Content-Language: en

{
  "type": "https://httpproblems.com/http-status/431",
  "title": "Request Header Fields Too Large",
  "detail": "Size of individual header field 'referer' too large",
  "instance": "/account/12345/msgs/abc",
  "trace": {
    "requestId": "4d54e4ee-c003-4d75-aba9-e09a6d707b08"
  }
}

Let Your Gateway Take Care of It

If you use an API gateway like Zuplo - you can easily add a policy in your request pipeline to handle this. Here's how to do it:

Navigate to your route in the Route Designer and click Add Policy on the request pipeline.

Add policy

In the Choose Policy modal, we have two options depending on what you want to do.

Limit the entire request size with the Request Size Limit Policy
Use the Custom Code Inbound Policy and copy the code sample from above (minus the express bits) into a typescript module which will be connected to the policy.

Conclusion

The HTTP 431 error is often triggered by overly large request headers. By optimizing your headers, compressing them as needed, and implementing request header size limits in your APIs, you can avoid encountering this error.

Additionally, implementing header size limit checks in your API is simple. Most servers already include defaults. You can also do it yourself at both the cumulative and individual header level from within your API routes - either with custom code or one of Zuplo's policies.