Creating Your Own API Integration Platform

I spent the past week at the API World conference and there was a lot of talk around API Integration Platforms. Clever salesmen will tell you its too complex for you to build - but I want to prove them wrong!

So, What's an API Integration Anyway?

I'm sure that you're familiar with APIs - and already use many 3rd party APIs to accomplish tasks. In fact you have probably built services and internal APIs that call multiple external APIs. For example, a marketing API that pulls contact information from the Hubspot API and sends them an email via Sendgrid's API. API Integration is all about connecting these internal and external APIs to each other properly so they can seamlessly exchange data, while API Integration Management refers to using tools or services to manage and analyze these connections.

What's an API Integration Platform?

An API Integration Platform is your centralized tool for performing API integration management. It's a solution that helps you manage and oversee integrations between different services and applications. Many paid platforms (ex. Boomi) handle the work of keeping your 3rd party APIs up to date and transforming data into standardized formats, making sure they play nice together with your services and apps.

Why Do Companies Use API Integration Platforms?

So, why are businesses jumping on the API integration bandwagon? Here are a few reasons:

Lack of Expertise

Not everyone who wants integrations between your systems and an external API is an engineer. Embedded iPaaS platforms like Zapier make it easy with their no-code approach - and I think this is a valid system worth paying for.

Easier Maintenance

If your system needs to connect to many 3rd party APIs, maintaining (ex. handling new versions and deprecations) all of them is can be a fulltime job, which might be better served by using an external platform. Unified APIs are often used for this purpose, and can be useful at scale.

Why You Might Want to Build Your Own API Integration Platform

Using ready-made solutions is cool, but sometimes building your own makes more sense, especially for us developers. Here's some tradeoffs to consider:

• Control: Your choice in external APIs is not limited to those offered in the iPaaS or unified API - which often don't include promising APIs from startups.
• Long-term Savings: You're essentially paying for a middleman when using an API integration platform - which can cost hundreds of thousands of dollars.
• Resource and Intensive: Building your own platform isn't a weekend project. It requires time and expertise in all of the APIs you are going to manage. You'll need to spend a lot of time reading docs and deciding on standardized formats for what your end-systems or applications will consume.
• Maintenance: You'll be responsible for updates, security patches, and keeping the system running smoothly. Luckily, most APIs you use won't break on you (either by accident or breaking change) that often.

Assuming the tradeoffs sound reasonable, let's finally get to building.

The Building Blocks of an API Integration Platform

Before we build our own, let's break down what makes up an API integration platform:

1. API Management

Here, you create and manage your APIs. You set up endpoints, control access, and handle versioning. It's like being the gatekeeper for your APIs.

2. Data Transformation

Not all apps speak the same language when it comes to data. Data transformation tools help convert data formats so when an external API switches from user_id to userId, your application can still get the old format.

3. Security and Authentication

Managing the security and authentication methods for various external APIs can be challenging. API keys or tokens often get mishandled and leaked, so these platforms often manage authentication for you.

4. Monitoring and Analytics

These tools let you keep an eye on how 3rd party APIs are performing. Track usage by your teams and monitor for errors on both ends.

5. Developer Tools

Think SDKs, documentation, and testing environments. These make it easier to build and test your integrations without wanting to throw your laptop out the window.

Step 1: Managing Your External APIs

Let's say I have an application that needs information about movies. For this, I am going to use the OMDB API. Making an API call for data on a movie is pretty simple.

curl --request GET \
  --url 'http://www.omdbapi.com?apikey=<YOUR_API_KEY>&t=The%20great%20gatsby' \

and you'll get back the response

{
  "Title": "The Great Gatsby",
  "Year": "2013",
  "Rated": "PG-13",
  ...
  "imdbID": "tt1343092",
  ...
  "Response": "True"
}

They also offer a Poster API which we will use. Let's put these both into a centralized place so we can better manage and integrate with them. For this tutorial, I am going to use the Zuplo gateway since it will help me with later steps. After you sign up for Zuplo and create a project, you will need to configure your project to have two routes - one for the Movie Data API and the other for the Movie Poster API. Make sure you set the Request Handler to URL Rewrite and the Rewrite URL to the corresponding URL on the OMDB API. It should look like this:

Congratulations - once you save the file, you now have a live API proxy over the OMDB API.

Now here comes the "management" part. If you read the docs - you'll see that you can either provide a movie title or an IMDb ID to query movie data, but need to use an IMDb ID for posters. Let's standardize this API integration to use the i param only and deny the t param. This will require a quick script - which you can create by clicking the + next to the modules directory, and adding a new Inbound Policy called deny-t-param.

The code is very straightforward - return a 400 Bad Request error if the t param is sent.

import { ZuploRequest } from "@zuplo/runtime";

export default async function policy(request: ZuploRequest) {
  if (request.query.t) {
    return new Response(
      "Use the i param instead of the t param for consistency between APIs",
      { status: 400 },
    );
  }

  return request;
}

Go back to routes.oas.json click the Policies dropdown and then Add Policy under Request. What you see is a menu of "inbound policies", code modules that run on the incoming request. Select the Custom Code Inbound policy, change the module to point to your new policy module, then click OK and save your project.

Now if you try calling your Zuplo proxy API (you can use the Test button), with your API key and the t param, you will get an error.

Sweet, we now have some lightweight management over this external API.

Step 2: Data Transformation

I'm not a huge fan of how the API response with a mix of uppercase and lower-camel-case properties - it makes it more difficult for developers to consume relative to other APIs which just use lower-camel-case. Let's perform a quick data transformation to fix this. To get started, Click Add Policy under Response. You will see a different menu of policies, as these are "outbound policies" which run on the API response. Select the Transform Response Body policy. This will open the template which we will customize - simply click Generate.

Navigate to transform-body-outbound.ts under modules. let's modify it a bit to transform all the keys to lower-camel-case.

import { ZuploContext, ZuploRequest } from "@zuplo/runtime";

function lowerFirstChar(str: string): string {
  return str.charAt(0).toLowerCase() + str.slice(1);
}

function convertKeysToCamelCase(input: any): any {
  if (Array.isArray(input)) {
    return input.map((item) => convertKeysToCamelCase(item));
  } else if (input !== null && typeof input === "object") {
    const newObj: { [key: string]: any } = {};
    for (const key in input) {
      if (Object.prototype.hasOwnProperty.call(input, key)) {
        let newKey = key;
        if (key.charAt(0) === key.charAt(0).toUpperCase()) {
          newKey = lowerFirstChar(key);
        }
        newObj[newKey] = convertKeysToCamelCase(input[key]);
      }
    }
    return newObj;
  }
  return input;
}

export default async function (
  response: Response,
  request: ZuploRequest,
  context: ZuploContext,
) {
  // Get the outgoing body as an Object
  const obj = await response.json();

  // Modify the object as required
  const newBody = convertKeysToCamelCase(obj);

  // Stringify the object
  const body = JSON.stringify(newBody);

  // Return a new response with the new body
  return new Response(body, request);
}

Save your changes and test out the API!

Awesome - we our data is now in a standardized format.

Step 3: Managing Authentication with Subkeys

One issue you will run into with using the OMDB API is that the service issues you a single API key which cannot be rolled or revoked easily. That means if this key gets leaked, you will be on the hook for all the associated charges until it gets revoked. Revoking is no easy task as well - you might have several internal systems and public applications relying on this key.

Lucky for you, this is one of the main benefits of using an API integration platform - we can take advantage of our proxy to decouple our API's auth from the external APIs. The method I am going to use is called Subaccount API Keys. A Subaccount API Key (subkey) is a virtual API key that lets you control access to an external API at a finer-grain level. Subkeys are translated to the 3rd party API key at the API integration platform layer. The main benefit of using subkeys is that you can create as many as you want (ex. for each org at your company) and the impact of revoking this key is limited to the scope of use of that individual key, not to the whole org. Additionally, revoking the key does not require you to interact with the 3rd party API vendor.

Let's create a subkey system using Zuplo. Click the Services tab and then the Configure button under the API Key Service.

Now click Create Consumer and create your first API Key consumer, with your email as the manager.

We've successfully create an API consumer with an API key that can be used to access our API. Click the copy button next to the key and head back to the Code tab > routes.oas.json > your Movie Data route. Add the API Key Authentication policy, and then save.

If you try calling your API now, it you will get a 401 Unauthorized error. Your API new requires that you pass an Authorization header with the value Bearer <THE_KEY_YOU_COPIED>

But there's a problem - we are still passing the apikey parameter! Let's build a translation layer from our Zuplo Subkeys to our OMDB API key. First, we must securely store our OMDB API key. Head over to Settings > Environment Variables and click Add variable. Save your OMDB API key as a secret.

Now head back to routes.oas.json and your Movie Data route. And add the Set Query Params inbound policy with the following configuration

the $env(OMDB_API_KEY) portion will be replaced by the environment variable we set at runtime. Now save your project and try calling the API again without the apikey parameter.

Success - we can now issue subkeys to our users!

Step 4: Permissions and Monitoring

Let's say that this movie API is going to be used by various different teams at your company - but the expected traffic from their services/applications are very different. Given that the whole company shares a single rate limit on the external API, we should impose rate limits on our internal consumers. These should not be the same rate limits given the expected traffic differs. Instead, we can perform Dynamic Rate Limiting to set different rate limits per user.

To get started, let's go back to the Services tab, and Configure out API key service. Create a new Consumer, but this time include an applicationId in the metdata. We will use this property to determine which rate limit to apply.

We should ensure our old consumer also has this property. Click the ... menu and then Edit the consumer:

Click Save and head back to the Code tab. Add a new module with the Empty Module template. Name it determine-rate-limit.ts. Add the following code

import {
  CustomRateLimitDetails,
  ZuploRequest,
  ZuploContext,
} from "@zuplo/runtime";

export function rateLimitKey(
  request: ZuploRequest,
  context: ZuploContext,
  policyName: string,
): CustomRateLimitDetails | undefined {
  const applicationId = request.user.data.applicationId;
  context.log.info(
    `processing applicationId '${applicationId}' for rate-limit policy '${policyName}'`,
  );
  if (applicationId === "1234") {
    // Override timeWindowMinutes & requestsAllowed
    return {
      key: applicationId,
      requestsAllowed: 10,
      timeWindowMinutes: 1,
    };
  }
  return {
    key: applicationId,
    requestsAllowed: 5,
    timeWindowMinutes: 1,
  };
}

This code will rate limit your API based on the applicationId we specified earlier. This level of abstraction is quite powerful - we can now issue multiple subkeys to an organization, and as long as those keys share an applicationId they will share a higher rate limit of 10 requests per minute. Let's put this code to use, go to routes.oas.json > Movie Data. Open the inbound policy menu and select the Rate Limiting policy. Configure it to connect with the module you just wrote:

Save your code and lets try testing these rate limits. You should be able to call the API 5 times per minute with your first key, and 10 times per minute with your second key.

Once you're done testing out your rate limits, head over to the Analytics tab and click on the Top Users report. From here, you can monitor usage by your API key consumers.

This report can be helpful in several ways:

• From a developer's perspective, you can find exactly which consumers are using the movies API the most, and adjust their rate limits accordingly
• From an accounting perspective, you can determine what percentage of usage can be attributed to each consumer, and invoice those teams accordingly.
• Lastly, from a security perspective, you can detect irregularly high usage which might indicate an API key was leaked.

Step 5: Provide Developer Tools

I won't spend too much time on this section since tooling varies from org to org. One nice feature of building your API integration platform on an API gateway is that you can typically export an OpenAPI file and use that to generate documentation and tooling. Zuplo automatically generates documentation using Zudoku. Check out the documentation generated from our complete project.

If you'd like to standardize how your developers call into your integration platform API - consider using an Open Source OpenAPI client generator like OpenAPI Fetch.

Wrapping Up

Building a custom API integration platform can be a rewarding challenge - and can save your company millions of dollars long-term. It gives you the freedom to create exactly what you need without any additional bloat. Although I showcased using the Zuplo API gateway to build, deploy, and manage the integration platform - you can host an Open source gateway and API key service.