Runtime typing with http-schemas

Runtime typing with http-schemas

Anthony Manning-Franklin's photo
Anthony Manning-Franklin
·

11 min read

http-schemas is a library for defining an API schema. It comes with utilities to help you use and enforce this schema in both your client and server codebases.

What's wrong with the way we write web apps normally?

  • API boundary is poorly defined in code: Adding duplicate type definitions in client and server creates an opportunity to introduce bugs with every change, as these definitions can fall out of sync. It often relies on casting with as more than strict enforcement, which creates an opportunity for runtime errors. For example, axios may accept a generic to define the response payload type, but it doesn't enforce that the received data actually conforms to the type. In this case, Axios gives you a footgun to create runtime bugs with.
  • Often no strict validation of the boundary: i.e. if i send the wrong payload, such as a string instead of a number, or extra properties, or missing properties, will your app error? with a 500 or a 400? Or worse, carry on incorrectly? Will it give meaningful feedback?
  • Expensive to write custom validation: How much discipline does it take to write extensive validation for every endpoint? How long do you spend on it? Sure it's valuable, but it isn't a high leverage way to help the business deliver on its core value proposition, not compared to writing business logic.
  • Poor developer experience: How often have you read code to find out what the payload for a request or response should be? Even checking a swagger doc is less than ideal when coding against/for your own API. Especially since a swagger doc cannot guarantee correctness against the actual application.
  • Types are contagious: A poorly typed API boundary causes a proliferation of unknown or any types throughout your code base.
  • Less resilient application code: Good types help illuminate error cases, edge cases, and flaws in our implementation of business logic. A lack of these types and a proliferation of any types leads to code that is often missing code branches and thus buggy.

How does http-schemas help?

  • Reduce duplicate type definitions: Define a well typed API schema once for your entire application:.
  • Improved developer experience: Use this API schema across client and server with fantastic type hinting and checking. Now a lot of runtime errors are instead raised at compile time, saving developers and testers time.
  • Positive contagion: Use the types from your API schema at every level of your codebase -- from route handlers, to domain logic, to your persistence layer. Now these types are a positive contagion proliferating through your code base.
  • Reduce the temptation to use any and unknown: The broken windows theory from clean code applies here. If your code has lots of well defined types, any new code submitted containing any is going to stand out, helping you maintain code quality.
  • Ease refactoring: Update a type at the API boundary and you are forced to update code in both the client and server, less you get a compile time warning.
  • Validate the API boundary at runtime: http-schemas validates your request payloads at runtime. It checks that they match the types specified in the API schema and will return a 400 when it receives a bad payload. This has a huge impact on the reliability of your application, as it is now much less likely to attempt to continue operation when it receives unexpected data, which could lead to bad application state or even corrupt data in your data store.
  • Improve application observability: Measure and report on status codes to see errors such as a bad client release show up as an increase in 400s. This will improve your time to detection, mean time to recovery, and change failure rate as it shines a light on any accidental breaking changes to your API schema.

For example, we once released a breaking change to a trivial API in our application. We knew it would impact customers only very mildly, and when we did release it, we immediately saw a huge spike in error rates as users with the old client were hitting the new endpoint. This spike asymptotically returned to a nominal level as users refreshed their browsers. If this had been an unexpected change to a critical endpoint, the resulting PagerDuty alarms would have allowed us to respond very quickly.

If it had been a vanilla express application, we might never have noticed.

Creating an application with http-schemas

I built a simple web app using http-schemas to demonstrate its value. I've attempted to avoid an overly contrived example, so I built an application that allows users to create polls where other users can vote on and contribute to choices. I only allowed myself one small concession -- to save time and simplify re-use by others, it uses an in-memory data store for its persistence layer!

The folder structure

The easiest way to set this up is to structure your application as a pseudo-monorepo. We do this by creating three main folders inside a thin top-level package.

  • client
  • server
  • shared/http

Screen Shot 2021-06-02 at 1.06.47 pm.png

Installing the API Schema in client and server

The crucial step is to install the package defined in shared/http as a package inside client and server. NPM is able to install packages from the filesystem using symlinks. Provide the path to the package as an argument to npm i as follows: npm i ../shared/http

NPM installs the package in your node modules via a symlink and adds an entry to your package.json as follows:

  "dependencies": {
    "compression": "^1.7.4",
    "cors": "^2.8.5",
    "express": "^4.17.1",
    "helmet": "^4.6.0",
    "api-schema": "file:../shared/http", // <= this bit here
    "http-schemas": "^0.9.1",
    "morgan": "^1.10.0"
  }

We've set up shared/http as a typical package with a src directory containing TypeScript and a lib directory containing compiled javascript with generated type declarations and type maps.

  "name": "api-schema",
  "version": "1.0.0",
  "description": "",
  "main": "lib/index",
  "types": "lib/index",

The shared API schema

Inside shared/http/src we have three main files that define our API Schema:

  • types.ts
  • schema.ts
  • index.ts

Types

All of the domain types for our API are declared in shared/http/src/types.ts. The domain model here is relatively simple: Polls contain choices, and choices have a count of votes. However, we model this domain with two types, the domain type and the input type.

Lets consider what the ideal domain type would be for a Poll:

export const Poll = t.object({
  text: t.string,
  type: PollTypes,
  choices: t.array(Choice),
  id: t.number,
});

This looks pretty good, but lets consider the ideal POST request to create this object:

POST /api/polls HTTP/1.1

{
  "text": "Is this poll good?",
  "type": "OPEN",
  "choices": [
    "Yes", "No", "Maybe"
  ]
}

We don't yet have an ID, nor do we need choiceId or a count of votes for the choices array — all we need is a string of the text for the choice.

Instead we create a simplified input type:

export const PollInput = t.object({
  text: t.string,
  type: PollTypes,
  choices: t.array(ChoiceInput),
});

What about consistency between input types and domain types? We use the spread operator to reuse the relevant types in the domain type and override or add the delta:

export const Poll = t.object({
  ...PollInput.properties,
  choices: t.array(Choice),
  id: t.number,
});

Schema

We declare the API schema in a single declarative file in shared/http/src/schema.ts

import {createHttpRoute, createHttpSchema, t} from "http-schemas";
import {ChoiceInput, Poll, PollInput} from "./types";

export const ErrorBody = t.object({error: t.string})

export const pollsApiSchema = createHttpSchema([
  createHttpRoute({
    method: 'GET',
    path: '/polls',
    responseBody: t.object({
      polls: t.array(Poll),
    })
  }),
  createHttpRoute({
    method: 'GET',
    path: '/polls/:id',
    paramNames: ['id'],
    responseBody: t.union(Poll, ErrorBody),
  }),
  createHttpRoute({
    method: 'POST',
    path: '/polls',
    requestBody: PollInput,
    responseBody: Poll
  }),
  createHttpRoute({
    method: 'POST',
    path: '/polls/:id/choices',
    paramNames: ['id'],
    requestBody: t.object({text: ChoiceInput}),
    responseBody: t.union(Poll, ErrorBody),
  }),
  createHttpRoute({
    method: 'POST',
    path: '/polls/:id/choices/:choiceId/vote',
    paramNames: ['id', 'choiceId'],
    responseBody: t.union(Poll, ErrorBody),
  })
]);

This defines the entire API schema in one place and returns a pollsApiSchema object we have exported for use later. In this file we don't declare the types that model our domain simply for code clarity sake. This file merges the domain types with HTTP types such as ErrorBody and declares their usage amongst a collection of routes.

Index

In shared/http/src/index.ts we collect and export only the types and objects we will need in our server and client code:

import { TypeFromTypeInfo } from "http-schemas";
import {ErrorBody, pollsApiSchema} from "./schema";
import {Poll, PollInput, ChoiceInput, Choice} from "./types";

export type Poll = TypeFromTypeInfo<typeof Poll>;
export type PollInput = TypeFromTypeInfo<typeof PollInput>;
export type ChoiceInput = TypeFromTypeInfo<typeof ChoiceInput>;
export type Choice = TypeFromTypeInfo<typeof Choice>;
export type ErrorBody = TypeFromTypeInfo<typeof ErrorBody>;

export {pollsApiSchema};

We use the TypeFromTypeInfo generic to transform our runtime type definitions back into TypeScript type definitions.

Server

For the most part, everything in here is a typical express web server. There are only two things we need to do a little differently:

  • Create a decorated router for our API using the schema we defined earlier
  • Create enhanced route handlers for the routes in our API

Creating a decorated router

In this project we create the decorated API router in server/src/index.ts:

const pollsApi = decorateExpressRouter({
  schema: pollsApiSchema,
  onValidationError: validationErrorHandler,
});

We supply this function our schema and optionally a validation error handler, and it gives us back an express router.

We can now mount this router and provide route handlers to the routes:

app.use('/api', pollsApi);
pollsApi.get('/polls', getPollsRouteHandler);
pollsApi.post('/polls', postPollsRouteHandler);
pollsApi.get('/polls/:id', getPollByIdRouteHandler);
pollsApi.post('/polls/:id/choices', postChoiceRouteHandler);
pollsApi.post('/polls/:id/choices/:choiceId/vote', postVoteRouteHandler);

The type checking on .get and .post will not allow me to provide handlers for routes I have not defined. It will also provide type hints and autocomplete suggestions for the available routes.

Creating enhanced route handlers

We import the route handlers from server/src/routes/polls.ts where we define them as follows:

export const getPollsRouteHandler = createRequestHandler(
  pollsApiSchema,
  'GET',
  '/polls',
  async (req, res) => {
    res.json({ polls: await pollsRepo.getPolls() });
  }
);

We use the createRequestHandler to return a well typed route handler. It also strictly enforces the types we provide in the handling function for the last argument, based on the previous three.

The enhanced request handler provides the following benefits:

  • It does not compile when we miss any properties in the object provided to res.json.
  • It removes any additional properties provided to res.json. This mitigates against leaking sensitive information.
  • It validates the incoming request payload, returning a 400 error if it does not comply with the schema for that route and HTTP verb combination.

Types flow through to the domain layer

We can import the types from our API schema for use throughout our code base, such as in domain code:

import {Poll} from 'api-schema';

type ValidateChoiceOutcome = 'VALID' | 'INVALID_POLL_TYPE' | 'DUPLICATE_CHOICE';

export const validateChoiceForPoll = (choiceText: string, poll: Poll): ValidateChoiceOutcome => {
  if (poll.type === 'OPEN') {
    if (poll.choices.map(c => c.text).includes(choiceText)) {
      return 'DUPLICATE_CHOICE';
    }
    return 'VALID';
  }
  return 'INVALID_POLL_TYPE';
}

Now we can be certain our domain implementation is in sync with our API definition.

Types flow through to the persistence layer

We can also use the API types down in our persistence layer code:

import { Choice, ChoiceInput, Poll, PollInput } from 'api-schema';

export type PollsRepo = {
  getPolls: () => Promise<Poll[]>;
  getPollById: (id: number) => Promise<Poll | undefined>;
  getChoicesByPollId: (pollId: number) => Promise<Choice[]>;
  createPoll: (poll: PollInput) => Promise<Poll>;
  createChoiceForPoll: (choice: ChoiceInput, pollId: number) => Promise<Choice>;
  addVoteForChoice: (pollId: number, choiceId: number) => Promise<Poll>;
};

When using an SQL backed persistence layer we may even provide mapping functions to and from our API types and our database IO types.

Client

Similar to the server, we can utilise our api-schema types throughout the client code. Additionally, we can use a client api package provided by http-schemas to:

  • Add type checking to our path parameters and json request bodies.
  • Add type information to the response data returned from our API.

Creating an apiClient

We initialise this api client in src/apiClient.ts:

import {createHttpClient} from "http-schemas/client";
import {pollsApiSchema} from "api-schema";

const baseURL = 'http://localhost:8080/api';

export const apiClient = createHttpClient(pollsApiSchema, { baseURL });

Now we can use it in hooks, such as this one that drives most of the client-side logic:

export const usePolls = (apiClient: HttpClient<typeof pollsApiSchema>): UsePollsReturn => {
  const [status, setStatus] = useState<PollsStatus>('READY');
  const [polls, setPolls ] = useState<Poll[]>([]);
  const refreshPolls = async () => {
    if (status === 'LOADING') {
      return;
    }
    setStatus('LOADING');
    const result = await apiClient.get('/polls'); // <= Result is { polls: Poll[] }
    setPolls(result.polls);
    setStatus('READY');
  }
  // REMAINDER TRUNCATED FOR BREVITY

Types flow through to logical components

Using our API types in our logical components allows us to ensure that our client behaves predictably when interacting with the API, for example:

  const addPollSubmitHandler = async (pollInput: PollInput): Promise<void> => {
    const result = await apiClient.post('/polls', {body: { …pollInput }});
    addPoll(result); // Result here is type `Poll`
  }

Types flow through to presentational components

Using these types in our presentational components keeps our presentation in sync with our API code, allowing us to ensure our data requirements are met for our presentation. For example here's a snippet of the Poll.tsx component:

import * as React from "react";
import {Button, Card, CardBody, CardHeader, Table} from "reactstrap";
import {Poll} from "api-schema";
import {ChoiceRow} from "./ChoiceRow";
import {useState} from "react";
import {ChoiceForm} from "./ChoiceForm";

type Props = Poll & {
  createChoiceVoteClickHandler: (choiceId: number) => () => void;
  onSubmit: (text: string) => void;
};

export const PollCard = ({id, text, type, choices, createChoiceVoteClickHandler, onSubmit}: Props) => {
  const [isFormOpen, setIsFormOpen] = useState<boolean>(false);
  return (
    <Card key={id} className='my-4'>
      <CardHeader>
        {type === 'OPEN' ? (
          <Button className='float-right mt-1'
                  color='primary'
                  onClick={() => setIsFormOpen(true)}>
            Add Choice
          </Button>
        ) : (
          <small className='float-right mt-4 text-muted'>This poll is fixed. You cannot add choices.</small>
        )}
        {
          // TRUNCATED FOR BREVITY
        }

Conclusion

I hope this helps convey the power of a well typed, validated API. This is a solution unique to TypeScript, and as such, uniquely powerful.

TypeScriptWeb DevelopmentAPIsJavaScriptnpm