# Runtime typing with http-schemas [http-schemas](https://github.com/yortus/http-schemas) is a library for defining an API schema. It comes with utilities to help you use and enforce this schema in both your client and server codebases. ## What's wrong with the way we write web apps normally? - **API boundary is poorly defined in code**: Adding duplicate type definitions in client and server creates an opportunity to introduce bugs with every change, as these definitions can fall out of sync. It often relies on casting with `as` more than strict enforcement, which creates an opportunity for runtime errors. For example, axios may accept a generic to define the response payload type, but it doesn't enforce that the received data actually conforms to the type. In this case, Axios gives you a footgun to create runtime bugs with. - **Often no strict validation of the boundary**: i.e. if i send the wrong payload, such as a string instead of a number, or extra properties, or missing properties, will your app error? with a 500 or a 400? Or worse, carry on incorrectly? Will it give meaningful feedback? - **Expensive to write custom validation:** How much discipline does it take to write extensive validation for every endpoint? How long do you spend on it? Sure it's valuable, but it isn't a high leverage way to help the business deliver on its core value proposition, not compared to writing business logic. - **Poor developer experience:** How often have you read code to find out what the payload for a request or response should be? Even checking a swagger doc is less than ideal when coding against/for your own API. Especially since a swagger doc cannot guarantee correctness against the actual application. - **Types are contagious:** A poorly typed API boundary causes a proliferation of `unknown` or `any` types throughout your code base. - **Less resilient application code:** Good types help illuminate error cases, edge cases, and flaws in our implementation of business logic. A lack of these types and a proliferation of `any` types leads to code that is often missing code branches and thus buggy. ## How does http-schemas help? - **Reduce duplicate type definitions:** Define a well typed API schema once for your entire application:. - **Improved developer experience:** Use this API schema across client and server with fantastic type hinting and checking. Now a lot of runtime errors are instead raised at compile time, saving developers and testers time. - **Positive contagion:** Use the types from your API schema at every level of your codebase -- from route handlers, to domain logic, to your persistence layer. Now these types are a positive contagion proliferating through your code base. - **Reduce the temptation to use `any` and `unknown`:** The broken windows theory from clean code applies here. If your code has lots of well defined types, any new code submitted containing `any` is going to stand out, helping you maintain code quality. - **Ease refactoring:** Update a type at the API boundary and you are forced to update code in both the client and server, less you get a compile time warning. - **Validate the API boundary at runtime:** http-schemas validates your request payloads at runtime. It checks that they match the types specified in the API schema and will return a 400 when it receives a bad payload. This has a huge impact on the reliability of your application, as it is now much less likely to attempt to continue operation when it receives unexpected data, which could lead to bad application state or even corrupt data in your data store. - **Improve application observability:** Measure and report on status codes to see errors such as a bad client release show up as an increase in 400s. This will improve your time to detection, mean time to recovery, and change failure rate as it shines a light on any accidental breaking changes to your API schema. For example, we once released a breaking change to a trivial API in our application. We knew it would impact customers only very mildly, and when we did release it, we immediately saw a huge spike in error rates as users with the old client were hitting the new endpoint. This spike asymptotically returned to a nominal level as users refreshed their browsers. If this had been an unexpected change to a critical endpoint, the resulting PagerDuty alarms would have allowed us to respond very quickly. If it had been a vanilla express application, we might never have noticed. ## Creating an application with http-schemas I built a simple web app using http-schemas to demonstrate its value. I've attempted to avoid an overly contrived example, so I built an application that allows users to create polls where other users can vote on and contribute to choices. I only allowed myself one small concession -- to save time and simplify re-use by others, it uses an in-memory data store for its persistence layer! * **View the app:** [https://http-schemas-demo.antman-does-software.com](https://http-schemas-demo.antman-does-software.com) * **View the code:** [https://github.com/Antman261/http-schemas-webserver-demo](https://github.com/Antman261/http-schemas-webserver-demo) ### The folder structure The easiest way to set this up is to structure your application as a pseudo-monorepo. We do this by creating three main folders inside a thin top-level package. * `client` * `server` * `shared/http` ![Screen Shot 2021-06-02 at 1.06.47 pm.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1622610494021/ru09Xa_mz.png) #### Installing the API Schema in client and server The crucial step is to install the package defined in `shared/http` as a package inside `client` and `server`. NPM is able to install packages from the filesystem using symlinks. Provide the path to the package as an argument to `npm i` as follows: `npm i ../shared/http` NPM installs the package in your node modules via a symlink and adds an entry to your `package.json` as follows: ```json "dependencies": { "compression": "^1.7.4", "cors": "^2.8.5", "express": "^4.17.1", "helmet": "^4.6.0", "api-schema": "file:../shared/http", // <= this bit here "http-schemas": "^0.9.1", "morgan": "^1.10.0" } ``` We've set up `shared/http` as a typical package with a `src` directory containing TypeScript and a `lib` directory containing compiled javascript with generated type declarations and type maps. ```json "name": "api-schema", "version": "1.0.0", "description": "", "main": "lib/index", "types": "lib/index", ``` ### The shared API schema Inside `shared/http/src` we have three main files that define our API Schema: * `types.ts` * `schema.ts` * `index.ts` #### Types All of the domain types for our API are declared in `shared/http/src/types.ts`. The domain model here is relatively simple: Polls contain choices, and choices have a count of votes. However, we model this domain with two types, the domain type and the input type. Lets consider what the ideal domain type would be for a Poll: ```ts export const Poll = t.object({ text: t.string, type: PollTypes, choices: t.array(Choice), id: t.number, }); ``` This looks pretty good, but lets consider the ideal POST request to create this object: ``` POST /api/polls HTTP/1.1 { "text": "Is this poll good?", "type": "OPEN", "choices": [ "Yes", "No", "Maybe" ] } ``` We don't yet have an ID, nor do we need `choiceId` or a count of `votes` for the choices array — all we need is a string of the text for the choice. Instead we create a simplified input type: ```ts export const PollInput = t.object({ text: t.string, type: PollTypes, choices: t.array(ChoiceInput), }); ``` What about consistency between input types and domain types? We use the spread operator to reuse the relevant types in the domain type and override or add the delta: ```ts export const Poll = t.object({ ...PollInput.properties, choices: t.array(Choice), id: t.number, }); ``` #### Schema We declare the API schema in a single declarative file in `shared/http/src/schema.ts` ```ts import {createHttpRoute, createHttpSchema, t} from "http-schemas"; import {ChoiceInput, Poll, PollInput} from "./types"; export const ErrorBody = t.object({error: t.string}) export const pollsApiSchema = createHttpSchema([ createHttpRoute({ method: 'GET', path: '/polls', responseBody: t.object({ polls: t.array(Poll), }) }), createHttpRoute({ method: 'GET', path: '/polls/:id', paramNames: ['id'], responseBody: t.union(Poll, ErrorBody), }), createHttpRoute({ method: 'POST', path: '/polls', requestBody: PollInput, responseBody: Poll }), createHttpRoute({ method: 'POST', path: '/polls/:id/choices', paramNames: ['id'], requestBody: t.object({text: ChoiceInput}), responseBody: t.union(Poll, ErrorBody), }), createHttpRoute({ method: 'POST', path: '/polls/:id/choices/:choiceId/vote', paramNames: ['id', 'choiceId'], responseBody: t.union(Poll, ErrorBody), }) ]); ``` This defines the entire API schema in one place and returns a `pollsApiSchema` object we have exported for use later. In this file we don't declare the types that model our domain simply for code clarity sake. This file merges the domain types with HTTP types such as `ErrorBody` and declares their usage amongst a collection of routes. #### Index In `shared/http/src/index.ts` we collect and export only the types and objects we will need in our `server` and `client` code: ```ts import { TypeFromTypeInfo } from "http-schemas"; import {ErrorBody, pollsApiSchema} from "./schema"; import {Poll, PollInput, ChoiceInput, Choice} from "./types"; export type Poll = TypeFromTypeInfo; export type PollInput = TypeFromTypeInfo; export type ChoiceInput = TypeFromTypeInfo; export type Choice = TypeFromTypeInfo; export type ErrorBody = TypeFromTypeInfo; export {pollsApiSchema}; ``` We use the `TypeFromTypeInfo` generic to transform our runtime type definitions back into TypeScript type definitions. ### Server For the most part, everything in here is a typical express web server. There are only two things we need to do a little differently: * Create a decorated router for our API using the schema we defined earlier * Create enhanced route handlers for the routes in our API #### Creating a decorated router In this project we create the decorated API router in `server/src/index.ts`: ```ts const pollsApi = decorateExpressRouter({ schema: pollsApiSchema, onValidationError: validationErrorHandler, }); ``` We supply this function our schema and optionally a validation error handler, and it gives us back an express router. We can now mount this router and provide route handlers to the routes: ```ts app.use('/api', pollsApi); pollsApi.get('/polls', getPollsRouteHandler); pollsApi.post('/polls', postPollsRouteHandler); pollsApi.get('/polls/:id', getPollByIdRouteHandler); pollsApi.post('/polls/:id/choices', postChoiceRouteHandler); pollsApi.post('/polls/:id/choices/:choiceId/vote', postVoteRouteHandler); ``` The type checking on `.get` and `.post` will not allow me to provide handlers for routes I have not defined. It will also provide type hints and autocomplete suggestions for the available routes. #### Creating enhanced route handlers We import the route handlers from `server/src/routes/polls.ts` where we define them as follows: ```ts export const getPollsRouteHandler = createRequestHandler( pollsApiSchema, 'GET', '/polls', async (req, res) => { res.json({ polls: await pollsRepo.getPolls() }); } ); ``` We use the `createRequestHandler` to return a well typed route handler. It also strictly enforces the types we provide in the handling function for the last argument, based on the previous three. The enhanced request handler provides the following benefits: * It does not compile when we miss any properties in the object provided to `res.json`. * It removes any additional properties provided to `res.json`. This mitigates against leaking sensitive information. * It validates the incoming request payload, returning a 400 error if it does not comply with the schema for that route and HTTP verb combination. #### Types flow through to the domain layer We can import the types from our API schema for use throughout our code base, such as in domain code: ```ts import {Poll} from 'api-schema'; type ValidateChoiceOutcome = 'VALID' | 'INVALID_POLL_TYPE' | 'DUPLICATE_CHOICE'; export const validateChoiceForPoll = (choiceText: string, poll: Poll): ValidateChoiceOutcome => { if (poll.type === 'OPEN') { if (poll.choices.map(c => c.text).includes(choiceText)) { return 'DUPLICATE_CHOICE'; } return 'VALID'; } return 'INVALID_POLL_TYPE'; } ``` Now we can be certain our domain implementation is in sync with our API definition. #### Types flow through to the persistence layer We can also use the API types down in our persistence layer code: ```ts import { Choice, ChoiceInput, Poll, PollInput } from 'api-schema'; export type PollsRepo = { getPolls: () => Promise; getPollById: (id: number) => Promise; getChoicesByPollId: (pollId: number) => Promise; createPoll: (poll: PollInput) => Promise; createChoiceForPoll: (choice: ChoiceInput, pollId: number) => Promise; addVoteForChoice: (pollId: number, choiceId: number) => Promise; }; ``` When using an SQL backed persistence layer we may even provide mapping functions to and from our API types and our database IO types. ### Client Similar to the server, we can utilise our `api-schema` types throughout the client code. Additionally, we can use a client api package provided by `http-schemas` to: * Add type checking to our path parameters and json request bodies. * Add type information to the response data returned from our API. #### Creating an apiClient We initialise this api client in `src/apiClient.ts`: ```ts import {createHttpClient} from "http-schemas/client"; import {pollsApiSchema} from "api-schema"; const baseURL = 'http://localhost:8080/api'; export const apiClient = createHttpClient(pollsApiSchema, { baseURL }); ``` Now we can use it in hooks, such as this one that drives most of the client-side logic: ```ts export const usePolls = (apiClient: HttpClient): UsePollsReturn => { const [status, setStatus] = useState('READY'); const [polls, setPolls ] = useState([]); const refreshPolls = async () => { if (status === 'LOADING') { return; } setStatus('LOADING'); const result = await apiClient.get('/polls'); // <= Result is { polls: Poll[] } setPolls(result.polls); setStatus('READY'); } // REMAINDER TRUNCATED FOR BREVITY ``` #### Types flow through to logical components Using our API types in our logical components allows us to ensure that our client behaves predictably when interacting with the API, for example: ```ts const addPollSubmitHandler = async (pollInput: PollInput): Promise => { const result = await apiClient.post('/polls', {body: { …pollInput }}); addPoll(result); // Result here is type `Poll` } ``` #### Types flow through to presentational components Using these types in our presentational components keeps our presentation in sync with our API code, allowing us to ensure our data requirements are met for our presentation. For example here's a snippet of the `Poll.tsx` component: ```ts import * as React from "react"; import {Button, Card, CardBody, CardHeader, Table} from "reactstrap"; import {Poll} from "api-schema"; import {ChoiceRow} from "./ChoiceRow"; import {useState} from "react"; import {ChoiceForm} from "./ChoiceForm"; type Props = Poll & { createChoiceVoteClickHandler: (choiceId: number) => () => void; onSubmit: (text: string) => void; }; export const PollCard = ({id, text, type, choices, createChoiceVoteClickHandler, onSubmit}: Props) => { const [isFormOpen, setIsFormOpen] = useState(false); return ( {type === 'OPEN' ? ( ) : ( This poll is fixed. You cannot add choices. )} { // TRUNCATED FOR BREVITY } ``` ## Conclusion I hope this helps convey the power of a well typed, validated API. This is a solution unique to TypeScript, and as such, uniquely powerful.