Yet even that is too simplistic a mental model of coupling. It's not really about too much or too little, it's about appropriate coupling between layers while aligning the vector of change throughout the code with the vector of change throughout the business/application/package — whatever extrinsic forces motivate software updates. The customer journey, the customer types and categories, the organizational chart & design of a business, the third party vendors and partners, etc — the software architecture needs to align with all of it.

Alignment isn't a lack of coupling, it's the right kind of coupling in the right places. You can't build a skyscraper without cohesion between discrete pieces, whether that is steel through concrete, nuts and bolts, welding, mortar, etc. Without cohesion (coupling) you don't have a skyscraper, you have a pile of rubble. But if everything is coupled you also don't have a skyscraper, instead you have an incredibly expensive piece of abstract art that is utterly impermeable since none of the doors can be opened 😅

However, some approaches guarantee that two distributed systems will follow an acceptable state progression over time. That is, state changes progress linearly and deterministically. It either halts or continues but never enters an unrecoverable state, e.g., missing one message in a series of messages.

One of the most common sources of problems in applications is the misuse of HTTP in machine-to-machine communication. In this article, we will examine system design patterns that significantly improve the reliability of HTTP communication between systems.

The first step to solving these problems is understanding the distinction between a delivery guarantee provided by message producers and a processing guarantee provided by message consumers.

Guarantees come in three flavours:

• Atmost-once: 0 or 1

• Atleast-once: 1 or more

• Exactly-once: 1

Delivery guarantees are either "atmost-once" or "atleast-once".
A processing guarantee could be any one of the three, depending on the delivery guarantee provided by the producer.

Next, we need to understand that every HTTP request/response cycle is TWO messages over one connection:

1. The connection opens

2. The requester writes their message

3. The responder writes a reply

4. The connection closes

This means the requester can receive an acknowledgement of their message via the response, but critically, the responder does not know if the requester received and processed their reply.

In both scenarios depicted above, Service B does not receive a confirmation that the message was processed. At best, it may receive confirmation that the packets were received but not confirmation that Service A successfully processed the message.

This lack of confirmation means sending important information via an HTTP Response is unreliable. Suppose that HTTP Response contains the result of the requester's operation to change the responding system's state. In that case, the requester cannot know for certain if their internal representation of the responder's system state is accurate.

That was a wordful and woefully abstract, so let's look at a concrete example.

Implementing Exactly-Once Processing

In this scenario, our system must

• Subscribe a user to a subscription plan, provided they meet the requirements for a subscription, e.g. verified email, active account, etc.

• Update a 3rd party billing platform that manages the actual charges, invoicing, etc.

Let's add that in this scenario, the business logic dictates that a user can have zero, one, or many active subscriptions. As you can imagine, this means it is essential that every subscription in the third-party billing platform be recorded in our database.

The code below shows a naive implementation using a single HTTP Request+Response cycle.

const createSubscription = async (userId: number, plan: Plan) => {
  const user = await userDb.get(userId);
  const { outcome } = validateCreateSubscriptionOperation(user, plan);
  if (outcome === 'SUCCESS') {
    const result = await billingApi.post(
      `/plan/${plan.id}/subscribe`,
      { userId },
    );
    await dbConnectionPool.query(`
        INSERT INTO subscriptions    
            (status, external_id, user_id, plan_id)
            VALUES ($1, $2, $3, $4)
        `,
        ['ACTIVE', result.subscription.id, userId, plan.id]
    );
  }
}

What happens if our server is terminated after the HTTP request to billingApi, but before updating the database? Our system won't know the subscription was created, and it won't have the subscription id created by the billing system. Ultimately, neither billingApi nor our system will be able to automatically detect and correct the issue.

For the initial subscription request sent by our system, we are the producers of that message, and we implicitly provide atleast once delivery. If we don't get a response from billingApi, or we fail to process the response, then we (or the user) could try again until we do.

For the response, the billingApi is the producer and it only provides an atmost-once delivery guarantee.

This can cause a serious problem for our users. They might try to subscribe to a plan, receive an error, try again, and then be billed twice each month. When they look at their active subscriptions in our system, they would only see one subscription. If they cancel that subscription, they would continue being billed once a month. Even if our customer support team tried to help this user, they would not be able to see the extra subscription. Only someone with access to the third-party billing system, such as a member of finance, or the billing provider's customer support team, would be able to find the problem. However, even they might not know what they are looking for: they could have to check every subscription in both systems, looking for a subscription that only exists in the billing service.

So how do we solve this problem? Luckily, the billingApi service also provides webhooks with an atleast-once delivery guarantee. Following every HTTP request we make to billingApi, they will send a webhook, a HTTP request, to our service with the result of our previous request. Each webhook they send to our service contains a unique messageId. If our service does not acknowlege the webhook by responding 200 OK within 5 seconds, they will resend the webhook with the same messageId again until we do.

First, we need the createSubscription operation to reduce its scope to only dispatching the request. We will rename it dispatchSubscriptionRequest:

const dispatchSubscriptionRequest = async (userId: number, plan: Plan) => {
  const user = await userDb.get(userId);
  const result = validateDispatchSubscriptionRequestOperation(user, plan);
  if (result.outcome === 'SUCCESS') {
    await billingApi.post(
      `/plan/${plan.id}/subscribe`,
      { userId },
    );
  }
  return result;
}

Instead of receiving the subscription.id from the billing provider via the HTTP Response, the billing provider will send it via a webhook.

Here is a condensed example of how we could handle that:

api.post('webhooks/billing', async (req, res) => {
  const { 
    messageId, 
    subscription: { id, userId, planId } 
  } = req.body;
  try {
    const result = await dbConnectionPool.query(`
      INSERT INTO billing_inbox
      (message_id, subscription_id, user_id, plan_id),
      VALUES ($1, $2, $3, $4)`, 
      [messageId, id, userId, planId]
    );
    res.sendStatus(200);
    return;
  } catch (e) {
    const isExistingMessage = (
      e instanceof DatabaseError 
      && e.code === DbErrorCodes.UniqueViolation
    );
    if (isExistingMessage) {
      res.sendStatus(200);
      return;
    }
    res.sendStatus(400);
    return;
  }
});

Of course, this isn't production grade code, for the sake of brevity it crosses many levels of abstraction in one function. The critical facts here are that

• there is one database transaction per HTTP Request received by the webhook endpoint — a single statement is a single transaction.

• we use a unique constraint on our idempotency key, the message_id, to ensure we only save a message once, even if it is sent multiple times.

• we return 200 OK if we receive a message that we had previously saved to our inbox, allowing the webhook producer to resend a message until they successfully record our acknowledgment.

Now we need to process the messages in our inbox, guaranteeing we process each message exactly once. For example:

function initialiseBillingInboxProcessor() {
  (async () => {
    while (lifecycle.isOpen()) {
       const client = await dbConnectionPool.connect();
       try {
           await client.query('BEGIN');
           const { rows: [ message ] } = await client.query(`
             SELECT * FROM billing_inbox
             WHERE processed IS FALSE
             LIMIT 1
             FOR UPDATE SKIP LOCKED`
           );
           if (!message) {
             await wait(200);
             continue; 
           }
           await client.query(`
              INSERT INTO subscriptions    
                (status, external_id, user_id, plan_id)
                VALUES ($1, $2, $3, $4)
             `, [
             'ACTIVE', 
              message.subscription_id, 
              message.user_id, 
              message.plan_id
           ]);

           await client.query(`
             UPDATE billing_inbox
             SET processed = true
             WHERE message_id = $1`,
             [message.message_id],
           );
           await client.query('COMMIT');
       } catch (error) {
           logger.error(
             'Unexpected error processing billing message',
             { error },
           );
           await client.query('ROLLBACK');
       } finally {
           client.release();
       }
    }
  })();
}

What's happening in this code snippet? First, we define a function that is synchronous, but contains an immediately invoked async function expression.

function initialiseBillingInboxProcessor() {
  (async () => {
    while (lifecycle.isOpen()) {

    }
  })();
}

This prevents other engineers from accidentally awaiting our infinite loop since it won't resolve until the server begins a graceful shutdown.

Next we begin our transaction and take an update lock on the row we select, while excluding rows that have been locked by another transaction.

const client = await dbConnectionPool.connect();
try {
  await client.query('BEGIN');
  const { rows: [ message ] } = await client.query(`
    SELECT * FROM billing_inbox
    WHERE processed IS FALSE
    LIMIT 1
    FOR UPDATE SKIP LOCKED`
  );

This allows us to run the inbox processor across multiple servers simultaneously while preventing any two servers from ever processing the same message at the same time.

If the query returned no results, we wait 200ms before returning to the top of the while loop and polling for new messages again.

if (!message) {
  await wait(200);
  continue; 
}

Next we process the message by creating a new subscription record, marking the message as processed, and committing our transaction.

await client.query(`
    INSERT INTO subscriptions    
    (status, external_id, user_id, plan_id)
    VALUES ($1, $2, $3, $4)
  `, [
  'ACTIVE', 
  message.subscription_id, 
  message.user_id, 
  message.plan_id
]);

await client.query(`
  UPDATE billing_inbox
  SET processed = true
  WHERE message_id = $1`,
  [message.message_id],
);

await client.query('COMMIT');

So in summary, this transaction has three steps:

1. Retrieval: Get and lock an unprocessed message row, ensuring other servers don't process it.

2. Processing: Insert the new subscription record.

3. Commit: Mark the message as processed and commit the transaction.

Voila! Exactly-once processing for each message. If an error occurs during processing, e.g. the server is terminated unexpectedly or the database connection drops mid transaction, then the transaction will be rolled back and processing the message will be automatically retried. Of course, step two can include many more steps, as long as every database query is part of the transaction.

Now we are much more reliable message consumers because we have implemented an exactly once processing guarantee using the billing provider's webhooks, which provide an atleast once delivery guarantee. There is still a flaw in our system design, which we will look at next.

Implementing Atleast Once Delivery

We have learnt that when a Message Producer implements Atleast Once Delivery with idempotency keys we can then implement Exactly Once Processing as a Message Consumer. But what about the messages we produce?

To implement Atleast Once Delivery we will need to

• Create an outbox supporting multiple destinations and providing a unique idempotency key per message

• Add messages to the outbox instead of making API requests directly

• Process messages in the outbox by sending them to third parties until we receive a response

In our example service we currently send messages to the third party billing provider when our users call the new subscription endpoint. Currently, a user could make a request, receive an error, then resubmit their request. However, the first request error could be a false negative, meaning the billing provider received the request but something went wrong while sending the acknowledgement, either between their server and our server or between our server and our user.

Now that we get the billing provider's response via a webhook and implemented exactly once processing, the user will be able to see and cancel the duplicate subscription. However, we can make this process much more reliable using an outbox that will

• prevent the duplicate subscription from ever occurring

• ensure that every successful call to the new subscription endpoint results in a new subscription in the billing provider's system

• significantly increase throughput and reduce latency of the new subscription endpoint

Creating The Outbox

Let's start by defining the outbox table schema

CREATE TYPE "outbox_message_status" AS ENUM (
  'pending',
  'failed',
  'sent'
);

CREATE TABLE "outbox" (
  "id" UUID PRIMARY KEY DEFAULT gen_random_uuid(), -- WARNING: https://www.2ndquadrant.com/en/blog/sequential-uuid-generators/ 
  "status" "outbox_message_status" NOT NULL DEFAULT 'pending',
  "attempts" SMALLINT NOT NULL DEFAULT 0,
  "retryLimit" SMALLINT NOT NULL DEFAULT 0,
  "retryWaitPeriodMs" INT NOT NULL DEFAULT 1000,
  "createdAt" TIMESTAMP NOT NULL DEFAULT CLOCK_TIMESTAMP(),
  "updatedAt" TIMESTAMP,
  "destination" VARCHAR(255) NOT NULL,
  "payload" JSONB
);

CREATE INDEX outbox_created_at_pending_idx
  ON "outbox" ("status", "createdAt") WHERE "status" = 'pending';

For the sake of simplicity I have used a random UUID as both the primary key and the idempotency key, however random UUID primary keys impact performance. The alternatives are to either use a partially sequential UUID generator or use a BIGSERIAL primary key.

This schema is, I hope, fairly intuitive. The novel decisions are:

• All columns preceding payload are fixed length. I mostly ordered these columns for readability and intentionality, but I cannot help playing a little Column Tetris to optimise for storage. Please forgive me my vices 😅

• using CLOCK_TIMESTAMP instead of NOW for createdAt so that two rows inserted in one transaction do not share a timestamp.

• using a partial index since we will only query for pending rows.

• using "camelCase" for columns to make the database repository easier to write.

• not indexing destination because in most systems it won't be very selective (not many distinct values) AND there won't be many pending rows. Even a backlog of 1,000 pending messages isn't worth indexing.

Here is how we might implement a database repository for this table:

type OutboxRepo = {
  add: (msg: OutboxInsert) => Promise<Outbox['id']>;
  getPendingMessage: (destination?: OutboxDestination) => Promise<Outbox | undefined>;
  incrementAttempts: (id: Outbox['id']) => Promise<number>;
  setStatus: (id: Outbox['id'], status: Outbox['status']) => Promise<void>;
};

type Outbox = {
  id: string;
  status: 'pending' | 'failed' | 'sent';
  attempts: number;
  retryLimit: number;
  retryWaitPeriodMs: number;
  createdAt: Date;
  updatedAt: Date | null;
  destination: string;
  payload: Record<string, unknown>;
};

type OutboxInsert = Pick<Outbox, 'destination' | 'payload'> & {
  id?: string; // for use cases where the client generates the msg id
  retryLimit?: number;
  retryWaitPeriodMs?: number;
}

export type OutboxDestination = 'billingProvider' | 'exampleProvider';

export const outboxRepo: OutboxRepo = {
  add: async (msg) => {
    const { keys, values } = Object.entries(msg)
      .reduce<EntriesOf<OutboxInsert>>((obj, [key, value]) => {
        if (value !== undefined) {
          obj.keys.push(`"${key}"`);
          obj.values.push(value);
        }
        return obj;
      }, {keys: [], values: []});
    const client = getTransactionAwareClient();
    const queryText = `
      INSERT INTO "outbox" 
      (${keys.join(', ')})
      VALUES
      (${keys.map((_, i) => `$${i + 1}`)})
      RETURNING id
    `;
    const { rows: [row] } = await client.query<Pick<Outbox, 'id'>>(queryText, values);
    return row.id;
  },
  getPendingMessage: async (destination) => {
    const client = getTransactionAwareClient();
    const { rows } = await client.query<Outbox>(`
      SELECT * FROM "outbox"  
      WHERE status = 'pending'
        ${destination ? 'AND destination = $1' : ''}
      ORDER BY "createdAt"
      LIMIT 1
      FOR UPDATE SKIP LOCKED;
    `, destination ? [destination] : []);
    return rows.shift();
  },
  incrementAttempts: async (id) => {
    const { rows: [row] } = await pool.query<Pick<Outbox, 'attempts'>>(`
      UPDATE "outbox"  
      SET attempts = attempts + 1
      WHERE id = $1
      RETURNING attempts
    `, [id]);
    return row.attempts;
  },
  setStatus: async (id, status) => {
    const client = getTransactionAwareClient();
    await client.query(`
    UPDATE "outbox"  
    SET status = $1
    WHERE id = $2
    `, [status, id]);
  },
}

The code above includes a few common utilities I always use:

export type ValueOf<T> = T[keyof T];
export type EntriesOf<T> = { keys: string[], values: ValueOf<T>[] };

Plus, getTransactionAwareClient represents a way for database repos to automatically use a transaction-bound database connection if the caller is currently mid-transaction. Typically implemented using AsyncLocalStorage.

It's worth noting that incrementAttempts does not use getTransactionAwareClient. Whenever we call incrementAttempts, we want that change committed separately and immediately, regardless of the outcome of the caller's subsequent operations. Otherwise, during a failure, our attempt counter could be rolled back with the rest of the operation, defeating the purpose of the column.

I made destination an optional parameter when calling getPendingMessage to support use cases where we must run outbox processors dedicated to particular destinations. Why? Assuming the number of concurrent outbound requests we can make is limited, there are a few possible reasons:

• A destination has higher latency: We prevent our faster destinations being delayed by this slower one by running an outbox processor dedicated to the high latency destination.

• A destination is more important to the business: We prevent messages to one destination being queued behind others by running an outbox processor dedicated to the high priority destination.

Adding Messages To The Outbox

While implementing Exactly Once Processing, we created a dispatchSubscriptionRequest function that validated the user's subscription request before sending it to the third-party billing provider. Now that we have an outbox, we will update dispatchSubscriptionRequest to add the request to the outbox:

const enqueueSubscriptionRequest = async (userId: number, plan: Plan) => {
  const user = await userDb.get(userId);
  const result = validateEnqueueSubscriptionRequest(user, plan);
  if (result.outcome === 'SUCCESS') {
    await outboxRepo.add({
      destination: 'billingProvider',
      payload: {
        path: `/plan/${plan.id}/subscribe`,
        body: { userId },
      },
    });
  }
  return result;
}

What have we changed?

• We replaced the verb in the function name from dispatch to enqueue so callers know this function will not immediately make a request to the provider.

• We renamed validateEnqueueSubscriptionRequest to match, but it is otherwise unchanged.

• We swapped the HTTP request with a call to outboxRepo.add

There were no other changes. Importantly, we still return the result from validateEnqueueSubscriptionRequest so that users are informed if their request does not satisfy our business rules.

Processing The Outbox

We've created our outbox and added messages to it, now let's start processing them. There are a few things to keep in mind while reading the next code snippet:

• This is not production code: It crosses multiple layers of abstraction. In professional code we would breakup this large function to improve readability, but an article does not allow you to CMD+Click a function to jump to its definition and back.

• For concepts I explained earlier in the article, I have encapsulated them in functions. Instead of writing raw SQL we call the outboxRepo, instead of explicitly beginning, committing, and rolling back a transaction I use a withTransaction higher order function to commit on promise resolved and rollback if promise rejected.

• At 90 lines long it is our largest snippet so far, but don't worry, we will break it down piece by piece.

type Resolver = (v?: unknown) => void;

function initialiseOutboxProcessor() {
  const destinationApis: Record<OutboxDestination, Axios> = {
    billingProvider: axios.create({
      baseUrl: 'https://api.fictional-payment-platform.com/',
      transformRequest: [(data, headers) => {
        headers['Authorization'] = `Bearer ${getBillingAuthToken()}`;
        return data;
      }],
      httpsAgent: new https.Agent({
        keepAlive: true,
        noDelay: true,
        timeout: 30_000, // milliseconds
      }),
    }),
    exampleProvider: axios.create({
      baseUrl: 'https://fictional-service.com/api/',
      httpsAgent: new https.Agent({
        keepAlive: true,
        noDelay: true,
        timeout: 10_000, // milliseconds
      }),
    })
  } as const;

  (async () => {
    const concurrencyLimit = 20;
    const pollingIntervalMs = 200;
    let concurrentRequests = 0;

    while (lifecycle.isOpen()) {
      // Each execution of the while loop's code block is called a tick
      let resolveTickContinuationPromise: Resolver = () => {};
      const tickContinuationPromise = new Promise((resolver) => {
        resolveTickContinuationPromise = resolver;
      });

      // DO NOT AWAIT withTransaction!
      withTransaction(async () => {
        const isAtConcurrencyLimit = concurrentRequests >= concurrencyLimit;
        if (isAtConcurrencyLimit) {
          await wait(pollingIntervalMs);
          return;
        }

        const message = await outboxRepo.getPendingMessage();
        if (!message) {
          await wait(pollingIntervalMs);
          return;
        }

        const attempts = await outboxRepo.incrementAttempts(message.id);
        const isAtRetryLimit = attempts > message.retryLimit;
        if (isAtRetryLimit) {
          await outboxRepo.setStatus(message.id, 'failed');
          return;
        }

        concurrentRequests++;
        resolveTickContinuationPromise();
        // resolving here so the while loop can process another 
        // message concurrently
        const externalApiClient = destinationApis[message.destination];
        const { status } = await externalApiClient.post(
          message.payload.path,
          message.payload.body,
          {
            headers: {
              'Idempotency-Key': message.id
            }
          }
        );
        const wasAcknowledged = status >= 200 && status < 300;
        if (wasAcknowledged) {
          await outboxRepo.setStatus(message.id, 'sent');
        }
        /* DO NOT ADD A CATCH BLOCK!
           If an error is thrown, withTransaction MUST catch, log,
           and rollback.
         */
      }).finally(() => {
        concurrentRequests--;
        resolveTickContinuationPromise();
      );

      await tickContinuationPromise;
      /* 
        awaiting tickContinuationPromise instead of withTransaction 
        allows the outbox processor to send HTTP requests concurrently
        because tickContinuationPromise resolves before starting the 
        HTTP request while withTransaction resolves once the request 
        and transaction have finished
      */
    }
  })();
}

Let's break this down!

function initialiseOutboxProcessor() {
  const destinationApis: Record<OutboxDestination, Axios> = {
    billingProvider: axios.create({
      baseUrl: 'https://api.fictional-payment-platform.com/',
      transformRequest: [(data, headers) => {
        headers['Authorization'] = `Bearer ${getBillingAuthToken()}`;
        return data;
      }],
      httpsAgent: new https.Agent({
        keepAlive: true,
        noDelay: true,
        timeout: 30_000, // milliseconds
      }),
    }),
    exampleProvider: axios.create({
      baseUrl: 'https://fictional-service.com/api/',
      httpsAgent: new https.Agent({
        keepAlive: true,
        noDelay: true,
        timeout: 10_000, // milliseconds
      }),
    })
  } as const;

First we declare our destination API map. By telling typescript that the value assigned to destinationApis must satisfy : Record<OutboxDestination, Axios>, we guarantee that TypeScript will not compile if someone adds a new OutboxDestination but forgets to add an axios instance here.

In our Axios instance configurations we

• Set the baseUrl of the destination

• Add an Authorization header with a bearer token for the billing provider

• Set some sensible defaults for the HTTPS agent:

  • keepAlive: true keeps the socket open between the requests, like we talked about earlier. This improves performance if we make multiple requests to the destination within the timeout

  • noDelay: true turns off Nagle's Algorithm, improving performance.

  • timeout: number determines how long the socket will stay open after the last data packet. Tweaking the timeout for different destinations can be beneficial depending on their behaviour, for example, setting a timeout shorter than the target's keep-alive timeout can help prevent ECONNRESET errors.

Then we finish the object instantiation with as const to tell TypeScript that the properties in this object are readonly.

  (async () => {
    const concurrencyLimit = 20;
    const pollingIntervalMs = 200;
    let concurrentRequests = 0;

We begin another immediately invoked function expression like we did with our inbox, only this time we instantiate a few constants and a counter of running requests. Each concurrent message being sent consumes a database connection, but limiting the number of concurrent requests prevents exhaustion of the connection pool.

while (lifecycle.isOpen()) {
  // Each execution of the while loop's code block is called a tick
  let resolveTickContinuationPromise: Resolver = () => {};
  const tickContinuationPromise = new Promise((resolver) => {
    resolveTickContinuationPromise = resolver;
  });

We want our while loop to return to the start of its code block once we are about to start our HTTP request, but allow the processing of the result to continue in the background. This will allow us to send requests for multiple outbox messages concurrently.

However, our while loop is inside an async function, and the retrieval and processing of messages will occur within an anonymous function passed to a withTransaction higher order function, we need an alternative to the continue statement that allows us to continue the while loop from within the anonymous function but without preventing it from executing the remaining work. We solve this problem by inverting control: The while loop will wait until the anonymous function tells it that it is ready.

In the snippet above, we create the promise the while loop will wait for, tickContinuationPromise, and we assign the promise resolver to a variable that will be in the anonymous function's closure.

// DO NOT AWAIT withTransaction!
withTransaction(async () => {
  const isAtConcurrencyLimit = concurrentRequests >= concurrencyLimit;
  if (isAtConcurrencyLimit) {
    await wait(pollingIntervalMs);
    return;
  }

  const message = await outboxRepo.getPendingMessage();
  if (!message) {
    await wait(pollingIntervalMs);
    return;
  }

  const attempts = await outboxRepo.incrementAttempts(message.id);
  const isAtRetryLimit = attempts > message.retryLimit;
  if (isAtRetryLimit) {
     await outboxRepo.setStatus(message.id, 'failed');
     return;
  }

  concurrentRequests++;
  resolveTickContinuationPromise();
  // resolving here so the while loop can process another
  // message concurrently

In the snippet above we perform a few checks before sending the request. Just before we send the request we increment the concurrent requests counter and resolve the continuation promise. This allows the while loop to return to the top of its code block while this function continues running in the background.

  const externalApiClient = destinationApis[message.destination];
  const { status } = await externalApiClient.post(
    message.payload.path,
    message.payload.body,
    {
      headers: {
        'Idempotency-Key': message.id
      }
    }
  );

  const wasAcknowledged = status >= 200 && status < 300;
  if (wasAcknowledged) {
    await outboxRepo.setStatus(message.id, 'sent');
  }
}).finally(() => {
  concurrentRequests--;
  resolveTickContinuationPromise();
});

Here we grab one of the clients we prepared earlier. Crucially, we include the message ID as the Idempotency Key Header when we make a request. That way the recipient, in this case the billing system, can detect if we have sent this message previously.

Once we get a response, we check the status code, and if they returned 200 we mark the message as sent to prevent sending it again. Then inside the finally callback of the withTransaction promise we decrement the concurrent request counter and resolve the continuation promise in case an error or early return had occurred.

      await tickContinuationPromise;
    }
  })();
}

And now we wrap everything up! We await the continuation promise as the final statement inside the while loop. Then we close the loop, close the immediately invoked function expression, and close the initialiseOutboxProcessor function definition.

Now we have a process that will send messages from our outbox. Of course, this is just one approach of many and it has some trade-offs to be aware of. The biggest one, is that it maintains a open transaction with the database while making the request to the third-party. This might be acceptable if the third-party is a good citizen who sees their messages to an inbox before processing them.

However, if they attempt to process the message before responding with 200 OK, then our transaction will be held open while their system is processing our message. This makes our system extremely vulnerable to their influence. Database connections are not an infinite resource. In this case we would want to change the status of our message to sending and put a system in place to detect messages that have been stuck in the sending status for too long so that they can be resent again. That approach has a few more failure cases, thereby trading simplicity for performance. I decided to keep it simple for this article.

The outbox in this article is unordered, but sometimes we need to send messages in a particular order. In that scenario we can use a cursor to track our progress through an ordered set of messages.

Summary

Using HTTP, producers can implement an atleast-once delivery guarantee by sending a message repeatedly until the producer receives an acknowledgement from the consumer via a 200 OK response. (Along with a timeout while waiting for an ack)

These relatively simple messaging guarantees can be built upon to power our more robust network protocols. Yet, many services offering a REST API expect users to mutate data with a HTTP POST request and then receive updated state via the HTTP response. As we just learned, in this context, a HTTP Response is only useful for receiving acknowledgement of our request. It is an unreliable method for receiving the result of our request.

In HTTP, only the requester knows if the message they sent was delivered. If you provide a REST API where other systems can issue changes to your system, always provide webhooks with an atleast-once delivery guarantee and idempotency key.

1. Identifying the correct problem to solve

2. Getting the right specifications to solve the problem

3. Distributing a shared understanding of the specifications

4. Implementing the specifications

5. Verifying correct implementation

6. Repeating the above process in the context of refining the problem and specifications over time

The hardest problems are 1, 2, 3, and 6.

#1: Problem Discovery & Definition

Identifying the correct problem to solve is difficult because human psychology is hard. No matter the nature of the problem, someone only wants us to solve it because of how the person or people purchasing the solution expects that solution will make them feel.

I don’t care what problem you’re solving, at the end of the day you’re working on it because of how one or more people feel about it. “But Anthony, I’m writing software for NASA’s next Mars rover”, you say. Sorry, but we only want it because humans feel curious about the world and perhaps want some greater sense of mastery over an indifferent universe.

Every software project’s greatest risk is a lack of sufficient value. If what we write doesn’t meet a need sufficient for someone to part with their money, our software won’t be run and we probably won’t be paid to write it much longer.

#2: Solution Discovery & Design

The next most challenging aspect of a software project is actually figuring out how to solve the problem. This solution needs to be fit for purpose in multiple dimensions.

It needs to

• be understood and usable for its users (those pesky humans and their psychology again!)

• be cost-effective for the purchaser.

• be economically viable for the business providing it.

• meet the requirements of customer support teams.

• meet legal and regulatory requirements.

• be feasible to implement.

• meet the performance, reliability, and durability expectations of its users.

• meet the maintainability, changeability, testability, and observability requirements of the team working on it over the life of the service (human psychology strikes again!)

• meet the financial reporting requirements of the business.

• meet the business intelligence needs of the business.

All aspects of this multidimensional fit need to be considered across many different time scales. What race condition could occur if two actors issue a command within a few hundred milliseconds? How many requests per minute can it handle? How many months until the database needs to scale? How many years ago was this piece of code last changed?

As humans, we tend to solve problems by thinking in 2D-dimensional Euclidean space. If we can draw it on a screen or a page, we can expand our thought process beyond the limited working memory of our brains. And we can even share it with others. But even if we drew sequence diagrams, entity relationship diagrams, process maps, component diagrams, deployment diagrams, infrastructure diagrams, network diagrams, state machine diagrams, interaction overview diagrams, or even the C4 diagrams — system context, container, component, and code — we still would not be able to verify we have a correct and valuable solution until we run code for human use.

And this brings us to the heart of the third fundamental problem of building software.

#3: Shared Understanding

Not only is it effectively impossible to perfectly share the correct specification with a team for implementation, but we can’t know how correct our specification is until after we’ve built it!

So the specification will change while we’re implementing it, and every person implementing it will have a slightly different understanding of what the specification even is. Worse still, every person will have a slightly different understanding of the problem the specification is supposed to solve.

That means that while finding the correct problem and correct solution is a matter of human psychology, creating software is a problem of organisational psychology! Software is written, sold, maintained, and improved in a technosocial and psychosocial context.

For this reason, every software leader should endeavour to find ways to keep both their individual teams as small as possible, as well as their total headcount. Interpersonal complexity increases combinatorially.

But this is in direct tension with the fact that creating a defensible business requires building a large system to address the essential complexity of a valuable (difficult) problem. If the problem could be solved by one or two people, it likely already would be.

#4 & 5: Implementation and Verification

Funnily enough, problems 4 and 5 are the easiest parts of creating a software solution, yet they’re the focus of most of the literature on creating software. Most books, articles, YouTube videos, and online courses are about implementation. How come? Probably because this part of the problem is easily repeatable, consistent, verifiable, and locally reproducible. I can run a Kubernetes cluster on my local, but not the customer’s mind.

That's not to say it isn't important. The people working on this part, the engineers, must have a thorough understanding of the fundamentals of their craft. The stronger their fundamentals, the more capable they are of reducing a problem to its essence and finding the simplest possible solution.

#6: Software Over Time

Problem 6 is the problem of our solution existing and changing over time. It is the fact that this whole process is recursive. Our software solution begets new problems, requiring us to either change our existing solution or implement new ones. This is where software engineering emerges out of programming over time.

Given that most of the complexity, and all of the value, of a software solution exists after it is delivered, this aspect is vitally important. Yet, humans tend to be bad at predictions and timescales. There is an enormous wealth of information addressing this problem from a technological perspective with engineering solutions, e.g. Site Reliability Engineering, but this problem also consists, perhaps equally, of an anthropological component.

Every technological component forms a feedback loop with the people working on it. The people working on it are in a feedback loop with their adjacent teams, the wider business, and the customers. The impact of supposedly technological choices such as branch protection rules or choosing an infrastructure-as-code (IaC) tool is mostly anthropological over a sufficient timescale. That branch protection rule will become your pull request review culture, which will become your team dynamics and hierarchy. That IaC tool choice might determine whether your teams become siloed.

Who Solves These Problems?

When we look at the approach of the people solving these problems, a maturity model emerges. One I see professionals grow through over time. This model consists of three archetypes.

First is the coder, who focuses on problem #4, and a little on #5. Their impact is largely out of their hands, they simply implement the solutions asked of them.

Next is the software engineer, who focuses on problems 4, 5, and 6. They also start thinking about problems 2 and 3, and sometimes problem 1. This person is increasing their impact over time, rather than optimising for the short term.

Finally, we have the product engineer. This person works to solve all six problems. They’re not just leveraging their impact over time, but also in multiple social dimensions. They build trust within their organisation to improve the multidimensional fitness of their solutions by receiving candid feedback, as well as vulnerable expressions of the problems and concerns faced by their colleagues. They also know they need trust across the organisation because aspects of their solution will inevitably be incorrect. Without sufficient trust, they won't get the opportunity to improve their solution. Product engineers aren't just communicating within their group, but also with customers. They know the human context where their solution is used will determine its usefulness. They’re deadly focused on understanding the problem.

Ultimately, the lesson in all of this is that anyone working in software needs to aspire to better understand people. To become a better communicator, to grow their empathy, to become a better person. The technology is just the tool. It's not enough to love our tools, or even the solutions they create. We need to love the people we're building software with, and the people we're building software for.

1. Insert their booking information into our database

2. Send a HTTP request to our SMS service provider

Let's pretend that it is really important that users receive this message, and receive these appointment confirmation messages in the order they were booked.

If we do both actions when a customer sends a request to our appointment/book endpoint, what happens if one fails but the other succeeds? Do we really want to cancel appointments due to latency or temporary errors from an external service? Will we hold open our database transaction until the SMS provider confirms or denies our request? Of course not. We should avoid letting an external service interfere with our application's key value proposition. Holding open database transactions for external services will lead to connection pool exhaustion in our service, especially when the third party's response times increase. Yuck!

How can we ensure that a text message is sent, and retried on failure, after an appointment is booked? By using The Outbox Pattern. This pattern is especially useful when messages must be dispatched in the correct sequence, such as the payment processor example in the image below.

In this article, we will use the Outbox Pattern to ensure that an HTTP request to appointment/book only inserts a booking into our bookings table, while a background process sends an SMS message for each record in the table.

What are some of the implementation concerns our solution needs to address? They are:

• Ensuring at least one message is sent per row.

• Allowing this background process to run on multiple machines or processes without any risk of sending duplicate messages.

• Ensuring messages are dispatched in the order appointments were booked.

• Durability in terms of instance termination or process failure. I.e. messages aren't lost because a deployment occurred or a server crashed.

Let's get building!

Implementation

For our solution, we are going to use TypeScript, nodejs, pg, and Postgres with SKIP LOCKED. It will consist of:

• A cursor repository that manages the transactions, locking, updating, and unlocking of a cursor

• An outbox process that checks for new records to process

• A handler that runs our action for each record

This approach allows us to create multiple outboxes or reuse the code for other purposes such as read model populators in an Event Sourced system.

The Cursor Repository

For our cursor repository, we want to allow callers to

• Retrieve a cursor for an outbox if another process is not holding a lock on it

• Update the cursor value

• Unlock the cursor

We want the repository to support multiple processes, multiple cursors, and protect against a few common programmer mistakes. Let's start by defining the cursor table structure:

CREATE TABLE "cursor" (
     "process_name" TEXT NOT NULL PRIMARY KEY, 
     "position" BIGINT NOT NULL DEFAULT 0,
     "created_at" TIMESTAMP NOT NULL DEFAULT CLOCK_TIMESTAMP(),
     "updated_at" TIMESTAMP
);

CREATE TRIGGER cursor_update_trigger 
 BEFORE UPDATE ON "cursor" 
 FOR EACH ROW 
 EXECUTE FUNCTION set_updated_at();

Let's insert a row for our SMS cursor

INSERT INTO "cursor"
("process_name") VALUES ('booking_sms_outbox');

Now let's write our cursor repository. We're going to instantiate a single instance of a cursor repository per process. Our solution will ensure that we never have multiple instances, and that we do not reuse an instance across multiple cursors. This is important because our repository needs to manage transactions and locking, but we want to hide these details from the caller.

type ProcessName = 'booking_sms_outbox' | 'foobar_populator';

type CursorRepository = {
   getPositionWithLock(): Promise<number | undefined>;
   setPosition(position: number): Promise<void>;
   unlockCursor(): Promise<void>;
 };

const CURSOR_REPO_MAP: Record<string, CursorRepository> = {};

export async function getCursorRepository(name: ProcessName): Promise<CursorRepository> {
   if (CURSOR_REPO_MAP[name]) {
     return CURSOR_REPO_MAP[name];
   }

   const client = await connectionPool.connect();
   let isLocked = false;

   CURSOR_REPO_MAP[name] = {
     async getPositionWithLock() {
       await client.query(`BEGIN;`);

       const { rows } = await client.query(`
         SELECT position
         FROM "cursor"
         WHERE "process_name" = $1
         LIMIT 1
         FOR UPDATE SKIP LOCKED 
        `, [name]
       );

       const cursor = rows.shift();

       if (!cursor) {
         await client.query(`ROLLBACK;`);
         isLocked = false;
         return;
       }
       isLocked = true;

       return Number(cursor.position);
       // casting bigint to number means this will work until 2^53-1 which is 9,007,199,254,740,991 -- if we inserted 10 million records per day it would take more than 2 million years to exceed this number.
     },

     async setPosition(position) {
       await client.query(`
         UPDATE "cursor"
         SET "position" = $1
         WHERE "process_name" = $2
        `, [position, name]
       );
     },

     async unlockCursor() {
       if (isLocked) {
         await client.query(`COMMIT;`);
         isLocked = false;
       }
     },

     releaseClient() {
        client.release();
     }
   };

   return CURSOR_REPO_MAP[name];
 }

We've used memoisation to implement a functional singleton. This means the caller can safely call getCursorRepository('booking_sms_outbox'); multiple times but only instantiate one instance of the repository.

Our approach also prevents the caller from mixing up process names between calls. Additionally, we ensure that all calls for a cursor are made using an exclusive database connection. This is critical for our transaction and locking behaviour, just be aware that each of these processes will consume one connection per instance.

Now let's create the background process that uses this cursor repository.

Pro-tip: Set a sensible value for idle_in_transaction_session_timeout in the connection config used by the cursor. This will help prevent zombie connections from holding the lock indefinitely. You will also want to ensure you have process lifecylce hooks to unlock any locked cursors in the event of process exceptions, SIGTERM, and SIGINT.

Background Processor

We want to be able to start a background process that retrieves a cursor, gets new records, and handles each record sequentially. We're going to build a generic processor creation function, and then in the final step, we tie it all together with our handler.

type HasId = { id: number };

type Props<DbRecord extends HasId> = {
  processName: ProcessName;
  retrieveRecords: (position: number) => Promise<DbRecord>;
  processRecord: (record: Record) => Promise<void>;
}

const CURSOR_POLLING_SLEEP_MS = 200;

export async function initialiseCursorProcess<DbRecord extends HasId>(props: Props<DbRecord>) {
  const cursorRepo = await getCursorRepo(props.processName);
  try {
    const runTick = await createProcessTicker<DbRecord>(props);
    while (lifecycle.isOpen()) {
      const numProcessed = await runTick(); 
      if (numProcessed === 0) {
       await wait(CURSOR_POLLING_SLEEP_MS);
      }
    }
    cursorRepo.releaseClient();
  } catch (error) {
    logger.critical('Terminating server: cursor process error', {
      processName: props.processName,
      error,
    }); // we use structured logging
    cursorRepo.releaseClient();
    await lifecycle.close(1);
  }
}

async function createProcessTicker<DbRecord>({
  processName,
  retrieveRecords,
  processRecord,
}: Props) {
  const cursorRepo = await getCursorRepository(processName);
  lifecycle.on('close', async () => {
    await cursorRepo.unlockCursor();
  });
  return async (): Promise<number> => {
    const position = await cursorRepo.getPositionWithLock();
    if (typeof position === 'undefined') {
      return 0;
    }
    const records = await retrieveRecords(position);
    let processedRecords = 0;

    try {
      for (const record of records) {
        await processRecord(record);
        await cursorRepo.setPosition(record.id);
        processedRecords++;
      }
    } catch (error) {
      logger.error('Could not process record', {
        processName,
        record,
        error,
      });
    } finally {
      await cursorRepo.unlockCursor();
    }

    return processedRecords;
  }
}

A lot is going on here so let's break it down.

  const runTick = await createProcessTicker<DbRecord>(props);
  while (lifecycle.isOpen()) {
    const numProcessed = await runTick(); 
    if (numProcessed === 0) {
       await wait(100);
    }
  }
  cursorRepo.releaseClient();

First, we pass in our props and create the function that we want to run every tick. Then we start an infinite loop that will run until the node process receives a SIGTERM thanks to lifecycle.isOpen() using a lifecycle manager.

If nothing happened during each tick, we want to wait 100ms before running again. This prevents the process from flooding the nodejs event loop with callbacks in every tick. (wait is simply export const wait = async (ms: number) => new Promise(resolve => setTimeout(ms, resolve);).

After the while loop we call cursorRepo.releaseClient(); to close the connection to the database. This is required for await pool.end() to resolve during server shutdown.

What about that error handling?

  } catch (error) {
    logger.critical('Terminating server: cursor process error', {
      processName: props.processName,
      error,
    }); 
    cursorRepo.releaseClient();
    await lifecycle.close(1);
  }

In this case, something pretty drastic has happened, we want to log our errors and shut down the server as quickly as possible.

Moving on to the meat of our implementation, the process ticker and its creator function.

async function createProcessTicker<DbRecord>({
  processName,
  retrieveRecords,
  processRecord,
}: Props) {
  const cursorRepo = await getCursorRepository(processName);
  lifecycle.on('close', async () => {
    await cursorRepo.unlockCursor();
  });
  return async (): Promise<number> => {
    // trimmed

In our createProcessTicker function we retrieve our cursor repo and include it in the closure space of our tick function that we define and return on the next line. I've also registered an onClose handler with our lifecycle manager to ensure any cursors release their locks before node exits.

Let's look at what happens in each tick.

    const position = await cursorRepo.getPositionWithLock();
    if (typeof position === 'undefined') {
      return 0;
    }

If we cannot retrieve a position it means another nodejs process, perhaps on another server, has a lock on this cursor at present. In that case, we want to return 0 since we won't be processing any records and wait at least 100ms before trying again.

    const records = await retrieveRecords(position);
    let processedRecords = 0;

    try {
      for (const record of records) {
        await processRecord(record);
        await cursorRepo.setPosition(record.id);
        processedRecords++;
      }
    } catch (error) {
      logger.error('Could not process record', {
        processName,
        record,
        error,
      }); 
    } finally {
      await cursorRepo.unlockCursor();
    }

    return processedRecords;

Now that we have some records, we simply initialise our counter then attempt to process each one and update our cursor position each time. We must make sure we always unlock the cursor when finished.

In the next tick, our process will re-attempt to process the record. This is handy if it failed due to reasons such as network error, but we will want to trigger alerts if we receive too many 'Could not process record' errors in a given period. It's also a good idea to measure how far behind the latest record each cursor is.

Time to bring this all together for our appointment booking SMS dispatcher.

Appointment Booking SMS Dispatcher

All of our process scaffolding is in place, now to build our SMS dispatcher.

export function initBookingSmsDispatcher() {
  initialiseCursorProcess<Booking>({
    processName: 'booking_sms_outbox',
    retrieveRecords: bookingRepo.getNewBookingsSinceId,
    processRecord: async (booking: Booking) => {
      await smsProvider.sendBookingConfirmation({ booking });
    },
  });
}

The observant amongst you will notice the unawaited promise in initBookingSmsDispatcher. This is because we would be awaiting a while loop that only closes when the server is terminating. Accidentally awaiting a call to initialiseCursorProcess can cause the entire server to hang. So not only does initBookingSmsDispatcher nicely encapsulate all of our requirements for our SMS Dispatcher, but it also prevents us from accidentally deploying a goofy change failure bug.

For retrieveRecords we supply bookingRepo.getNewBookingsSinceId which is a repository function to retrieve all new rows since a given id. It might look something like this:

const INSERT_LATENCY_MS = 5;

async getNewBookingsSinceId(id: number): Promise<Booking[]> {
  const isZero = id === 0;
  const queryText = isZero ? `
    SELECT * FROM "bookings" ORDER BY "created_at" ASC
` : `
    SELECT * FROM "bookings"
    WHERE "created_at" > (
      SELECT "created_at" FROM "bookings"
      WHERE "id" = $1
      LIMIT 1
    ) AND "created_at" < (NOW() - INTERVAL '${INSERT_LATENCY_MS} milliseconds')
    ORDER BY "created_at" ASC
   `;
  const { rows } = await client.query(
    queryText, 
    isZero ? undefined : [ id ]
  );
  return rows.map(bookingRowToBooking);
}

This prevents the skipping of records that were inserted out of order. There is a race condition in Postgres where sequence numbers are granted to transactions concurrently. This means row 1234 might be inserted after row 1235. Looking up by created_at means we still get row 1234 if we look for events created after 1235.

We also avoid looking up records inserted within the last 5 milliseconds using AND "created_at" < (NOW() - INTERVAL '${INSERT_LATENCY_MS} milliseconds'). The reason for this is that when inserting many rows rapidly, there is no way to guarantee that each row will be available for lookup in the same order as their created_at timestamp.

For example, if I rapidly insert rows A, B, C, D, E, F and perform this lookup at the same time, Postgres could return rows A, C, F. But a moment later, the same query could return rows A, B, C, D, E, F. Why? Because rows B, D, and E may have a larger payload that takes longer to write to disk. Without this minor 5 millisecond latency, the cursor would jump to position F, and rows B, D, and E would never be processed.

Why 5 milliseconds? It depends on the size of your payloads. At SKUTOPIA, our largest payload is <20 kB, and we expect 5ms to work reliably up to ~2,000kB. For us, this is a very acceptable trade off.

Pro-tip: Consider a BRIN index for your created_at column, depending on the size of your table. Also consider using CLOCK_TIMESTAMP instead of NOW for your created_at column, in case the application ever inserts multiple rows in one statement or transaction.

When to Avoid the Outbox Pattern

The Outbox Pattern is not a replacement for a job system. This outbox process will halt if it encounters an unprocessable message. That's a design feature for some high-integrity use cases, but a flaw in others. It also trades off throughput for guaranteed ordering. If you need to send a lot of messages concurrently as fast as possible with no consequence in terms of failure, then this pattern is a poor fit.

For this article's SMS example, I would personally use a job system like pg-boss, and push back on our fictional PM's request for sequential SMS messages. Customer communication rarely needs an outbox implementation, it's typically programmatic systems that have trouble receiving messages out of order.

Conclusion

Now that you have the foundations for any cursor-based process to progress through a series of database records, you can use it for any outboxes you might want to create. It also works fantastically with Event Sourced systems, which is where we use it at SKUTOPIA. Each of our domain services uses event sourcing for their application state and publishes events to a PubSub topic for communication amongst services. We also use it to ensure usage events are sent to our payment processor. A critical use-case where at least once delivery and message ordering are both required.

Where might you use an outbox? Let me know in the comments.

Makes sense at first glance, right? Engineers write the code. They’re the final step in value creation for the business. But is that really where the company’s spend has the most leverage? We want to maximise our return on investment, and as we’re about to see, hiring more engineers is one of the most expensive ways to achieve that. Surprisingly, hiring more engineers can actually cause revenue to drop compared to staying the course with the current headcount!

A primer on value chains

To understand the following analysis, we need to understand value chains. A value chain is every step involved in creating a business’s product. In a manufacturing business, this might be

1. acquiring raw materials,
2. shipping them to a refinery,
3. refining them,
4. shipping the refined materials to a manufacturing plant, and then
5. manufacturing the final goods.

Each step in this process takes its inputs from the previous step, performs some operation on them, and produces outputs that will be the inputs for the following step. Often, upstream improvements in efficiency have a multiplicative effect on the outputs downstream.

So what is the value chain in a software company? It’s roughly

1. Research: Identifying valuable problems to solve
2. Designing a feasible, viable, valuable, and usable solution
3. Experimenting to de-risk that solution: Usability testing, tech spikes, wizard-of-oz tests, fake door tests.
4. Creating the solution via code

There’s a much simpler way I like to think about this: Every solution you build is a gamble; engineers let you place more bets, and product and design help you choose the bets with the greatest odds of winning.

The Software Value Chain

Let’s explore each of these steps. Every company does them to some extent, but the quality of execution at each step is not equally distributed amongst software product companies. In companies without a strong product culture, the early parts of the software value chain happen in a haphazard design-by-committee way from various stakeholders.

Research: Identifying valuable problems to solve

This is the responsibility of Product Managers. Many companies mistake Product Managers for Project Managers. They wind up paying lots of money for someone to spend their time writing tickets and micromanaging smart people. That is not the job of a Product Manager. Their primary responsibility is identifying problems worth solving and then de-risking those solutions.

That de-risking aspect is critical. Solutions are incredibly expensive to implement, and we have a limited number of bets we can place per year on features we hope will have a material impact on the business. But Product Managers cannot and should not do it alone. This brings us to the next step.

Designing a feasible, viable, valuable, and usable solution

Excellent product design doesn’t happen in a vacuum, and it isn’t a one-and-done process. If you think design is about aesthetics, you have completely misunderstood design. The goal of design is to marry the behaviour of the solution with the customer’s mental models of the problem, the task, and your application. Aesthetics are one tiny component of that incredibly complicated process. During the design phase, designers and product managers work together to address the four risks:


• Value risk: Will the solution solve the problem in a way that provides adequate value for the customers and the business?
• Viability risk: Can the business actually execute on this solution? Is the solution defensible and profitable? Do stakeholders in legal, marketing, customer support, and other functional departments have any objections to the solution?
• Usability risk: Will a large enough segment of our customer base be able to understand and use this feature well enough to unlock value?
• Feasibility risk: Are we sufficiently confident that engineering will be able to complete the feature in an acceptable timeframe? Will we be able to meet our non-functional requirements, such as performance?

To address these risks, we can use several discovery techniques, such as usability testing, empathy interviews, contextual enquiries, prototyping, tech spikes and proof of concepts, as well as working with stakeholders. The more experiments we can run, the more iterations design can create, the higher our confidence that the solution will provide value, AND the more value that solution will provide to customers and the business.

The critical thing to understand here is that every problem has multiple solutions, each of which has many variations itself. It is 100x cheaper to iterate through these solutions in the design phase than in the implementation phase.

Delivering the solution

The solution has been de-risked through several design and experiment cycles until we were confident it would achieve our desired business outcomes. Now, we’re ready to build it. Building and releasing software is hard. This is the most expensive and complicated part of the value chain.

Optimising The Software Value Chain

Now that we have thought about the entire value chain, let’s think about optimising it. We can’t do that without first considering how much we spend on each phase. For the sake of simplicity, I’m going to say every person gets paid $150,000 per annum, and I will only count the base salary cost. If we have

• 1x Product Manager
• 1x Product Designer
• 5x Software Engineers

We are spending

• $150,000 on Product Management
• $150,000 on Product Design
• $750,000 on Software Engineering

Which is $1,050,000 in total.

What does this spending get us? Spending on Product Management and Product Design increases our confidence in our solutions. Spending on Software Engineering increases the number of solutions we can ship each year. Or, to put it another way, spending on Engineering increases the number of bets we can place, and spending on Product increases the likelihood those bets will succeed.

However, there is a critical second-order effect here. As the software engineer-to-designer ratio tips further towards engineering, the workload on Product & Design increases, decreasing their ability to contribute to the solution's success. That is, spending more on engineering can actually cost you revenue due to a reduction in winning bets.

This is a lose-lose outcome for the business. They spend more to make less revenue. If you double the number of engineers, you don’t get twice as many bets per year because diminishing returns eat productivity pretty hard. But you do create twice as much work for Product, and double the salary spent on engineering, which already accounted for over 70% of expenditure.

Why does this happen? Simply, the Product & Design team have less time per feature for experimenting and iterating. Fewer tests and fewer iterations mean a lower chance of success.

Unfortunately, most companies don’t make time for Product & Design to do their discovery work at all. They simply build their best guess, which can work if you have plenty of cash to burn finding out what sticks, but in this climate, an ill-advised strategy.

So, what can we do to maximise our successful bets for the least expense? To maximise the company’s profit?

Well, we can’t increase the number of Product Managers per squad. That centralisation of context and stakeholder comms is critical to the role's success. Instead, we can double the number of designers. This is approximately only a 15% increase in expenditure, but it will almost double the capacity for discovery and research.

What impact does that have on profitability? Well, I built a spreadsheet to model this system! It shows that a 15% increase in spending results in over 50% more winning bets. It makes sense; you’ve effectively doubled the quality of the inputs going into engineering for a fraction of the price of the whole value chain.

Feel free to copy my spreadsheet and play with the numbers yourself. If you’re a weirdo like me, it’s actually a lot of fun.

Conclusion

It seems insane to me that the tech industry has been working on the assumption of one designer per squad for almost 15 years without questioning it. Yet, nearly every designer in tech I speak to is burned out and scrambling to get designs ready for engineering, let alone having time to run proper usability testing and discovery. Instead, the solution most tech companies choose seems to be to ask engineers to do more, and hire more of them. Not only is it addressing the wrong part of the value chain, but it makes the problem worse!

This also comes back to the theory of constraints and bottlenecks. It looks as though engineering is the bottleneck because that is where the most activity occurs. But if you change how you measure the system, from the number of bets placed to the number of bets won, then the bottleneck shifts upstream in the value chain. The bottleneck is product and design.

If we address this problem as an industry, not only might we make much better products and regain some credibility after a glutenous frenzy on VC funding that has come crashing down. We might also make tech companies a great place for product designers to work, reduce the pressure on engineers, and save people from needless burnout and the health risks that come with it.

When I joined SKUTOPIA in the middle of the pandemic as a fully remote Tech Lead, I had to figure out how to lead a team on the opposite side of Australia, over 3,000 km away. As the first remote team member, I began growing our Engineering department from 3 to 14. Along the way, I also added Product, UX, Data, and IT departments. Distributed across six cities and three countries, figuring out how to foster psychological safety and belonging on a remote team was not optional. It was do or die.

Culture: Safety and Belonging

Let’s take a minute to define the most important parts of team culture. Just two things are the primary predictors of both team success and staff retention. It isn’t average intelligence or training; it isn’t free food, foosball machines or ping-pong tables. It’s whether psychological safety and a sense of belonging are prevalent among the group. That’s it.

You will need more than those two things to succeed, but success is not probable without them.

What is psychological safety, and why is it so critical to performance? Psychological safety is whether you consider your social environment to be a hostile one or a safe one. In the workplace, this means we feel like we won’t be punished for speaking up, voicing concerns, suggesting ideas, making mistakes, or asking questions.

Similarly, a sense of belonging is the feeling that we are wanted and accepted in our group, now and into the future. That we can contribute to the group, be recognised for that contribution, and thus maintain our place in the group.

In short, you can’t have psychological safety without belonging, and you rarely find belonging without psychological safety. You must have both for your team to succeed.

Our brains are wired to live in groups; we die on our own. Our amygdala is constantly scanning our social interactions for signs of a threat to our status in the group. If we perceive a threat to our status, our prefrontal cortex is hijacked by the amygdala while our entire focus becomes reaffirming our status.

I can tell you from experience, a perceived threat to your continued place in a group sucks shiitake (that’s the totally scientific term for it). Ironically, this undermines our ability to perform, and in a modern workplace, further weakens our status in the group. When groups cannot offer their members psychological safety, their culture becomes pathological, meaning:

• People cease cooperating (someone else’s success is a threat to your position)
• Those who deliver bad news are punished for it
• People attempt to do as little as needed to remain a small target: “It’s not my responsibility!”
• Communication and cooperation between groups ceases, and territorialism takes its place
• Failures lead to scapegoating and punishment rather than inquiry and learning
• New and novel ideas are rejected, and innovation stops completely (too risky)

The bad news is that psychological safety and belonging are both temporary. It requires a constant stream of affirming signals, not just an absence of threats. What serves as an affirming signal, then? In short, anything that fosters a sense of belonging. But what are the ingredients of belonging?

First, is that our role in the group is likely to continue, called future orientation. These signals indicate that our inclusion in the group is expected to continue into the future. This is why career development plans can help increase retention, but they are a rather clunky belonging signal. We will consider others later in this article.

Second is energy. These signals occur when people show enthusiasm for us. If you’ve ever had a boss who avoids one on ones or checks his phone during them, you’ve unfortunately experienced the absence of energy.

Third is individualisation, which occurs when we are treated as unique and individual. This is commonly referred to as “feeling seen”. It happens when people have bothered to notice traits that are our own. It could be something as small as your boss asking about your kid’s dance performance over the weekend and remembering their name.

What about the affirming signals for psychological safety?

First is vulnerability. A signal that we’re in a safe environment is that others are comfortable being vulnerable. This means we see people freely admitting their mistakes and being supported rather than humiliated or punished. For leaders, it is especially important to model vulnerability.

Second is embracing bad news. In a safe environment, bad news flows freely. People don’t have quiet conversations about how to avoid bringing bad news to their boss; they go to their leaders for support.

Third is that everyone has a voice. In a safe environment, senior people aren’t expected to talk more than junior people. There aren’t rules governing who can ask questions or offer suggestions.

With all of these signals, frequency matters more than intensity.

Strong Signals in a Remote Workplace

Creating these signals in a remote environment is challenging but not impossible. It requires a little more intentionality; things that may have been intuitive in the office require conscious effort in a remote team. Although, for some of us neurodivergent folk, none of this was ever intuitive. Learning to consciously build the habit of using a particular communication style is a process we are very familiar with.

Let’s run through some of these habits and considerations that foster a great remote culture.

Speedy handovers: Leave your mic on

The human brain is wired for conversation. It is one of our most extraordinary evolutionary feats, and we take it for granted every day. While turn-taking time between cultures ranges from 200ms to 1,000ms, the processing power our brain devotes to the art of gracefully coordinating conversations is immense.

When we switch speaking roles in under a second, this includes the time it takes our brain to prepare our vocal muscles, change our expression, and begin to speak. Before that, it has to begin planning what we are going to say, which means it has already parsed and processed not only what the other person is saying and what it means but also what the other person is likely to say.

This means our brain has already predicted when the conversational handover will occur long before the person speaking finishes their sentence. As this happens, we begin giving the speaker cues that we are about to take a turn speaking. This can be small sub-vocalisations such as “ah” or “eh” (in all cultures, these are open-mouthed vowels as these take the least effort to form), but we also use facial expressions and body language.

Now consider this in the context of a video call. The participant cannot see all of your body language. The signals they do receive are delayed 300 to 800 milliseconds. If your microphone is muted, they miss the sub-vocal signals. Plus, without conscious effort, you will probably unmute your microphone when you should already be talking, further delaying handover.

All up, we’re adding about 2,000 milliseconds to a process that usually takes 200 milliseconds. Yikes! This is one of the reasons it can feel so exhausting to be in video calls all day.

We can do a few things to improve turn-taking in a video call, but the most effective one is the simplest: leave your microphone on.

There are typically two reasons we mute our microphones, either because we’re doing something else, like checking Slack, and don’t want people to hear us, or because we’re not using headphones and our speakers are causing some feedback.

Both of these issues are easy to solve. First, if you’re going to be in a video call, either be present or get the fluck out. I’m prone to distraction myself, so I make my video calls full-screen and mute notifications.

Secondly, if you will be making video calls from your workstation a lot, then invest in good equipment. Buy a decent pair of headphones or earphones and a microphone. Trust me, being understood is worth the effort.

And finally, if the acoustics in your home office are problematic, buy some acoustic treatment or add some large soft furniture. Acoustic treatment is about mass, airflow resistivity, and positioning. The goal is to absorb air particle movement at the point where particle movement is greatest and turn it into friction. This means you want to place acoustic treatment at a one-quarter wavelength from the wall at your lowest target frequency. For speaking, this is 1 kHz, so you want your acoustic treatment at least 8 cm (3 inches) off the wall if possible. I use acoustic insulation mounted in timber frames and covered in cloth, with spacers behind the frame.

Lastly, you want your “active listening” body language to work effectively in a video call. This means ensuring your camera is placed on the screen with the other participants. There’s nothing quite so disconcerting as talking to someone who looks away when you’re speaking because their camera is beside them. I, personally, try to keep the video as close to the top and centre of the screen as possible so that it is as close to eye contact as possible during a video call.

And it almost goes without saying, but include as much of your upper torso in the frame, from an eye-level camera, with good lighting. I’ve never felt terribly connected to disembodied eyebrows or looking up a hairy nostril.

Leaders, Highlight Your Fallibility

This is relevant in co-located and remote teams, but remote teams need a little more consideration regarding who has witnessed your fallibility. Showing vulnerability is the first step to forming trust; psychological safety cannot exist without trust. For leaders, this means highlighting your mistakes and making others aware of them when they otherwise wouldn’t be. This can be as simple as a Slack message in a team channel announcing that you made a wrong decision on X, and thanks to feedback from person Y, you are now going to do Z. I cannot overstate the effectiveness of broadcasting your failures publicly.

Embrace The Messenger

Getting bad news flowing in a co-located team is hard, and bad news is critical to success. It’s even harder in a remote team but simpler in other ways. The key is to set up multiple streams of bad news in different formats. A mixture of pull and push, weekly, fortnightly, bi-quarterly, one-on-one and group, anonymous and identifiable. Each of these different formats brings forth different types of bad news from different people. I’ll run you through each practice I use.

One on Ones

fortnightly, pull, individual, identifiable

This is the most common approach to seeking out bad news. Make sure you specifically ask your direct reports about the challenges they face, their concerns, the things they have learned, their opinions of their teammates, and their suggestions for improving the team. The people you need to listen to most are the ones who are most reluctant to bring up issues, especially regarding other team members. You need to give them a non-judgmental way of speaking about these topics with open-ended questions.

Retrospectives

fortnightly, pull, group, identifiable

This is almost the group equivalent of a one-on-one. While we don’t use Scrum anymore, we kept the fortnightly retrospectives because it is great for the team to discuss their issues and concerns. I try to stay quiet in these meetings. Our job as a leader in a retro isn’t to solve problems. It is to understand them. The team will come up with actions to solve the short-term problems. You need to pay attention to the systemic causes. Solving these issues goes beyond the retro.

My rule of thumb during a retro for anyone in a leadership position is that they must ask at least two questions before offering a solution for a given topic. And be very cautious offering any “solution” that isn’t a concrete action anyone on the team could accomplish before the next retro. I’ve seen too many aspiring leaders completely undermine themselves in retros by offering a so-called “solution” that really was a suggestion that the complainant themselves are the problem, and they should adjust their expectations. Not good, at all.

(If someone really does need to adjust their attitude, the retro is not the place for that conversation.)

Keep an eye on who is contributing the most to a retro, and who isn’t. It’s worth explicitly asking for input from quieter participants from time to time. One cause of an imbalanced retro is that the most frequent speaker is actually speaking on behalf of others. A large imbalance implies there is a trust issue in your team. One person is exceptionally safe, and quite likely, one person in particular is making other people feel threatened. People are afraid to speak up in front of the threat, so they channel their concerns through the safe person. This is a big red flag for an imminent issue that could otherwise fly under the radar on a remote team. Start investigating!

Team Health-Checks

bi-quarterly, pull, group, identifiable

Every six weeks, each of our squads runs a health check session. The metrics we use for the health checks are factors that the team decided were critical to their success. It is their opportunity to hold leadership (me) accountable for giving them what they need. I reiterate that at the start of every session, and then shut my mouth, letting someone else facilitate the session.

We also make a point that the score we record for each factor is the lowest score, not the mean or mode. We’re a team, and that means that any one person suffering is a loss to all of us. We’re all responsible for helping each other; we can’t let our teammates set themselves alight to keep the others warm.

Anonymous Surveys

ad-hoc, pull, group and one-on-one, anonymous

I will regularly put together anonymous surveys on hot topics for the team. This might include communication preferences, tooling state, psychological safety, etc. Anonymous surveys let the team be more candid than they otherwise might be. Be careful; if your anonymous surveys are a lot more negative than your retros or one on ones would suggest, it is likely that your team don’t feel safe speaking their minds.

If your team uses Google Workspaces, these are easy to run using Google Forms with a single response per person while maintaining anonymity.

I refer to surveys as a group and one-on-one channel because the quantitative data is group, while the qualitative data can be individualised. Include a mix of both in your surveys, through Likert-style questions and open-ended questions.

The Insider

bi-quarterly, pull, one on one, anonymous

I will regularly have a peer on the team conduct one-on-ones with the team members themselves. By speaking among equals rather than with their direct manager, they may feel safer raising other issues. This can be especially true regarding interpersonal conflicts. The time to resolve interpersonal conflicts is as soon as possible, but people generally want to avoid conflict and are particularly averse to going to their manager with “petty” issues.

I will then have my insider report back to me with the general zeitgeist of the team. This is about discovering the issues, not who said what. Confidentiality is essential for the team’s confidence and the ongoing success of the insider.

Guild Slack Channels

continuous, push, group, identifiable

We have guild channels for engineering, product, and design. In many instances, people will raise concerns they have about process, tooling, or workflow. We also have team channels. The hard part is getting the ball rolling with suggestions. If the team doesn’t have people who feel comfortable doing this already, then you need to ask questions and solicit feedback. After doing this enough times, people will begin raising their questions, concerns, and ideas.

The Missing Element

Astute readers will notice a missing combination: continuous, push, individual, and anonymous. This would be the digital equivalent of an anonymous suggestions box. I haven’t personally implemented this combination myself, but it could be feasible with a repeatable anonymous form and a regular Slackbot reminder to solicit contributions. If you try it, let me know how it goes!

Now, that’s enough on the topic of soliciting bad news. Let’s get back to our contributing factors for a healthy remote culture.

Over-the-top Gratitude

Showing gratitude and appreciation on a remote team can be as varied and fun as it is with a co-located team. But as with anything remote, it requires a little more conscious effort.

Everyone has different ways they like to receive praise. I ask new direct reports if they are comfortable receiving praise in public. Occasionally, someone will prefer not to be called out in front of the company for their awesome achievements!

There are a few things to consider when giving praise remotely, they are:

• Medium
• Tone
• Audience

The mediums being chat, email, video call, voice call, pre-recorded or live. Tone is a question of formal or informal, and audience is, well, the audience.

For example, celebrating someone's success in the company-wide announcements channel will need a different approach than in a video call with their squad, and so on. The guidelines roughly work out like this:

• The faster the medium, the less formal the tone — Slack is less formal than email
• The larger the audience, the more formal the tone — A Slack message in the company-wide channel might be equally formal as an email to the squad.

There’s another underlying factor, but company-wide praise of your direct reports can seem like a political game of optics rather than genuine praise. Context and tone are essential here. And remember earlier when I said individualisation was a signal for belonging? It’s harder to individualise your message when sending it to a wider audience.

For this reason, my favourite way to celebrate people’s success is in a Slack channel with their primary team. This allows me to go all out, using some ALL-CAPS, emoji bombs 🎉🤩🕺, an excessive number of exclamation marks (!!!), and some insider references or jokes.

This might sound over the top, too informal, too vulnerable. That is why it works. Text communication needs the extra oomph, but also, when you’re celebrating success, you need to be a person, not a boss. Done right, it’s a key moment for bonding the whole team because it creates a moment of shared pride.

I cannot stress enough how important it is for the people observing you giving praise to someone else, to feel joy themselves; To empathetically feel and share that sense of pride.

Imagine going to someone’s graduation ceremony. Is the healthy response beaming with pride, or seething with jealousy? Healthy teams experience the former when celebrating each other’s success. So many of these rituals we have as humans aren’t about celebrating the individual’s success as much as they are for the group to bond over a sense of shared pride.

In a healthy remote culture, this means your celebratory message should receive many emoji reactions and enthusiastic replies. A dry, formal congratulatory email is not likely to elicit a positive emotional response from its audience. But an over-the-top, emotion-laden, emoji-bombed message has a certain infectious joy!

As important as that asynchronous Slack message is, you need to back it up with another congratulatory moment in a video call with the team. This is critical; we need both the written form and the body language of a video call. Neither on their own is quite enough in a remote team. I also back it up with more congratulatory praise in our next one-on-one. There’s no such thing as too much gratitude.

If you can, sending gifts can also help your team seem real and connected. This could be tasty treats, plushies, vouchers, books (not work-related!), or whatever you think will be appreciated. If you’re lucky, the team will adopt this practice themselves, sending each other gifts to show appreciation. This hits a belonging double-whammy of energy AND individualisation.

As important as big moments of recognition are, it's important to have plenty of little moments of recognition too. I recommend setting up a #high-fives Slack channel where anyone can publicly show gratitude for people who look out for their teammates, whether small or big.

Manufacturing Collisions

Co-located workplaces lead to many accidental collisions. A collision is a random encounter that winds up being beneficial in terms of either information sharing, belonging, or psychological safety. It can be a conversation in the elevator, grabbing a coffee, or overhearing a conversation near your desk.

In a remote team, accidental collisions happen almost exclusively in chat. The rest we need to consciously plan for. I’ve tried each of the following ideas to varying degrees of success with different teams.

Coffee-run Slack Huddles

The morning coffee run can be a great bonding experience in the office, but what is its equivalent on a remote team? One method I’ve tried is a regular slack huddle. It has had varying success, but I think my approach with this technique fell into a negative network effect. I recommend trying with small groups in the same time zone. Even better if you can actually consume a hot beverage at the same time.

Donut: Random one-on-ones

Donut is a Slack app that pairs everyone up together randomly every week or so for a “virtual coffee”. Some people will love having time to get to know their colleagues and talk about things other than work. Others will loathe it.

Gather: Avatar Proximity Video Chat

Apps like Gather aim to mimic the spatial element of real-world conversations. Everyone has an avatar in a 2D world, but you can only hear and see the people standing near you. This allows people to drop in and out of conversations, break into groups, and swing by your desk.

I’ve used this for two purposes: mimicking the office and socialising. It works to varying degrees for either purpose, but again, it resonates with some people but not others.

Standups: A Double-Edged Sword

We dropped stand-ups in favour of asynchronous status updates. Then we returned to doing them three times a week in groups of less than six people. The larger the group, the less each person talks relative to the length of the meeting. On a video call, this is even worse.

We returned to stand-ups, not for productivity, but because we felt socially unmoored. It was possible to go an entire week without seeing the face of one of your squad mates.

Generally, I would say never use meetings for status updates, but this is one exception. Just keep the number of attendees low and the meeting very short! Anything bigger will not scale with the team.

Pick Up The (digital) Trash

Remote teams still have necessary but mundane tasks. As a leader, do them! After we split our Jira board into two projects, we wound up with both teams’ tickets in the new board, and all past and present tickets were reset to to-do and turned into tasks! I reviewed nearly 500 tickets individually and deleted roughly 400 of them. Just like cleaning up around the office, it shows that no one is above caring for the team and doing what needs to be done.

Leverage Threshold Moments

Onboarding on a remote team can be even more daunting than in person in some ways. That means you need to perfect every single element of the experience. If you have flashy promotional videos, include them in your onboarding docs. Set up plenty of one-on-ones with different team members. Don’t be afraid to use over-the-top language and raise the emotional intensity a little, be cheesy and fun, with some self-awareness of your awkwardness.

This is also an excellent moment for connecting their role to a sense of purpose. It can be challenging, but you must connect their work to the company’s mission. In a physical office, you can cover your walls with purpose and symbolism. In a remote team, you need videos, messages, and other constant reminders, rather than relying on ambient indicators of purpose.

Bring together groups of teammates for get-to-know-you video calls instead of team lunches. Prepare some icebreakers; the more awkward they are, the better. I find five people is the most you can have on a call before turn taking breaks down and people begin to disengage. You have to work harder to turn the abstract concept of the remote team into a concrete experience, but it is definitely achievable.

Bringing It All Together

No two teams are the same. In fact, someone once said to me, “teams are immutable”, meaning that every time someone leaves or joins a team, it’s really a whole new team. While analogies are never entirely true, this one holds some truth.

What has or hasn’t worked for me may work for you and your team. It might even work for my team at some point in the future when it hasn’t in the past. These ideas are worth trying at least once, if not twice.

That said, the fundamentals never change. Belonging will always require a steady stream of future orientation, energy, and individualisation. Psychological safety will always require vulnerability, open discussion of bad news, and everyone’s voices being heard.

I hope this article has given you some food for thought, and as always, I would love to hear what has worked for you in the comments. What are your techniques for fostering psychological safety and belonging in remote teams?

A user's request is said to be satisfied when it occurs within some T value, such as 400ms, and is successful, e.g. 2xx or 3xx status codes.

A tolerating request is successful in more than T, and less than 4T.

Frustrated requests exceed 4T or fail, e.g. 4xx and 5xx status codes.

So how can we build this measure in SumoLogic? Let's take a look

| json auto field=raw_log
| if(statusCode matches "2*", if(responseTime <= {{Apdex_Time}}, 1, 0), 0) as satisfied_counter
| if(statusCode matches "2*", if(responseTime < {{Apdex_Time}} * 4 && responseTime > {{Apdex_Time}}, 1, 0), 0) as tolerating_counter
| timeslice 150 buckets
| count as total_logs, sum(satisfied_counter) as satisfied, sum(tolerating_counter) as tolerating by _timeslice
| ((satisfied+tolerating/2)/total_logs)as apdex
| fields apdex, _timeslice

We use structured logging, so our logs are JSON formatted, but you could do this just as easily via a regex capture on apache style access logs to extract the status code and response time.

This simply creates a counter for satisfied and tolerating using nested if functions with the matches operator. The frustrated queries are everything not captured by these two counters, so count as total_logs gives us everything else we need, assuming our log source only contains access logs.

And that's it! You can even overlay the percentage tolerating, frustrated, and satisfied if you like:

| ((satisfied+tolerating/2)/total_logs)as apdex
| satisfied/total_logs as satisfied_pct
| tolerating/total_logs as tolerating_pct
| (total_logs - satisfied - tolerating)/total_logs as frustrated_pct
| fields apdex, _timeslice, satisfied_pct, tolerating_pct, frustrated_pct

In this article, we will consider what factors make an automated test suite great. We will bring together a lot of ideas from Software Engineering at Google. Whenever I refer to "Google" in this article, I am referring to the authors' depiction of Google's engineering practices in the book.

I will also translate some of Google's testing methodology to TypeScript. We will then see how it fits with Functional Domain-Driven Design (fDDD).

Good Test, Bad Test

Let's start by defining what a good or bad test suite is.

There are five dimensions we can use to consider the quality of our tests. These are brittleness, flakiness, speed, and readability. In the same sense that optimising outside the bottleneck is wasteful, the metric a team needs to focus on improving is whichever one is worst at any given time.

Brittleness (Durability)

A test is considered brittle when it breaks due to unrelated changes. If you have ever changed a small piece of code as part of a (supposedly) simple ticket and wound up failing scores of seemingly unrelated test cases, you have experienced the frustration of brittle tests.

Good tests should be updated less frequently, while bad tests sap productivity with needless changes. Measuring how often tests are changed can be an effective way of finding brittle tests.

Flakiness (Reliability)

A flaky test is a non-deterministic test. That is to say; sometimes it fails despite neither the code nor the test changing. When tests are flaky, our continuous integration and deployment pipelines suffer from unnecessary re-runs, our lead time for changes grows unnecessarily, and worst of all, engineers pay less attention to the tests.

We can measure flakiness in the number of times we can run a test suite and have it return the same result. That is to say that a test suite that, if I ran it twice, failed once and succeeded once, its flakiness would be 100%. The equation for flakiness percentage is failures / successes

Speed

Tests work best when they can provide real-time feedback as part of an engineer's workflow; The best tests can run in the IDE while the engineer is coding. However, as we will see later, some tests must sacrifice speed to interrogate the subject under test.

Fast tests improve the local development experience, increase the utilisation of tests amongst engineering teams, and improve the lead time to deployment.

Readability

Ultimately our tests need to document expected behaviour for other engineers to read. Since the best tests are updated the least, they will likely be read many more times than they are written. The best tests are clear, concise, and simple. Each test should only test one behaviour. The prerequisites, action, and expected result should be clearly expressed such that even someone unfamiliar with the code can understand the test.

A good litmus test for test cases can be checking if a non-technical stakeholder such as the Product Manager understands them.

Accuracy

It's great having durable, reliable, fast, and readable tests, but it is all for naught if the tests continue passing when the system's behaviour changes in a breaking way. However, accuracy doesn't become a focus for many teams because they struggle with the other dimensions of testing. Most teams only write example-based tests, which have the least accuracy, but other methods such as property-based testing and mutation testing can help ensure that our tests are rigorous and improve their accuracy.

Big Test, Little Test

While most people consider tests in terms of unit, integration, and end-to-end tests, Google has a more precise hierarchy of tests: Small, medium, and large. But what do these classifications mean? They might sound subjective, but they each have a specific objective definition.

Small Tests

Google's rules for small tests are:

• The test must run on one thread on one machine
• The test must not use sleep
• The test cannot perform I/O
• The test cannot make blocking/async calls

In TypeScript, these translate to:

• The test must run in one Node process
• The test must run synchronously: it cannot use async/await/promises
• It must run in a single tick of the event loop (this means no using setTimeout or setInterval for side-effects)
• The test cannot perform I/O (it cannot use synchronous file-system access, for example)

I would also add two more constraints:

• The test must not utilise system time
• The test must not utilise pseudo-random number generators or randomness of any kind

In practical terms, this rules out testing against a database, mocking asynchronous systems (a cause of much brittleness AND inaccuracy), and even starting an Express server in the test suite.

Medium Tests

Google's rules for medium tests are:

• The test may run on multiple threads or processes, but only on one machine
• The test may make blocking calls
• The test may call localhost, but not the network

In TypeScript, these translate to:

• The test may run an additional node process, may utilise a local database, etc
• The test may use async/await (Promises)
• The test may use multiple ticks of the event loop
• The test may perform I/O
• The test may call localhost, but not the network

In practical terms, you can now test against a local database, use a tool such as Supertest to test your Express server, and more.

Large Tests

For large tests, all the constraints are removed. An example of a large test would be using Cypress to run end-to-end tests against a staging deployment. As we know, networks are both laggy and unreliable, so these tests have the highest flakiness and slowness.

Comparing Test Sizes

With these constraints in mind, we can see that small tests are the best performing in all categories:

• Durability: Small tests necessitate a limited scope, so it will be less likely that they will inadvertently rely on related components
• Reliability: The constraints for small tests remove all opportunities for non-determinism to enter the test suite
• Speed: Small tests are CPU bound, consume the least resources and are therefore faster than their larger counterparts
• Readability: Small tests are generally simpler and thus easier to understand
• Accuracy: While small tests are not intrinsically more accurate on their own, their other attributes make it easier for engineers to write and run more of them, more examples, and even use approaches such as property-based testing.

For this reason, we strive to define our test pyramid, not in terms of unit or integration tests, but instead based on this size taxonomy:

As we move further up the pyramid, we lose out on all five factors of a good test. For this reason, we want to structure our code such that this distribution of test sizes is feasible. This code structure is where fDDD can help us!

Testing in Functional DDD

Writing Small tests can be highly challenging unless we take care to structure our code in specific ways. Luckily Functional DDD gives us precisely this structure. From here, I will continue assuming you are familiar with fDDD.

Small Tests: Derivers & Invariants

Since both Derivers and Invariants are Pure Functions, their tests automatically meet the criteria for smallness. This functional purity allows us to test our most valuable code quickly, easily, and to a high standard.

Medium Tests: Controllers

Controllers break the synchrony constraint and so cannot be deemed small tests. However, we can still benefit significantly by using the Partially Applied Controller pattern to reduce dependencies such as a local database, remove network requests, etc. We should strive to make even medium-sized tests as small as possible!

Another way of improving the balance between small and medium-sized tests is to remove tests for Controllers where they provide little value. Testing a Controller that simply flushes the derived result to the database on success is not a very valuable test.

If the test provides almost no value, then it may have a net-negative impact due to the increased brittleness and flakiness it could introduce. Having a solid foundation of small tests makes it easier to remove low-value medium-sized tests.

Large Tests

Large tests are not a concern of fDDD, but we will mention them for completeness. Large tests are most useful for testing configuration, emergent behaviour, and evolutionary architecture against fitness functions.

Types of large tests include:

• Automated UI tests on deployed applications
• Automated API tests on deployed applications
• Load testing on deployed applications
• Performance testing on deployed applications

The critical distinction here is that we are testing the entire system in the context of a deployed application. We could implement UI, API, visual regression, and performance testing as medium-sized tests, which would make them easy to run on pull-request, but they wouldn't be testing the system.

However, large tests are expensive, and most teams need to prioritise which non-functional requirements are critical to their applications.

Improving Readability

This article has spoken about flakiness, brittleness, accuracy, and speed, but we haven't talked much about readability yet. Let's discuss a few simple ways we can improve the readability of our tests.

Hermeticity

A test should contain everything it needs for setup and teardown while making no assumptions about the external environment, such as the state of the database.

For example, we once had a flaky test suite that was hard to track down. It was slightly more reliable in CI, but it was extremely flaky on local. It turned out that there were two pieces of code referencing a feature toggle in the database. One of these tests would change the state of the toggle, so the order that the tests were executed in could change the outcome of the test suite! But worse than that, people's local databases often weren't in a state compatible with the test.

In this case, the test's readability was poor because we defined part of the test's behaviour in an entirely separate system. A system that the test code didn't reference.

No logic

Tests should be so simple that they must not use control-flow statements such as if, for, and while. If your test is so complex it could have its own test, it isn't going to be effective, and it certainly isn't going to be clear to another developer when their change breaks your test.

Behaviour-based tests

Each individual test case must test one and only one behaviour. This rule is made clearest with an example:

it('updateUser', () => {
  const user: User = { email: 'foo@bar.com' };
  const result1 = updateUser(user, { name: 'bronson' });
  expect(result1).to.deep.eq({ email: 'foo@bar.com', name: 'bronson' });

  const result2 = updateUser(user, { email: 'bar@foo' });
  expect(result2).to.deep.eq({ error: 'invalid email address' });
});

The test above is trying to test an entire function, rather than a single behaviour of that function. We can improve the clarity of this test by splitting it into two behaviour based tests:

describe('updateUser', () => {
  it('should add a name property to an existing user', () => {
    const user: User = { email: 'foo@bar.com' };
    const result = updateUser(user, { name: 'bronson' });
    expect(result).to.deep.eq({ email: 'foo@bar.com', name: 'bronson' });
  });

  it('should return an error when supplied an invalid email address', () => {
    const user: User = { email: 'foo@bar.com' };
    const result = updateUser(user, { email: 'bar@foo' });
    expect(result).to.deep.eq({ error: 'invalid email address' });
  });
});

Now if I accidentally break the email validation logic, the test failure cause will be apparent even before I read the test code.

DAMP, not DRY

Google defines DAMP as promoting "Descriptive And Meaningful Phrases" -- a reverse-engineered acronym if I ever saw one. However, the principle is sound; tests benefit more from clarity than code reuse.

In the following example, we have created helper functions to setup test state:

it('should allow users to send friend request', () => {
  const users = createTestUsers(2);
  const { outcome } = sendFriendRequest({ from: users[0], to: users[1] });
  expect(outcome).to.eq('SUCCESS');
});

it('should not allow a banned user to send friend request', () => {
  const users = createTestUsers(2, true);
  const { outcome } = sendFriendRequest({ from: users[0], to: users[1] });
  expect(outcome).to.eq('BANNED_USER_CANNOT_SEND_REQUEST');
  // This test suite is failing
});

And then elsewhere in the file:

const createTestUsers = (numUsers: number, ...bannedUsers: boolean[]): User[] => {
  const users = Array.from({ length: numUsers }, (_, idx) => ({
    email: `user${idx+1}@test.co`,
    banned: bannedUsers[idx+1] ?? false,
  });
  return users;
};

The issue is that while reading the test cases, it is unclear what is happening, and is actually dependent on iteration logic inside the helper. Did you spot the bug? Consider instead:

it('should allow users to send friend request', () => {
  const user1: User = { email: 'user1@test.co' };
  const user2: User = { email: 'user2@test.co' };

  const { outcome } = sendFriendRequest({ from: user1, to: user2 });

  expect(outcome).to.eq('SUCCESS');
});

it('should not allow a banned user to send friend request', () => {
  const user1: User = { email: 'user1@test.co', banned: true };
  const user2: User = { email: 'user2@test.co' };

  const { outcome } = sendFriendRequest({ from: user1, to: user2 });

  expect(outcome).to.eq('BANNED_USER_CANNOT_SEND_REQUEST');
});

Here the difference between test cases has been made explicit within the test cases themselves.

Conclusion

Now that we've defined the five factors of a good test suite, Google's test size taxonomy, the benefits of fDDD in test quality, and a few tips for improving test readability, hopefully, you have a few ideas for improving your team's automated testing. Of course, there is a lot more that one could say about writing great tests, but I will leave that for another article.

Summary:

• You can measure the quality of a test suite in terms of brittleness, flakiness, speed, accuracy, and readability
• Google break their tests down by a taxonomy of sizes:
  • Small: Single machine, single-threaded, synchronous, with no I/O or sleep
  • Medium: Single machine, multi-threaded, asynchronous, localhost only
  • Large: No constraints
• fDDD provides a pattern for implementing a high number of small tests protecting the most valuable business rules
  • Use small tests to test derivers and invariants
  • Use medium tests to test controllers
  • Use partially applied controllers to keep medium-sized tests as small as possible
  • Use large tests to test non-functional system factors such as scalability
• Readability can be improved by:
  • Hermeticity: Remove environmental dependencies from tests
  • No logic or control flow in tests
  • Writing individual test cases to test an individual behaviour rather than functions
  • Writing explicit DAMP test cases where relevant information is repeated rather than abstracted

How can we translate the tactical patterns of DDD into a functional programming paradigm? This article will show you how I do it in TypeScript and the many simplifications and benefits. Let’s begin with a refresher on DDD.

Domain Driven Design

Domain-Driven Design focuses on business rules, business language, and business problems as the primary focus of software design. Instead of primarily designing software in architectural layers (e.g. database, data transfer objects, models, controllers, views), it focuses on building software with discrete domains or bounded contexts (e.g. billing, authentication, insurance sales, insurance claims, etc.)

DDD aligns our software architecture along the axis of most significant change; the business generates requests for changes to implement new business rules implemented in the domain. Focusing solely on software layers may produce greater code reuse, but reuse inhibits the ability to respond to changing business requirements. If both the Sales and Claims domain use an Insurance Policy entity, then a change on behalf of the sales team could inadvertently create a bug for the claims team. In DDD, we would make a separate SalesPolicy entity and a PolicyClaim entity, thus decoupling these two business domains in the codebase.

Domain-Driven Design splits its patterns into two categories: strategic patterns and tactical patterns. The strategic patterns are the easiest to reuse in any language, framework, and programming paradigm. They are universal aspects of software design.

However, the tactical patterns in DDD are tightly coupled to an object-oriented programming style and influenced by the capabilities of languages prevalent during DDD’s conception. These tactical patterns are where we will apply a functional approach to DDD and modify, remove, or create new tactical patterns.

Before we move into Functional implementations of tactical DDD patterns, let’s review the strategic patterns in DDD. Feel free to skip ahead if you are already familiar with these concepts.

Strategic DDD Patterns

These patterns govern the overall approach to your code and even your system architecture. They are relevant regardless of the language, framework, or paradigm. Since there is already a wealth of information about these patterns, we will only briefly cover some of the most crucial strategic patterns.

Bounded Context

A Bounded Context is a conceptual boundary in designing a system wherein the meaning of business terms is ubiquitous and consistent. For example, we may have the concept of a Policy entity in insurance. However, different teams in the business will have different interpretations of a policy, i.e. the sales team, the claims team, and the actuarial team will all assign the Policy entity different meanings.

By creating a bounded context in our design, we can effectively isolate each domain from the other. I use the term bounded context (context) and domain interchangeably throughout this article.

Context Map

Context mapping is how we identify, understand, and communicate the Contexts/Domains in our system. You will often identify common entities in context mapping — an important tenet of DDD is that we do not attempt to remove or share common entities between contexts. Instead, we allow each context to maintain its implementation and version of the data within its domain.

Anti-corruption Layer

In the above diagram, the "claims context" would implement an anti-corruption layer to adapt the incoming policy data into the format and structure meaningful to the "claims context". This pattern prevents details of the upstream entity from leaking into the downstream context and adds some protection against upstream changes.

Functional shortcomings in OOP DDD

Much of Classical DDD refers to methods of correctly allocating behaviour to the correct entity, aggregate root, domain service, or value object. Many of its rules aim to give DDD practitioners confidence in correctly distributing behaviour amongst various classes in their implementation.

However, Functional Domain Driven Design (fDDD) does not share these problems as data and behaviour are uni-directionally coupled rather than bi-directionally. That is to say that entities in Functional Programming cannot have their own behaviour but instead couples functions to data in the form of parameter types.

Any number of functions can use entities, but functions can only use entities matching their type signature.

Let's take stock of the Classical DDD concepts we have abandoned in fDDD:

• Aggregate Root: In fDDD, transactions are bound to the Controller

• Value Object: In fDDD, values are just literal values

• Service: In fDDD, this is most closely related to the Controller

• Domain Service

• Factories

Functional DDD Tactical Patterns

It’s time to consider the practical implementation of DDD’s Tactical Patterns within a Functional Programming paradigm. We have two layers within these patterns: a Functional Core and an Imperative Shell.

The Functional Core implements tactical patterns using only Pure Functions. This layer is where we implement the bulk of our business rules since Pure Functions provide us with the greatest degree of predictability, reliability, testability, and changeability.

In the Imperative Shell, we want to use tactical patterns that help us coordinate between our systems and our business rules. For example, code in the imperative shell may be responsible for retrieving necessary data from the database, checking our invariants, inserting the changes back into the database, and dispatching an email.

Let’s start from the top with our Functional implementation of entities, which lives across both Imperative Shell and Functional Core.

Entities

In fDDD, you will commonly implement Domain Entities as type definitions. For example, I might define a Policy in the Sales domain as follows:

type Policy = {
  id: number;
  customer: Customer;
  salesPerson: Staff;
  createdAt: Date;
}

While this approach to defining the entity is straightforward, it alludes to our implementation of various adaptors either in an anti-corruption layer or in our repositories, as we will cover soon.

A useful example might be our approach to parsing this entity when we receive it across the network. In that case, we may implement a parser function such as:

type PolicyParser = (rawData: unknown) => Policy | ParseError;

This parsing function would create a type-safe run-time implementation for bringing the Policy entity into our domain layer. Rather than implementing these parsers by hand, I often use a run-time typing package like zod.

Invariants: Functional Core

Invariants are Pure Functions with only one job: check that the provided entity meets a given business rule. In the case of an address entity, we might write an invariant that confirms that the provided phone number has an area code matching the address region.

export const validatePhoneMatchesCountry = (address: Address): boolean => {
  if (address.phoneNumber ?? false) {
    return false;
  }
  return getCountryFromAreacode(address.phoneNumber) === address.country;
}

As you can see, invariant functions can be tiny and very easy to test.

We primarily use Invariants by composing them inside Derivers. We may also, cautiously, create an invariant comprised of other Invariants if many Derivers share the exact composition of Invariants.

Derivers: Functional Core

Derivers are Pure Functions created to support a specific operation, such as createAddress or cancelSubscription. When we call a deriver, we must pass it all the information it requires to either derive the delta for our change or return an outcome indicating the cause for failure.

type UpgradeSubscriptionOutcome = UpgradeSucceeded 
  | AccountOverdrawn 
  | InvalidSubscriptionStatus;

export const deriveUpgradeSubscriptionOutcome = (
  newPlanLevel: Subscription['planLevel'],
  subscription: Subscription, 
  customer: Customer,
  upgradePriceMap: PriceMap
): UpgradeSubscriptionOutcome => {
  if (!validateCustomerBalance(customer)) {
    return {
      outcome: 'ACCOUNT_OVERDRAWN',
      payload: { balanceOwing: customer.balance },
    };
  }
  if (subscription.status !== 'ACTIVE') {
    return {
      outcome: 'INVALID_SUBSCRIPTION_STATUS',
      payload: { 
        currentStatus: subscription.status, 
        expectedStatus: 'ACTIVE' 
      },
    };
  }
  const prorataDays = calculateProrata(subscription);
  const upgradeFee = upgradePriceMap[subscription.planLevel][newPlanLevel];
  return {
    outcome: 'SUCCEEDED',
    payload: {
      prorataDays,
      upgradeFee,
    },
  };
}

In the Deriver above, we check two potential failure cases and then calculate the change required by the requested upgrade. Consider what we are not doing in this function:

• Not getting the customer's details from the database

• Not sending the customer a receipt via email

• Not calling a payment provider to charge their credit card

• Not saving the changes to the database

The core business rules for this operation have been condensed into a single function, enabling us to test every business rule. By keeping the scope of this function narrow, we can maintain its functional purity. This limited scope makes writing unit tests for this function less complex and makes the test itself more reliable. There are no opportunities for non-determinism, no database fixtures, and no stubs or mocks.

Controllers: Imperative Shell

We use Controllers to coordinate asynchronous parts of the system during an operation and call the Deriver. In the above example, the Controller would be responsible for each step we identified as out of scope for the Deriver.

export const upgradeSubscription = async (
  customerId: Customer['id'], 
  newPlanLevel: Subscription['planLevel']
): Promise<UpgradeSubscriptionOutcome> => {
  const [customer, subscription] = await Promise.all([
    customerRepo.getById(customerId),
    subscriptionRepo.getByCustomerId(customerId),
  ]);

  const { outcome, payload } = deriveUpgradeSubscriptionOutcome(
    newPlanLevel,
    subscription,
    customer,
    UPGRADE_PRICE_MAP,
  );

  switch (outcome) {
    case 'INVALID_SUBSCRIPTION_STATUS': {
      return { outcome, payload };
    }
    case 'ACCOUNT_OVERDRAWN': {
      await sendRepaymentReminder(customer, payload.balanceOwing);
      return { outcome, payload };
    }
    case 'SUCCEEDED': {
      const updatedSubscription = await subscriptionRepo.update({ 
        ...subscription,
        prorataDays: payload.prorataDays,
        planLevel: newPlanLevel,
      });
      const transactionId = await chargeCreditCard(customer.defaultCard);
      await sendInvoice(
        customer, 
        payload.upgradeFee, 
        updatedSubscription,
        transactionId
      );
      return {
        outcome,
        payload: {
          ...payload,
          transactionId,
        }
      }
    }
    default: {
      isNever(outcome);
      break;
    }
  }
}

As you can see in the above function, we are now dealing primarily with the asynchronous parts of the system. In these controller functions, we want as little business logic as possible. The example above is particularly complex; often, controllers only retrieve data from the database and save on success.

Notice, however, that the Controller does not make any business decisions. It only takes actions based on the result of the Deriver. It does not validate the subscriptions, and it does not check any rules.

However, the actions we take in certain circumstances, such as sending an email or charging a credit card, could be something we want to test. In that case, we can use the Partially Applied Controller pattern.

Testing via Partially Applied Controllers

If we want to make the previous example more testable, we can similarly use a partial function as we might use dependency injection in Object-Oriented programming. Let's take a look at an example:

export const createUpgradeSubscriptionController = (
  customerRepo: CustomerRepository,
  subscriptionRepo: SubscriptionRepository,
  sendRepaymentReminder: (customer: Customer, balanceOwing: number) => Promise<void>,
  chargeCreditCard: (card: CreditCard) => Promise<Transaction['id']>,
  sendInvoice: InvoiceSender,
) => async (
  customerId: Customer['id'], 
  newPlanLevel: Subscription['planLevel']
): Promise<UpgradeSubscriptionOutcome> => {
  /* Remaining implementation is identical to the previous example */
};

// In an adjacent test file
const fakeSendRepaymentReminder = sinon.fake.resolves();
const fakeChargeCreditCard = sinon.fake.resolves('1a2bc');
const fakeSendInvoice = sinon.fake.resolves();
const upgradeSubscriptionTestController = createUpgradeSubscriptionController(
  customerRepoInMemory,
  subscriptionRepoInMemory,
  fakeSendRepaymentReminder,
  fakeChargeCreditCard,
  fakeSendInvoice,
);

Now I can supply a test implementation of customerRepo & subscriptionRepo while also providing fakes for the other functions. Since this is a much more complex test, we don't want to repeat ourselves testing the Deriver's business rules. Instead, we only want to assert what functions the Controller calls for each of the Deriver's three possible scenarios.

Repositories: Imperative Shell

Repository patterns for retrieving entities on a CRUD basis are not solely a DDD pattern. Nor is it a significantly changed pattern between Functional and OO paradigms. For the sake of fDDD, I will only mention a few constraints to keep in mind for your repositories:

• Keep database concerns constrained entirely to your repository layer

• Parse, transform, or map all data types going into and out of your repository layer

A generic repository type could look like this:

type Repository<T> = {
   getMatching: (query: Query<T>) => Promise<T[]>;
   getById: (id: string) => Promise<T | undefined>;
   create: (item: Omit<T, 'id'>) => Promise<T>;
   update: (item: T) => Promise<T | undefined>;
   upsert: (item: T) => Promise<T>;
   delete: (id: string) => Promise<undefined | MissingEntityError>;
 };

type Query<T> = {
  [k: keyof T]: { operator: Operator, value: unknown },
}

Where the implementation of a create function may look like:

const subscriptionRepo: Repository<Subscription> = {
  create: async (subscription) => {
    const objKeys = Object.keys(subscription);
    const result = await pool.query<T>(
      `
      INSERT INTO "subscriptions"
      (`${objKeys.join(',')}`)
      VALUES (`${objKeys.map((_, i) => `$${i+1}`).join(',')}`)
      RETURNING *
      `,
      Object.values(subscription)
    );
    return result.rows.map(parseSubscriptionFromDb)[0];
  },
  // remaining repo functions omitted
};

In the above example, our repository maps from the database type back to the domain entity types using the parseSubscriptionFromDb function. This mapping helps us maintain a layer of separation between the database and our implementation. It also allows us to narrow the types further while throwing errors if the database returns unexpected data.

Software Architectural Context

Now that we have defined the patterns of our fDDD implementation, we need to consider its place in our overall software architecture. If we were to implement fDDD in a typical Express.js application, we could implement it as follows:

Our Route Handler for a POST request to /api/subscriptions may look like this:

export const handleSubscriptionPost: RequestHandler = async (req, res) => {
  const customerId = req.jwt.customerId;
  const { subscriptionLevel, paymentToken } = req.body;

  const { outcome, payload } = await createSubscription(
    customerId, 
    subscriptionLevel, 
    paymentToken
  );

  switch (outcome) {
    case 'PAYMENT_FAILED': {
      res.status(400);
      break;
    }
    case 'SUCCEEDED': {
      res.status(201);
      break;
    }
  }
  res.json({ outcome, payload });
};

All our route handler is responsible for is:

• Preparing data from HTTP requests for consumption by the Controller

• Mapping outcomes back to HTTP status codes

The route handler has no opinions on the business rules, and the domain code has no knowledge of content types or status codes.

When it comes to planning the structure of our codebase, there are primarily two dimensions we could choose as the basis for our files and folders:

• Application layers, e.g. database, route handlers, domain, services

• Operations, e.g. createSubscription, updateUser, etc

The former has the advantage that all of the code for a particular area is easy to find, but the trade-off is that implementing a change requires opening many folders. The latter has the advantage that all related code for any given change is likely to be nearby, but it is harder to find other code that may operate similarly or be affected by your change.

When we structure our codebase for a Domain Service, our preference is to keep all of the domain code as closely co-located as possible.

This preference means our top-level folder structure will be along the lines of:

However, we will then split it into "entities" and "operations" sub-folders inside the domain folder.

This structure increases the likelihood of reusing an entity's invariants in different Derivers for each operation.

Wrapping Up

Hopefully, this article has shown you how to implement Domain Driven Design within a Functional Programming paradigm and successfully make complex software more manageable. As you can see, fDDD simplifies many of the complicating factors in a classical DDD implementation. Here's a quick refresher on what we have learned:

• Entities are at minimum a type definition and may include a parser for run-time validation

• Invariants take an Entity as an input parameter, validate it against a simple business rule, and return a boolean

• Derivers take one or more Entities as an input parameter, validate it against operation-specific business rules, compose related Invariants, and return a discriminated union of potential outcomes

• Controllers coordinate asynchronous parts of the system, including databases, on behalf of Derivers

• The wider application accesses the Domain Layer strictly via Controllers

This is a problem because we use these functions not just for their behaviour but to communicate our code's intent. These functions reveal crucial details about the author's mental model of the code, the problem, and the solution.

In this article, I will attempt to couple a technical understanding of these functions to a semantic understanding in your mind. As a result, you should have a mental model of when to use these functions and why, plus what it means when you read code using them.

Array.prototype.map

Let's start with a simplified version of the official TypeScript definition for map and break it down. I have trimmed the type parameters down to the simplest and most common use case:

map<U>(
  callbackfn: (value: T) => U
) => U[];

Above, we have a map function that accepts a type argument U, a callback function, and returns an array (U[]).

The callback function takes a value of type T and returns a U. So it accepts a function that will take a value and transform it into another value.

Map calls this transformation function, the callback, for every element (T) in the array. The callback function transforms T into U, which map uses to transform each element, returning a new array of U[].

Now I have lied to you slightly in the above explanation. Transforming implies a mutative operation. However, map does not and should not mutate anything. Instead, it returns an entirely new array (U[]) where every element has a 1 to 1 relationship with each element of T[]. We call the function map because it maps one type onto another. However, thinking of it as a transformation helps us think about when to use it.

Use .map to "transform" an array of one thing into an array of something else.

I will rewrite the above type definition again in a slightly different format, using intention revealing names for the type parameters. Then we will look at some real-world examples.

Array<InputType>.map<OutputType>(
  callbackfn: (value: InputType) => OutputType
) => Array<OutputType>;

This is an identical type definition to the first one I showed you. I've included it because I know that seeing it like this will help some people grok it.

In the next scenario, we have an array of Person objects, and we want to map these into an array of ReactNodes:

type Person = {
  firstName: string;
  lastName: string;
  age: number;
};

type Props = { people: Person[] };

export const PeopleList: React.FC<Props> = ({people}) => (
  <ul>
    {people.map(
      (person) => (
        <li>{person.firstName} {person.lastName} | {person.age}</li>
      )
    )}
  </ul>
);

In this example, our map function transforms each Person object from the people array into a list item ReactNode. It's a simple use case and tightly coupled to the context we are using it -- the anonymous function we define for the map function isn't beneficial outside of the PeopleList component since it always returns a <li/> node.

Let's consider a use case where we might reuse our callback function in other contexts. In our next example, we are going to map an array of People into an array of full names:

type Person = {
  firstName: string;
  lastName: string;
  age: number;
};

function getFullnameFromPerson(person: Person): string {
  return `${person.firstName} ${person.lastName}`;
}

function listPeople(people: Person): string {
  return people.map(getFullnameFromPerson).join(', ');
}

In the above example listPeople passes the getFullnameFromPerson function to map, but we could conceivably use getFullnameFromPerson in other contexts too.

Let's try a more complicated example now, where we want to create a layer between our database implementation and our TypeScript implementation. There are several things map can help us with:

• In Postgres, we use snake_case column names, but in TypeScript and JavaScript, we use camelCase. We'll want to map between these types
• Enum types often come from another table; we'll map these too
• We'll also map our 64bit integer ids from strings to BigInt

Javascript Types:

type ArticleType = 'GENERAL' | 'REVIEW' | 'EDITORIAL';

type Article = {
  id: BigInt;
  articleType: ArticleType;
  author: string;
  title: string;
  urlSlug: string;
  content: string;
  publishedAt: Date;
}

Database Types:

type ArticleRow = {
  id: string; // Bigints come back from pg-node as a string
  article_type_id: number;
  author: string;
  title: string;
  url_slug: string;
  content: string;
  published_at: Date;
}

type ArticleTypeRow = {
  id: number; // small int
  type: string;
}

Mapping:

Now we have our cast of types, let's consider how we might map from these rather rough database types to these very usable javascript types. I'll write a function that selects the ten most recent articles, and we will use a series of maps to transform the query results.

const getRecentArticles = async (): Promise<Article[]> => {
  const result = await pool.query<ArticleRow>(`
    SELECT * FROM articles ORDER BY published_at LIMIT 10;
  `);
  return Promise.all(result.rows.map(getArticleFromArticleRow));
}

const getArticleFromArticleRow = async (row: ArticleRow): Promise<Article> => ({
  id: BigInt(row.id),
  articleType: await getArticleTypeById(row.article_type_id),
  author: row.author,
  title: row.title,
  urlSlug: row.url_slug,
  content: row.content,
  publishedAt: row.published_at,
});

/**
  This function memoises lookups to article type.
*/
const getArticleTypeById = (() => {
  let cache: Map<string, ArticleType>;
  return async (id: number): Promise<ArticleType> => {
    if (!cache) {
      const result = await pool.query<ArticleTypeRow>(`
        SELECT * FROM article_types
      `);
      cache = result.rows.reduce(
        (current, previous) => previous.set(current.id, current.type),
        new Map()
        // This gives us an object where we can lookup article types by id
      );
    }
    return cache.get(id);
})();

The first function in the above example, getRecentArticles, simply returns the ten most recently published articles.

If you're wondering why we use Promise.all in the return statement, it is because our mapping callback function getArticleFromArticleRow returns a Promise. That means that the result of result.rows.map(getArticleFromArticleRow) is an array of promises. Still, the caller of getRecentArticles expects a single Promise containing an array of Articles.

Promise.all takes an array of promises and maps them to an array of resolved values within a single promise, thus fulfilling our contract to the caller of getRecentArticles.

Why did we need getArticleFromArticleRow to be asynchronous (return a promise)? Because we look up the article type from the article type enum table in Postgres. You are correct in thinking that a Postgres query for every row in getRecentArticles is wasteful and slow. That is why getArticleTypeById actually caches the value locally.

Looking at the implementation, we use an immediately invoked function expression to create a cache value protected in the closure of the function definition for getArticleTypeById. This closure means no other part of the application can access cache, but all calls to getArticleTypeById utilise the same cache. It is effectively a functional singleton, providing a memoised lookup, so we only have to query the database for ArticleTypes once per node process.

As you can see, Array.prototype.map is a powerfully simple concept. You can build complex transformations with it, create asynchronous processes, build recursive solutions, and even chunk and distribute a map across multiple processes because it is monadic.

Now that we understand map let's see why forEach is different.

Array.prototype.forEach

Let's start with a simplified TypeScript definition of forEach like we did with map:

forEach(
  callbackfn: (value: T) => void
): void;

This looks pretty similar to map, except that the callback function doesn't return anything, and neither does forEach itself. Why might we want a function that returns nothing for each element of an array? Because we want to create side effects.

Use forEach for creating side-effects beyond the local scope

Let's understand side effects quickly by thinking about its opposite -- a pure function. Pure Functions always return the same value given the same inputs. The synchronous map functions in the previous example were pure, but the asynchronous functions accessing a database were impure because the return value could change between calls.

A function can also have a side effect when it affects part of the system beyond its input or output values. Let's move on to a concrete example.

If we wanted to send an email to every user in an array of users, we would use the forEach function if we do not care about the aggregate outcome of the email sending function. I'll give you an example, where our handleMeetingBooked function will email all attendees:

type Meeting = {
  name: string;
  attendees: User[];
  date: Date;
}

type User = {
  email: string;
  name: string;
}

const handleMeetingBooked = (meeting: Meeting): void => {
  const { attendees } = meeting;
  attendees.forEach((attendee) => {
    sendMeetingEmail(attendee, meeting);
  });
}

const sendMeetingEmail = (attendee: User, meeting: Meeting): void => {
  const emailBody = `Hi ${attendee.name},
    you have been invited to ${meeting.name} 
    on ${meeting.date} 
    with ${meeting.attendees.map(({name}) => name).join(', ')}`;
    sendEmail({ to: attendee.email, body: emailBody });
}

Notice how no part of handleMeetingBooked cares about what happens inside the callback function provided to attendees.forEach. We choose forEach instead of map because the behaviour is different and because it tells the readers of our code that the application shouldn't care about or depend upon the outcome of this email sending function.

Could I implement identical behaviour with map? Absolutely. Should I? Absolutely not.

Let's look at a typical forEach code smell -- mutating an external collection from within the callback provided to forEach.

// BAD CODE, DO NOT DO THIS
const getUsersNames = (users: User[]): string[] => {
  const names = [];
  users.forEach((user) => {
    names.push(user.name);
  });
  return names;
}

What's wrong with the above code? Aside from a few unnecessary lines, it lies to the reader. It tells the person reading this code that the callback function won't change anything within the function, and then it mutates one of the function's variables! It mightn't seem so bad in a small function like this, but this is a recipe for bugs in larger functions.

Code smell: Using .push inside forEach. Consider alternative e.g. .map or .filter

The correct implementation of the above would simply be:

const getUsersNames = (users: User[]): string[] => 
  users.map(({name}) => name);

Now that we have understood the difference between map and forEach, it is time to consider the role reduce plays in our code.

Array.prototype.reduce

Reduce is the array function that seems to cause the most confusion. Rather than starting with the type definition, I will offer you my mental model of what reduce is for, and then we can look at the code and see how it supports this way of thinking.

Use reduce to reduce a collection of elements to a single aggregate entity.

That is to say; I use reduce when I have many (an array) and want one (anything but an array). A typical example of this is summing the numbers in an array.

Let's take a look at a simplified version of the official type definition as we did before, this time keeping the goal of reducing down to a single aggregate entity in mind:

reduce<U>(
  callbackfn: (previousValue: U, currentValue: T) => U, 
  initialValue: U
): U;

The first thing to notice is that reduce returns a single U where map returned U[]. We can also see that we can supply an initialValue, but the type of this value must match the return type of the reduce function and that the callback function must also return this same type.

I'll write this again with intention revealing names:

Array<InputType>.reduce<OutputType>(
  callbackfn: (previousValue: OutputType, currentValue: InputType) => OutputType, 
  initialValue: OutputType
): OutputType;

We'll reconsider the summation example, equipped with our new mental model and expanded type definition fresh in our minds:

const total = [1, 2, 3, 4, 5].reduce(
  (previous, current) => previous + current,
  0
);
console.log(total); // 15

Above, we started with a collection of numbers and wound up with just one number. We reduced it down to its aggregate: the variable total with a value of 15.

How else could we use this? Perhaps we want to reduce an array of users down to a count of common names:

type User = {
  firstName: string;
  lastName: string;
}

type CommonNamesAggregate = Record<string, number>;

const countCommonNames = (users: User[]): CommonNamesAggregate => {
  return users.reduce<CommonNamesAggregate>(
    (aggregate, user) => {
      const currentCount = aggregate[user.firstName];
      if (currentCount) {
        aggregate[user.firstName]++;
      } else {
        aggregate[user.firstName] = 1;
      }
      return aggregate;
    },
    {} // We create our new CommonNamesAggregate here
  );

In this case, our new aggregate entity is an object where each key is a first name from users, and the value is a count of the frequency of that name in the users array.

Let's write another flavour of the same countCommonNames solution and see if it helps us grok reduce:

const countCommonNames = (users: User[]): CommonNamesAggregate => {
  const uniqueNames = new Set(users.map(({firstName}) => firstName));
  const commonNamesAggregate = [...uniqueNames].reduce(
    (obj, name) => {
      obj[name] = 0;
      return obj;
    },
    {}
  );

  return users.reduce<CommonNamesAggregate>(
    (aggregate, user) => {
      aggregate[user.firstName]++;
      return aggregate;
    },
    commonNamesAggregate
  );

In this implementation, we first create the aggregate object separately and initialise its values to 0 by reducing our set of unique names, saving us from checking if the name already exists in our final reducer.

Now let's revisit forEach regarding the above code example. I'll implement the same behaviour using forEach, and then explain why it is a code smell.

// BAD CODE, DO NOT DO THIS
const countCommonNames = (users: User[]): CommonNamesAggregate => {
  const uniqueNames = new Set(users.map(({firstName}) => firstName));
  const commonNamesAggregate = [...uniqueNames].reduce(
    (obj, name) => {
      obj[name] = 0;
      return obj;
    },
    {}
  );

  users.forEach(
    (user) => commonNamesAggregate[user.firstName]++
  );

  return commonNamesAggregate;
};

This might look simpler because it has fewer lines of code, but it gives the reader less information about the intent of the code. By using forEach, we tell the reader to hold onto their hats and read carefully because we're about to do something. The issue is that the only clues about what that something is, come from code that is mainly outside and before the forEach callback.

Code smell: Mutating a local variable from within a forEach.

Ultimately, forEach is the array function with the least semantic meaning and the least ability to reveal intentions. As such, forEach should be used as a last resort when there isn't a more suitable function or when you explicitly want to indicate that what occurs within the callback is not relevant to the adjacent code.

I also want to talk briefly about a code smell in reduce usage: returning an array from reduce.

Code smell: Returning an array from reduce. Consider map or filter instead.

People usually do this by mistake when they want to chain .filter and .map together, but instead, they do it in one function. The damage from this anti-pattern is that we mislead readers of our code entirely.

Conclusion

As programmers, we spend a lot of time working with collections of things, whether arrays, sets, maps, or objects. JavaScript comes with a fantastic suite of array functions that can make our days easier and our code more readable. Hopefully, this article has given you the confidence to know which array function to use and why.

TLDR

• Use .map to "transform" an array of one thing into an array of something else
• Code smell: Mutating a local variable from within a map.
• Use forEach for creating side-effects beyond the local scope
• Code smell: Using .push inside forEach. Consider alternative e.g. .map or .filter
• Code smell: Mutating a local variable from within a forEach.
• Use reduce to reduce a collection of elements to a single aggregate entity.
• Code smell: Returning an array from reduce. Consider map and/or filter instead.

However the costs of a Microservices Architecture are not talked about nearly as often as they should be. The trade off when we create a new Microservice is increased technical complexity, increased coupling, dependent deployments, greater risk of introducing distributed transactions, and overall a decreased rate of change in our applications.

Wait a minute! The costs sound almost identical to the benefits, so what's going on here? Some of the dogmatic enthusiasm for Microservices is due to our familiar old friend the causal fallacy. Microservices architectures correlate with other beneficial behaviours, up to a point, beyond which their costs exceed their benefits and the approach begins providing negative value.

This benefit is non-linear, the first few Microservices provide a lot of extra value! But these returns quickly diminish, until the Microservices architecture itself becomes a hindrance, in some instances reducing productivity to the point where change is now harder than it was before.

The irony of the negative value phase is that the system has created a feedback loop where the cost to create a new Microservice is low, and the pain is distributed across teams/engineers in the organisation. No one can solve the problem individually, everyone must agree, but whoever continues creating new Microservices reaps the benefits of another team's good behaviour. Microservices architecture can thereby create a system implementing tragedy of the commons. This is how companies wind up with thousands of Microservices, even after the approach shows its flaws.

My goal in this article is not to persuade people that Microservices Architecture is bad. Instead, I want to expose the root cause behind the initial rise in productivity with Microservices, and then find ways to maintain or extend it. My objective is to create an approach that helps us find and stay at the top of the hill where we have the greatest rate of change. I want to give us a set of tools to know when to apply this architectural approach, and how far to go.

The reason Microservices work so well at first is because, when you break up your system into tiny components, it is highly improbable that a single new service with a single responsibility will cross domain boundaries. Your first Microservice was a success, so you build a few more. Along the way you re-allocate these services between teams quite easily as you intuit where the domain boundaries are in your business. So far so good, and quickly you build up tens, then hundreds, and maybe eventually thousands of Microservices, as Uber did.

This is what I call the shotgun approach to Microservices architecture, and it usually occurs due to a lack of design. If you make the pieces small enough, no one has to do the hard design work of identifying bounded contexts or designing domains up front. Instead you can always trade service ownership between teams. The person who championed the approach in the beginning gets lots of praise as the team climbs the hill in the above diagram.

And then the wheels fall off. The top of the hill is a narrow precipice.

Probably around the time you (accidentally) create your first distributed transaction, you realise your Microservices aren't so decoupled, and perhaps you have built a distributed monolith. Every feature that made them great before, now makes them challenging. If you make a change to a service, you can't reasonably tell which other services are directly or indirectly affected. Observability becomes a nightmare, where a single user interaction may be handled by a dizzying number of services talking to each other. What was once function calls in code has now become distributed network requests that must be traced. An outage in one service results in a set of failures that light up your alerting systems like a Christmas tree, making the root cause a nightmare to diagnose.

Whole new solutions are invented to solve problems that teams created for themselves, further increasing complexity and the likelihood of unpredictable emergent behaviour creating incidents of change failure, which themselves become increasingly difficult to solve.

You might be building and deploying small Microservices independently, but the meaningful non-functional requirements live at the system level. You can't reasonably test the suitability of a Microservice deployment in isolation, because the system behaviour is contingent on the whole. As we reduced the responsibilities of a service down to a single purpose, we too reduced it's overall contribution to the -ilities that make our system manageable.

Likewise, Microservices may be deployed independently, but meaningful, value-adding change that the business asks for requires changes to multiple Microservices. With too many services, these changes themselves cascade across services. You've gone from one deployment containing many changes, to one change requiring many deployments.

What's the alternative? Like abstract versus concrete classes, decoupling vs cohesion, the answer is applying things appropriately, and moderately. The key benefit of the first few Microservices is that we create an architecture where teams are empowered to deploy code changes that align with the axis of change in the business. So how do we define moderate and appropriate?

Typically, the first system we split out is one that is obviously discrete, such authentication or payments. The boundary between the new service and the old service is easy to identify: a small surface area of interaction, either side of which the two services are relatively independent. It's easy to get right, hard to get wrong the first time. But taking this first win and going from a crawl to a sprint is ill advised.

Instead, take the time to do planning and analysis before creating your Microservices. This planning typically has to occur sometime after the business has actually begun operating. Planning a Microservices architecture in a greenfield product before achieving product market fit is a sure way to get the bounded context wrong. You're aiming for a fast moving target. During the incubation phase, there's more to be gained by focusing on building differentiators and buying the rest than there is in architecting Microservices.

However with a product in the growth or extraction phase, you can analyse existing code in a well architected monolith, and don't split your services if you can't create a well architected monolith first. Look for clear business domains with very few dependencies on other parts of the code. These are vertical slices of the code you are likely to be asked to change by a product manager, and would be relatively easy to do so in one atomic change.

One technique is to go through Jira comparing past tickets to the code base. The crucial factor here is to align your domains along the axis of change, and our ticketing system is a great way to identify what tends to change discretely and what changes dependently.

You see, the rate of change is determined by the number of dependencies affected by that change. This is true in terms of code AND organisation; that is, your team structure should reflect your business domains which should reflect your code and your architecture. The more these four elements align, the lower the resistance and the faster the rate of change.

The perfect example of this is simply aligning services or teams along the wrong axis. That is, if you have a frontend Microservice, an API gateway Microservice, a few business logic services, and something crazy like a database service. Then imagine that each service had its own team. In this (slightly) contrived example, we have built teams and services around the layers of our application, rather than vertical slices. Since no change can occur without affecting all layers, teams AND services have to communicate a lot in order to deliver change or value.

A Microservices Architecture may start out looking like the diagram on the right, but if you're not careful, the quest for smaller and smaller services will inadvertently lead teams to create something akin to the diagram on the left. Remember, you're optimising for changeability, not reusability. They are two competing goals!

What you ultimately want to do is apply Domain Driven Design to find a small number of discrete, well bounded domains. Then you can structure your teams, code, architecture, and business around these domains.

I believe that finding the correct name for this approach and its services is crucial. The term Microservices seems to imply that more and smaller is better. However as we have seen, discrete and well bounded is far more important than size or number. That is why I have chosen to call them Domain Services*. This name reveals our true objective; aligning teams, code, and architecture with business domains.

My rule of thumb is that the number of Domain Services an engineering department should aim for is the number of feature engineers divided by three or five. That means a team of 10 engineers developing features should have 2 or 3 Domain Services, not 20! A team of 30 feature engineers could have 6 to 10 Domain Services.

This ensures that the services are large enough that they are easy to maintain transaction boundaries within them. The number of discrete domain boundaries that you have to find is reduced, and teams only have to create and maintain a contract/context map for the entities that are communicated across that exposed boundary, keeping communication costs low while greatly enhancing rate of change.

You may have heard that "Microservices should do only one thing!" but I would argue that this only seems true because it implicitly enforces the more accurate rule:

A domain must only be implemented in one service, but a service may implement more than one domain

This matches its sister rule in Domain Driven Design:

A domain must only be worked on by one team, but a team may work on more than one domain

Each Domain Service you add should be planned and executed with extreme caution! Every time you add a service, you increase the number of exposed domain boundaries rapidly. An exposed boundary is a domain boundary that is implemented across the network, between services. Every exposed boundary introduces an API surface that enforces a contract. A contract must be maintained or versioned through changes.

The more boundary exposure you have, the slower your rate of change behind that boundary. Heuristically speaking this means that if your service is too small, the ratio of behavioural code and API code tilts toward API code. As it does the utility of the service approaches zero, since any change in behaviour will more likely result in a change in API. The more Microservices you have, the smaller their behavioural code, and the greater their boundary exposure, soon every Microservice is more API than behaviour and productivity halts.

The reason we began creating Microservices in the first place was to increase the rate of change in behavioural code — a benefit we lose as we add more services. The business doesn't ask us to change API code, the business asks us to change the behaviour of the system. The API is meant to help facilitate that change, not hinder it.

For example, two Domain Services can have only one boundary and zero possible dependent services, three services gives us three boundaries, four services gives us up to six, and so on. This is without considering indirect dependencies, which can accumulate even faster.

The indirect dependencies column describes how a command from one service to another could subsequently depend on the remaining services. That is, if we have 4 services, for each service one of those services could issue commands to, each of those could depend on the two remaining services. When all of those possible dependencies are accounted for at a single depth, we have 12 opportunities for dependent calls.

Of course, these numbers represent the upper bound, the worst case scenario. Importantly they show how too many Microservices quickly turns into a new big ball of mud — the very thing we wanted to avoid, just with added network, concurrency, and deployment issues.

As you can see from the above table, the change from two Domain Services to three is almost as important as the change from one to two because a third service introduces new types of complexity. You will likely be carving this new Domain Service out of an existing one, so you must be thorough. Check that the candidate Domain Service doesn't cross transaction boundaries, investigate all the events and commands within the domain, and work with your product manager and/or business analysts to ensure your model of the domain matches theirs.

Modelling domains as a collection of commands, events, and aggregates, entities, or projections helps a lot with this process. Even if the code here is well established, it can be worth running an event storming session.

At this point in the article it is worth running through a definition of commands and events, because understanding the distinction between the two is critical. The simplest definition I have come up with is this one:

A command is a request to change the persisted state; An event is the delta of that change.

That is, every time you would write to a database, whether that is an insert, update, or delete, emit an event which describes what changed (NOT the new state!) An event cannot be rejected because it is a historical statement of fact. It might not be a happy fact, but it is one nonetheless.

Every time you have the intent of changing state, issue a command, knowing that it could be rejected. Commands can have outcomes, which might be the reason the command was rejected, or it might be acknowledgement that the command was accepted. When a command is accepted, an event must eventually follow.

It's important to note that events are the transaction boundary. In an event based system with multiple services, your system has to be and expect eventual consistency.

Let's say your checkout service issues a CompletePurchase command, and checks its simplified inventory data before accepting the command. It then emits a PurchaseCompleted event to the inventory service,which has a much more sophisticated understanding of inventory. At the same time, the inventory service has sent a StockAdjusted event because someone has reported an item as damaged at the warehouse. The inventory service now realises that it can't service the order because the available stock is no longer available. Should it reject the PurchaseCompleted event?

No. The purchase happened, the user's credit card has been charged. This is a matter of historical fact and cannot be rejected. You might say we should have prevented the purchase then, "if only we had kept these domains together!" you lament. But remember, the inventory domain only updated its available inventory count after the damage physically occurred in the warehouse. The real world itself is eventually consistent anyway!

In this example, you can see that the domain events are the true transaction boundary. A PurchaseCompleted event might lead to an ItemAllocated event from the inventory domain in the green path, reducing the available stock count, but the fact that these two events are correlated does not make them an atomic action.

If you model your domains this way, you will find that the domain boundaries are quite clear, and that the events and commands that are shared between them define the API boundaries between our Domain Services.

The reason this distinction between commands and events is crucial is because it relates to another key rule of Domain Services:

Communication between Domain Services should consist of events, not commands

If your Domain Services are issuing lots of commands to each other, this is an indication that your domain boundary is incorrectly placed. A single user interaction should result in a command within a single Domain Service. The primary Domain Service handling the initial command may dispatch one or more events to other Domain Services, which may themselves issue commands to themselves on receipt of the event (but not when replaying the event from their own event store). However these must be events, not commands, and must fail independently.

The result is that a user interaction may be satisfied by a single Domain Service without issuing dependent commands. This does not however, preclude additional responses from the events occurring asynchronously. E.g. if the Checkout Service emits a PurchaseCompleted event then the Notification Service may email the user an invoice based on the event it received and their preferences it has stored. The Dispatch Service may issue a CreateFulfilmentOrder command and begin the process of shipping goods to the user. Each of these commands may fail, but they fail independently.

Let's compare this to a shotgun Microservices Architecture implementation. In the shotgun approach, the CompletePurchase "command" may be a call to a Microservice that validates the cart, which then calls another service to check the billing, while also calling another service to reduce the stock levels, and another to create a carrier label for shipping, another service to update the users order history, and another service to email the invoice. All of these calls may be commands to other services, and all of them may fail in dependent ways which are then (hopefully) fed back to the user.

Coming back to the crux of this article, I am confident that a team of 35 developers working across 7 Domain Services will in the majority of cases, outperform a team of 35 developers working across 60 Microservices. Of course, no quantitative measure exists, but the stories of development teams reducing and reorganising their Microservices provides qualitative evidence to support my hypothesis, in addition to my personal experience, and deductive reasoning outlined in this article.

Ultimately the goal of any architecture is to facilitate ease of change into the future, without sacrificing our target non-functional requirements. This architectural objective is predicated on our ability to predict the nature of changes into the future. Of course, all predictions are imperfect, and increasingly so the further our outlook. However I believe Domain Services better aligns our architecture with our team structure and business structure by utilising tools such as Domain Driven Design to identify numerous independent (or at least loosely coupled) axes of change in our systems.

Summary of Heuristics and Rules

We covered a lot of ideas in this article, so we will quickly recap the key thoughts. I have split these into heuristics and laws.

The heuristics are rules of thumb that are usually true in any software system with "services", whether that is Microservices or Domain Services. I find these are useful ideas to keep in mind when consider the trade offs of introducing new services.

The laws are rules that define best practices for a Domain Services approach, and thus uses the language of RFC 2119

Heuristic of Multi-Service Systems

• As new services are added, complexity increases non-linearly
• As new services are added, rate of change increases by at most +1
• The total rate of change increases as the deployment to change ratio approaches 1:1
• The rate of change is reduced by the number of dependent/affected systems/teams for each change
• The utility of a service is its amount of behavioural code compared to API code
• Increasing the number of services reduces the amount of behavioural code in each service
• Increasing the number of services increases the amount of API code in each service

The Laws of Domain Services

• A domain must only be implemented in one Domain Service, but a Domain Service may implement more than one domain
• A domain must be owned by only one team, but one team may own more than one domain
• Domain Services must only communicate asynchronously
• Domain Services must communicate via events
• Domain Services should not dispatch commands to other services
• Events must be an atomic transactional boundary
• Events must never be rejected (but they can be ignored)
• Commands must be a rejectable request to change the persisted state
• Events must be a semantic encapsulation of the delta of altered state, and an immutable historical fact
• An accepted command must produce an event
• A Domain Service may issue a command to itself when it ingests an event from another Domain Service
• Domain Services must not share persistence layers
• A Domain Service should only receive data it requires from other Domain Services via events
• A Domain Service should not request additional data from another Domain Service in order to handle a command

Footnotes

* Fans of Domain Driven Design, especially those whom follow the Object Orientated implementation, might recognise the term as an implementation level pattern in code. Given the term isn't referenced in DDD: Distilled, seems to cause confusion, and isn't well distinguished from other DDD patterns, I feel that we can make better use of this term at the architecture level. Therefore I have stolen and repurposed it, sorry not sorry.

In this painfully familiar story, before every other sprint, stakeholders in the business raise some crucial feature that needs to be developed as quickly as possible. The business proclaims "it's an emergency!" (it isn't), and the team is asked to take on more technical debt and told "don't worry, we'll make time to deal with it later". Every day since has been "today" and not one of them seems to be this day called "later". The engineers have stopped wondering if later will ever come...

Over time, the team's productivity asymptotically approaches zero. Product managers are getting frustrated, engineers are frustrated, stakeholders have given up expecting anything from Product (unless they declare it an emergency), and the tech debt seems insurmountable. Developing new features is neither easy nor enjoyable. Everyone has a different solution, from re-writes to an entire quarter without feature work.

In this article, not only do I offer a viable way out of the morass, but a strategy to ensure things don't get this bad in the first place. It's outstandingly simple, and requires only a little focus to execute on. I call it The Three Stream Backlog

The what now?

If this idea catches on I'm going to really regret not coming up with a better name, but The Three Stream Backlog is dividing your backlog up into well... three streams of work.

Feature Work

This stream contains the epics that Product Managers and the team have poured blood, sweat, and tears into developing and preparing for development. The epics in this stream have been through discovery, user research, user testing, prototyping, tech spikes, technical feasibility planning, etc. The team is confident that these epics are a good use of engineering resources.

We're not going to spend too long talking about this stream because the entire Product Management discipline is devoted to doing this well.

Allocate 50 - 70% of the team's engineering resources to this stream.

Business as Usual

This stream contains only tickets for bug fixes, copy improvements, legal document updates, and minor changes. There are no epics in this stream! It's really important that large features do not try to sneak in.

Allocate 20 - 25% of the team's engineering resources to this stream.

Technical Work

This stream contains both tech debt tickets AND epics. However, these epics are created by engineering leadership with the same rigor, research, planning, and confidence in deliverable value as the epics in the feature stream.

This means engineering leadership need to learn some product management skills and maintain their own backlog. The crucial step here is really stepping back and looking at the big picture. Make sure your tech backlog supports and empowers the team to deliver on the feature roadmap.

Does that mean you need to plan an upgrade to Hasura to support web hooks for the upcoming integration feature? Do you need to implement a repeatable solution to building audit tables alongside the mutable tables that Data & Analytics teams are relying on for their reports? Is the developer experience slowing everyone down and causing "It works on my machine!" bugs? Maybe deployments are too difficult and error prone.

The tech stream is partially recursive, because you will find it contains a stream of big features and smaller bug fixes, just like the three streams themselves. You need to put on your Product Management hat and figure out how to efficiently allocate limited resources to seemingly unlimited problems. This leads me to my next point.

Executing on The Three Streams

The critical detail in implementing this effectively is allocating people to particular streams for entire sprints. Context switching kills focus, kills productivity, and saps our team's morale.

So given a team of 4 engineers, you would allocate them to the 3 streams as such:

This focus is the critical detail that lets you plan big, empowering tech epics and deliver serious improvements to the infrastructure, developer experience, and non-functional requirements of the product! You now have dedicated resources each sprint to work on the tech.

What's the alternative? I think we have all tried the "30% of the sprint's story points are for tech debt tickets" approach. Has this ever worked for anyone? The problems with that approach are:

• You can't plan significant tech work
• Tech debt tickets are prioritised last in the sprint, meaning they get done in a rushed and haphazard way, if picked up at all
• You can't predict delivery of tech work
• Often, no one is really looking after these tickets or planning the tech backlog

In the end, the story point approach leaves the tech team feeling even more disempowered, and the technical challenges seemingly even more insurmountable.

Tips and Tricks

Win over Product Managers

Product Managers might push back on the idea of them handing over up to 25% of the available engineering resources every sprint for tech to work on their own backlog. However, remind them that this means they don't have to prioritise tech debt tickets ever again. It means the team will deliver faster and more efficiently. It means the team will be able to estimate work better because the code is easier to work with. It means deployments will become more reliable and less stressful.

Try it for a quarter and ensure you deliver a significant technical challenge that improves your ability to deliver Product features reliably. Then, tell everyone that will listen that this was possible because you were able to plan technical work via the three stream backlog and deliver it with uninterrupted focus. When I first introduced it, my team introduced Continuous Deployment for the first time in a business that many thought would never pull it off.

Avoid Stream Fatigue

It's important to swap people around between the feature, BAU, and tech streams. Everyone needs that shared context, and often the things people learn working in one stream will make them more effective in another.

However, be mindful not to break peoples' flow. Let people stay in the tech stream until their technical epic is complete, and same for the feature stream. Since the BAU stream is made up entirely of small tickets, it's easier to swap people in and out of this stream between sprints than the feature and tech streams.

Report on Outcomes with Discipline

As a new Product Manager of the technical backlog, you need to put in place some systems to measure the success of your tech epics. This means your epics need to not only be well formed coming in, but well analysed on the way out. I do this by creating a report in Confluence for each big feature, and include:

• What went well
• What could be improved next time
• What outcomes occurred that I expected (i.e. reduced lead time, increase release frequency, better team health metrics, etc)
• What outcomes occurred that I did not expect

When people ask you if this approach is working, you will have a repository of delivered work and measured outcomes to direct them towards.

Process Compatibility

So far I've seen the three streams work best on agile teams running Scrum with sprints. It works decently on Kanban teams, but swapping engineers between streams gets a little messy.

It is not compatible with Shape Up, but then I believe Shape Up is incompatible with life, healthy teams, kittens, and basically anything good and bright in the world.

A mistake I have seen many teams make, and regret, is managing these release toggles from the database. This approach has several issues:

1. Every release must test both paths, as the toggle could change at any time
2. Developers working on the code base have less visibility into which flags are currently enabled in which environments
3. Product Managers, developers, and testers often have to coordinate on the removal of feature toggles, this often leads to toggles simply not being removed
4. Following on from above, old feature toggles often stop being tested. This means every stale feature toggle is a liability that could cause an incident.
5. A full history of changes is either lost or needs tooling built to support it

Sure, many of the above issues could be solved with good tooling and strict processes, aside from #1. However all of these issues can be solved with tools your team has now: Source control & code.

By putting your toggles in code, you get the following advantages:

1. Changes only need to test the active code paths in each release, as they cannot change without another release
2. Changes to toggles have to go through the full SDLC/release pipeline and thus we can be confident they are working as expected
3. Developers can clearly see what features are enabled in what environments, who worked on them, and when they were changed
4. Developers can easily remove toggles once they have finished with them, as part of their feature clean up.

Because this approach seriously empowers developers to manage these toggles themselves, I believe this approach needs its own term: Dev Flags

The advantage of this name is that it is clear who is responsible for them: Developers! This is just part of how we get work done, Product managers don't need to be involved except to confirm they would like a new feature to go live in production.

This clearly delineates the costs involved in maintaining a feature toggle for other purposes, such as A/B Testing, Ops Toggles, and Canaries. By using two seperate systems, we can be clear with product managers, maintaining a new toggle will incur extra overheads on our time, impact our release cycles, and introduce an element of fatigue and complexity over time. This can help us ensure we collaborate on an effective plan to manage the lifecycle of our feature toggles when they are needed, and remove them as soon as they are no longer necessary.

How to implement Dev Flags? In projects using TypeScript for frontend and backend, I like to include these in a shared package. My folder structure might be:

And the contents of devFlags.ts:

type RuntimeEnvironments = 'dev' | 'stg' | 'prd';

type DevFlags = {
  useNewProfileScreen: boolean;
  usePhoneCheck: boolean;
};


type DevFlagsEnvironmentMap = Record<Exclude<RuntimeEnvironments, 'stg'>, DevFlags>;

const getEnvironmentDevFlagSet = (env: RuntimeEnvironments): DevFlags => {
  const envs: DevFlagsEnvironmentMap = {
    dev: {
      useNewProfileScreen: true,
      usePhoneCheck: true,
    },
    prd: {
      useNewProfileScreen: true,
      usePhoneCheck: false,
    },
  };

  const flagEnv = env === 'stg' ? 'prd' : env;

  return envs[flagEnv];
};

export const DEV_FLAGS = getEnvironmentDevFlagSet(env.RUNTIME_ENVIRONMENT);

Why do we exclude staging and instead treat it as prod? Because staging is where we confirm that the current deployment artefact is suitable for release to production. We can't do that if we're executing different code paths in staging and production.

In this one file I can see what features are in development, where they're active, who added which lines, and I can lookup usages to find these flags in the codebase.

As a rule of thumb, you want the number of dev flags in code to be equal to or less than the number of developers working in the code base. This requires some discipline to ensure you remove the flag after releasing the feature. Advantageously, developers are motivated to remember to remove flags because we like to keep our code clean, and we're empowered to do it, because it just another merge, same as we do all day.

Will you try Dev Flags with your team? Does your team do something else? Does it work for you? Let us know in the comments!

Cover photo: The Twin Jet Nebula, or PN M2-9, is a striking example of a bipolar planetary nebula. Bipolar planetary nebulae are formed when the central object is not a single star, but a binary system, Studies have shown that the nebula’s size increases with time, and measurements of this rate of increase suggest that the stellar outburst that formed the lobes occurred just 1200 years ago.

In this article, I will use the queue from my es-reduxed package. The purpose of this queue is to:

• Store a list of unprocessed event ids
• Maintain the correct order of events
• Ensure an event is only processed once

These assurances must be made because there is no guarantee that the events will be sent from the subscription only once and in order. While the system is processing one event, it may receive several more.

As you can guess, if we were to wind up with two instances of the event queue, we could no longer guarantee order and single processing constraints.

Let's take a look at the queue itself, starting with the types:

export type Queue = {
  enqueue: (id: number) => void;
  registerPromise: (id: number, resolver: PromiseResolver) => void;
};

export type PromiseResolver = (value: Store<any, any>) => void;

So we can see a Queue is just an object with two properties:

• enqueue: a function taking a number and returning void
• registerPromise: a function taking a number and a function

The registerPromise function is just associating a resolve function from a Promise with an event id so that the queue can resolve the promise with the given redux state when it finishes processing that event.

Let's take a look at the queue implementation:

/**
 * This queue system uses a recursive loop and a primitive state machine to
 * ensure that events are dispatched to redux in exactly the order they were
 * received.
 */
const startQueue = <T extends EventBase>(
  reduxStore: Store<any, any>,
  eventsRepo: EventsRepo<T>
) => {
  const queue: number[] = [];
  const dedupeSet = new Set<number>();
  const promiseMap = new Map<number, PromiseResolver>();
  let state: 'READY' | 'PROCESSING' = 'READY';

  const processEvent = (event: EventBase) => {
    reduxStore.dispatch(event);
    if (event.id === undefined) {
      throw new Error(`Malformed event is missing id: ${event}`);
    }
    const resolver = promiseMap.get(event.id);
    if (resolver) {
      resolver(reduxStore.getState());
      promiseMap.delete(event.id);
    }
  };

  const processQueue = async () => {
    if (state === 'READY') {
      queue.sort((a, b) => a - b);
      const eventId = queue.shift(); // So we only process if something was in the queue
      if (eventId) {
        state = 'PROCESSING';
        if (queue.length) {
          // More than one event in queue, so do bulk processing
          const lastEventIndex = queue.length - 1; // Save queue length in-case it changes during the await
          const lastEventId = queue[lastEventIndex];
          const events = await eventsRepo.getEventRange(eventId, lastEventId);
          events.forEach(processEvent);
          queue.splice(0, lastEventIndex + 1);
        } else {
          const [event] = await eventsRepo.getEvents(eventId - 1, 1);
          processEvent(event);
        }
        state = 'READY';
        processQueue();
      }
    }
  };

  return {
    enqueue: (id: number | string) => {
      const idCoerced = typeof id === 'string' ? parseInt(id, 10) : id;
      if (!dedupeSet.has(idCoerced)) {
        dedupeSet.add(idCoerced);
        queue.push(idCoerced);
        processQueue();
      } else {
        console.warn(`Out of order event: [${idCoerced}]`);
      }
    },
    registerPromise: (id: number, resolve: PromiseResolver) => {
      promiseMap.set(id, resolve);
    },
  };
};

Okay, let's break this down:

const startQueue = <T extends EventBase>(
  reduxStore: Store<any, any>,
  eventsRepo: EventsRepo<T>
) => {

The first thing to notice here is that we do not export this function, startQueue is available only inside the queue.ts module. This is a critical point we will come back to later.

  const processEvent = (event: EventBase) => {
    reduxStore.dispatch(event);
    if (event.id === undefined) {
      throw new Error(`Malformed event is missing id: ${event}`);
    }
    const resolver = promiseMap.get(event.id);
    if (resolver) {
      resolver(reduxStore.getState());
      promiseMap.delete(event.id);
    }
  };

This function is defined inside startQueue, so it is only available within the startQueue function. This is similar to a private method. However, it has access to all variables included in its closure. In this case, we are making use of promiseMap and reduxStore. This is similar to private properties in a class, but we use closures to make them inaccessible outside this context.

  const processQueue = async () => {
    if (state === 'READY') {
      queue.sort((a, b) => a - b);
      const eventId = queue.shift(); // So we only process if something was in the queue
      if (eventId) {
        state = 'PROCESSING';
        if (queue.length) {
          // More than one event in queue, so do bulk processing
          const lastEventIndex = queue.length - 1; // Save queue length in-case it changes during the await
          const lastEventId = queue[lastEventIndex];
          const events = await eventsRepo.getEventRange(eventId, lastEventId);
          events.forEach(processEvent);
          queue.splice(0, lastEventIndex + 1);
        } else {
          const [event] = await eventsRepo.getEvents(eventId - 1, 1);
          processEvent(event);
        }
        state = 'READY';
        processQueue();
      }
    }
  };

Here we continually process the queue in a recursive loop, as long as there are events remaining and the queue is not already processing events. This ensures we only process events once. Because this function is async (it returns a promise), and because it will always call await before it recursively calls processQueue again, this function will not lock the event loop.

It's also important to realise that there might be calls to enqueue during the await step. This would grow the queue, but not be included in the call to getEventRange, so this is why we save the last event index before we await; otherwise, we could accidentally splice out events we hadn't processed yet.

Now that we understand the queue's "private methods" and properties, let's take a look at the "public methods" in the return statement:

  return {
    enqueue: (id: number | string) => {
      const idCoerced = typeof id === 'string' ? parseInt(id, 10) : id;
      if (!dedupeSet.has(idCoerced)) {
        dedupeSet.add(idCoerced);
        queue.push(idCoerced);
        processQueue();
      } else {
        console.warn(`Out of order event: [${idCoerced}]`);
      }
    },
    registerPromise: (id: number, resolve: PromiseResolver) => {
      promiseMap.set(id, resolve);
    },
  };

Aha! So when we enqueue an event, we call processQueue if it is an event we haven't seen before. Remember the state checks in processQueue? We can safely call this function here because it won't do anything if there is already a process running, and it will eventually get to our enqueued event through the recursive loop.

Meanwhile, registerPromise maps a promise to the event id, which will be used later. In this implementation, we only allow resolving one promise per event processed.

Let's get to the meat of this article, the function that instantiates or retrieves this queue as a singleton:

export const getQueue = (() => {
  let instance: Queue;
  return <T extends EventBase>(
    reduxStore: Store<any, any>,
    eventsRepo: EventsRepo<T>
  ) => {
    instance =
      instance === undefined ? startQueue<T>(reduxStore, eventsRepo) : instance;
    return instance;
  };
})();

First, take note of the fact that this is the only run-time export from queue.ts. You can only retrieve a queue by calling getQueue, but what's actually happening here?

export const getQueue = (() => {
 // Trimmed
})();

Above is an immediately invoked function expression. We are defining a function and then calling it. The result of this function is then assigned to the variable getQueue. Well, getQueue is a function -- we can tell because the first word is a verb -- so this immediately invoked function expression needs to return a function.

(() => {
  let instance: Queue;
  return // Trimmed
})();

Before it returns, we declare a mutable variable called instance of the type Queue but do not define a value for it, which means it will be undefined.

(() => {
  let instance: Queue;
  return <T extends EventBase>(
    reduxStore: Store<any, any>,
    eventsRepo: EventsRepo<T>
  ) => {
    // Trimmed
  };
})();

Aha! So after we declare our instance variable for storing a queue, we define a function for our immediately invoked function expression to return. This function signature should look familiar -- it is identical to startQueue's function signature.

(() => {
  let instance: Queue;
  return <T extends EventBase>(
    reduxStore: Store<any, any>,
    eventsRepo: EventsRepo<T>
  ) => {
    instance =
      instance === undefined ? startQueue<T>(reduxStore, eventsRepo) : instance;
    return instance;
  };
})();

The final step: Inside the function returned to getQueue by our immediately invoked function expression, we will: If instance is undefined, call startQueue, passing in our generic type parameter, reduxStore, and eventsRepo, OR, in the case that it is defined, we simply leave instance as instance. Then we return instance.

This means that instance is stored in the closure scope of getQueue. It is no longer accessible anywhere in javascript except by getQueue itself. We can be confident that getQueue will only ever return a single queue instance. The first time it instantiates the queue, and subsequent calls return it.

You can test this fact by asserting:

expect(getQueue()).to.equal(getQueue());

This will return true because both calls return a reference to the same object!

console.log(getQueue() === getQueue()); // true

Remember object equality in javascript compares references.

There's a massive caveat in this implementation; can you spot it? We only use the parameters the first time getQueue is called! This means that if I try to start one queue for one store and another queue for another store, it will simply ignore the second store and return my queue for the first store.

In this way, this pattern is not memoization. If we were using memoization, we would be able to create a singleton per combination of reduxStore and eventsRepo. However, this makes it harder to enforce a singleton pattern. What defines whether reduxStore and eventsRepo are the same as the last call? Would we compare object equality? Deeply nested properties?

For example, if I were to use a proxy-based memoization implementation such as proxy-memoize then changes to properties of, or child properties of reduxStore and eventsRepo would cause calls to getQueue to return a new queue instance! Uh oh!

I like to think of this pattern as brutal memoize. You get to provide your parameters once, and after that, we ignore them.

Will you be using this functional singleton pattern in your code? Do you have a different approach? Let me know what you think in the comments!

Photo by Michael Dziedzic on Unsplash

For the unfamiliar, the migrations we are talking about here are a set of schema changes for a relational database such as Postgres, MySQL, etc. Sometimes these are generated by an ORM based on models, and sometimes they are written by developers. They're almost always (and should be) committed to version control systems, and must be executed in a particular order.

Our "up" migrations might create some tables, alter some columns, set constraints, and so forth. This is normally a straight-forward affair. Developers do this day in and day out, we test these on our local before committing them, and if we're going to run alterations on existing tables, we usually test them out on prod-like data to measure impact. We're confident about up migrations.

My case against down migrations starts rather predictably; when do you run them? Hardly ever. How confident are you that every down migration in source control was tested before it was executed? Was it tested after prod-like data was inserted into the table?

I know the answer to both those questions is "Oh s#%T!" or an amused but panicked chuckle to yourself. The fact is that when it comes time to fix a schema problem in production, those down migrations you've been saving for a rainy day? There is NEVER a situation where they are the right solution.

Teams sometimes laugh nervously amongst themselves that their disaster recovery plans are an entirely unknown quantity because they've never tested them. Down migrations are like hundreds of untested disaster recovery plans, only much much worse.

Putting down migrations in source control, and then deploying them to production, is like enjoying a TV dinner with unexploded ordnance on the coffee table. At some point, someone is going to mistakenly run "down" in prod thinking it was dev/local/etc. At least you've practiced your disaster recovery though, right? Hey how often do your automated backups run and when did you last check them? Go, I'll be here when you get back!

...Oh you're back? Are you okay? You look stressed! Anyway, even if you do successfully run a down migration at some point and it proves useful... now what? Your migration history no longer makes sense. The fact that X, Y, or Z was run in production to change the schema in some particular way and unpick some mess, is lost to history.

I propose you forget all about down migrations. Delete them all from source control right now, I promise you no one will miss them.

What happens if you do make a change you want to rollback in production? Ah yes, you carefully figure out how to roll forward! You write a new "up" migration to clean up your mistake. That's right, you write your drop table or alter column statements in an up migration. You prepare a run sheet, you test, and you go through your normal SDLC for deploying schema changes (You do run migrations via CD right?) Now your changes are in source control. They're part of history, as they should be, because it happened, and all future migrations will safely run on top of that history.

So what do you think? Are you going to delete your down migrations from source control? I'd love to hear about it if you do! Come back and tell us how it went in the comments!

Photo by Avery Nielsen-Webb from Pexels