Test storage failover and recover writes after an outage

Run an outage drill with failover() and two local MinIO servers, see which backend answers each read, write, and delete, then copy the missed writes back and verify both sides match.

With failover(), reads and writes keep working while the primary store is down. But a write made during the outage lands only on the secondary, and the primary doesn’t know about it. When the primary comes back, files written during the outage disappear from view, and files deleted during the outage come back. The plugin doesn’t reconcile anything. That part is yours, and a drill is the only way to find out whether your plan works.

This guide runs that drill against two local MinIO servers. It records every write the secondary accepts in a journal, takes the primary down, and notes which backend answered each call. It then brings the primary back and shows why a plain sync() gets the reconciliation wrong. Finally, it copies the missed writes back from the journal and checks every object on both sides by SHA-256. The results below come from running exactly this code.

Before you start

• The MinIO server binary (brew install minio/stable/minio on macOS, or a download from min.io) and three terminals.
• Node.js 22 or later, or Bun.
• Written against files-sdk 3.0, @aws-sdk/client-s3 3.1148, and MinIO RELEASE.2025-09-06T17-38-46Z. The scripts ran under Bun 1.4.2.

npm install files-sdk @aws-sdk/client-s3 @aws-sdk/lib-storage @aws-sdk/s3-presigned-post @aws-sdk/s3-request-presigner

pnpm add files-sdk @aws-sdk/client-s3 @aws-sdk/lib-storage @aws-sdk/s3-presigned-post @aws-sdk/s3-request-presigner

yarn add files-sdk @aws-sdk/client-s3 @aws-sdk/lib-storage @aws-sdk/s3-presigned-post @aws-sdk/s3-request-presigner

bun add files-sdk @aws-sdk/client-s3 @aws-sdk/lib-storage @aws-sdk/s3-presigned-post @aws-sdk/s3-request-presigner

nub add files-sdk @aws-sdk/client-s3 @aws-sdk/lib-storage @aws-sdk/s3-presigned-post @aws-sdk/s3-request-presigner

aube add files-sdk @aws-sdk/client-s3 @aws-sdk/lib-storage @aws-sdk/s3-presigned-post @aws-sdk/s3-request-presigner

The reconcile step streams objects from one server to the other, and the minio() adapter uploads a stream through @aws-sdk/lib-storage. The presigner packages are needed for url() and signedUploadUrl().

What the plugin promises

failover() wraps every operation. It tries the primary first, then each secondary in order, and returns the first success. Three rules decide what you’ll see in the drill, and the reference covers the rest:

• It fails over only when a backend looks down. That means a Provider error: a refused connection, a 5xx, or a timeout. A NotFound or Unauthorized is a definitive answer from a healthy backend, so it comes back as-is.
• A write goes to one backend. It lands on the first reachable backend and isn’t copied to the others.
• Each call starts at the primary. The plugin keeps no state between calls. Once the primary answers again, every read goes there first.

Start two MinIO servers

Run each server in its own terminal, with its own data directory:

mkdir -p drill/data-primary drill/data-secondary

# Terminal 1: the primary
MINIO_ROOT_USER=drilladmin MINIO_ROOT_PASSWORD=drillsecret123 \
  minio server drill/data-primary --address :9000 --console-address :9001

# Terminal 2: the secondary
MINIO_ROOT_USER=drilladmin MINIO_ROOT_PASSWORD=drillsecret123 \
  minio server drill/data-secondary --address :9100 --console-address :9101

Open each console (http://127.0.0.1:9001 and http://127.0.0.1:9101), sign in with those credentials, and create a bucket named uploads on each. The third terminal runs the scripts.

Wire up failover with a write journal

onFailover tells you an operation moved to the secondary, but it carries only the verb and backend indices, not the key. To reconcile, you need the keys. The simplest place to collect them is the secondary adapter itself: the plugin only sends a write there when the primary failed, so every write the secondary accepts is a write the primary missed.

import { appendFile, readFile } from "node:fs/promises";

import { Files } from "files-sdk";
import type { Adapter, OperationOptions } from "files-sdk";
import { failover } from "files-sdk/failover";
import { minio } from "files-sdk/minio";

const credentials = {
  accessKeyId: "drilladmin",
  secretAccessKey: "drillsecret123",
};

// In production these are your real backends, for example r2() and s3().
const primaryAdapter = () =>
  minio({
    bucket: "uploads",
    endpoint: "http://127.0.0.1:9000",
    ...credentials,
  });
const secondaryAdapter = () =>
  minio({
    bucket: "uploads",
    endpoint: "http://127.0.0.1:9100",
    ...credentials,
  });

// Direct handles on each backend, for replication and reconciliation.
export const primary = new Files({ adapter: primaryAdapter(), timeout: 2000 });
export const secondary = new Files({
  adapter: secondaryAdapter(),
  timeout: 2000,
});

export interface JournalEntry {
  op: "upload" | "delete";
  key: string;
  at: number;
}

export const JOURNAL = new URL("failover-journal.jsonl", import.meta.url);

const record = (op: JournalEntry["op"], key: string) =>
  appendFile(JOURNAL, `${JSON.stringify({ op, key, at: Date.now() })}\n`);

export const readJournal = async (): Promise<JournalEntry[]> => {
  const text = await readFile(JOURNAL, "utf8").catch(() => "");
  return text
    .split("\n")
    .filter(Boolean)
    .map((line) => JSON.parse(line) as JournalEntry);
};

// Every write the secondary accepts is a write the primary missed.
const journaled = (adapter: Adapter): Adapter => {
  const { move } = adapter;
  return {
    ...adapter,
    async upload(key, body, opts) {
      const result = await adapter.upload(key, body, opts);
      await record("upload", key);
      return result;
    },
    async delete(key, opts) {
      await adapter.delete(key, opts);
      await record("delete", key);
    },
    async copy(from, to, opts) {
      await adapter.copy(from, to, opts);
      await record("upload", to);
    },
    // Without a native move, Files runs copy + delete, which are journaled.
    ...(move && {
      async move(from: string, to: string, opts?: OperationOptions) {
        await move(from, to, opts);
        await record("upload", to);
        await record("delete", from);
      },
    }),
  };
};

// The instance the app uses.
export const files = new Files({
  adapter: primaryAdapter(),
  timeout: 2000,
  retries: 2,
  plugins: [
    failover({
      secondaries: journaled(secondaryAdapter()),
      onFailover: ({ operation, failed, next, error }) =>
        console.warn(
          `  failover: ${operation} ${failed} -> ${next} (${error.code}${error.timedOut ? ", timed out" : ""})`
        ),
    }),
  ],
});

A few details matter here:

• The journal is written after the secondary accepts the write. If the process dies between the two, that write isn’t journaled. The verification step at the end catches it. In production, write the journal to your database, not a local file, so every app instance appends to the same one.
• Spreading the adapter copies its capabilities and other methods, so the secondary still answers reads and declares what it supports. The minio() adapter’s raw getter is read once at the spread, before the client has loaded, so the wrapped copy’s raw is undefined. The plugin doesn’t use it.
• timeout: 2000 matters most. A refused connection fails fast. A primary that accepts the connection and then never answers stalls the call until the timeout fires. Without one, a call can hang for good. The plugin’s internal instance for each secondary inherits the same timeout and retries.
• retries: 2 delays the failover. Retries run against the primary before the plugin moves on, so each failed-over call pays for them. A timeout isn’t retried, so a hung primary costs one timeout per call.

Seed and replicate

import { sync } from "files-sdk";

import { primary, secondary } from "./storage";

await primary.upload("docs/a.txt", "alpha v1");
await primary.upload("docs/b.txt", "bravo v1");
await primary.upload("docs/c.json", '{"c":1}');
await primary.upload("docs/deleted-before.txt", "gone soon");

// Replicate. In production this is native replication or a scheduled sync.
const { uploaded } = await sync(primary, secondary, { compare: "size" });
console.log(`replicated ${uploaded.length}`);

// Replication lag: a delete and a write the replica hasn't seen yet.
await primary.delete("docs/deleted-before.txt");
await primary.upload("docs/lag.txt", "written after the last sync");

The last two lines model the lag every asynchronous replica has. Whatever your replication interval is, an outage can start inside it.

Take the primary down

Stop the primary with Ctrl-C in its terminal, then run a mix of operations through the app’s instance:

import { FilesError } from "files-sdk";

import { files } from "./storage";

const steps: [string, () => Promise<unknown>][] = [
  ["download a.txt", async () => (await files.download("docs/a.txt")).text()],
  [
    "download lag.txt",
    async () => (await files.download("docs/lag.txt")).text(),
  ],
  [
    "download deleted-before.txt",
    async () => (await files.download("docs/deleted-before.txt")).text(),
  ],
  [
    "upload new.txt",
    () => files.upload("docs/new.txt", "written during the outage"),
  ],
  ["overwrite b.txt", () => files.upload("docs/b.txt", "bravo v2")],
  [
    "upload a stream",
    () => files.upload("docs/stream.txt", new Blob(["streamed"]).stream()),
  ],
  ["delete a.txt", () => files.delete("docs/a.txt")],
  ["copy c.json", () => files.copy("docs/c.json", "docs/c-copy.json")],
  ["url c.json", () => files.url("docs/c.json", { expiresIn: 60 })],
];

for (const [label, run] of steps) {
  console.log(label);
  try {
    const value = await run();
    console.log(`  ok ${typeof value === "string" ? value : ""}`);
  } catch (error) {
    const err = FilesError.wrap(error);
    console.log(`  ${err.code}: ${err.message}`);
  }
}

What the drill showed

With the primary stopped:

Three more runs used the same instance with the primary in a different state:

What stands out:

• url() and signedUploadUrl() didn’t fail over. The minio() adapter signs URLs locally, with no request to the server, so nothing failed and the plugin had no reason to move on. S3 and the S3-compatible adapters sign the same way. During an outage, redirect downloads and direct browser uploads point at a backend that isn’t there.
• Streams went to the primary only. A ReadableStream can be read once, so the plugin hands it to the primary and surfaces the error. A resumable upload with a control does the same, since an UploadControl drives exactly one upload. That includes uploads proxied through the gateway, which streams the request body into files.upload(). Buffer the body first if an upload has to survive an outage.
• The replica’s lag became visible. For the length of the outage, a file deleted before it existed again, and a file written just before it didn’t exist.
• Credential errors don’t fail over. That’s by design: a bad key is a definitive answer. A revoked or rotated key on the primary takes reads and writes down rather than quietly moving them to the secondary.
• Each failed-over call paid for the primary first. With retries: 2, calls took 0.3–1.1 s against the stopped primary. The AWS SDK also retries a refused connection on its own. With retries: 0, the same head took 0.1–0.3 s.

Bring the primary back

Restart the primary with the same command. The data directory still holds everything it had before the outage. Reads now go to the primary, which answers definitively:

To make the reconciliation realistic, the drill then uploaded docs/new.txt again through files, as a user would after seeing their file missing. That write landed on the primary.

Why a plain sync gets this wrong

The obvious fix is to mirror the secondary back onto the primary:

import { sync } from "files-sdk";

import { primary, secondary } from "./storage";

const plan = await sync(secondary, primary, {
  compare: "size",
  prune: true,
  dryRun: true,
});
console.log({
  uploaded: plan.uploaded,
  skipped: plan.skipped,
  deleted: plan.deleted,
});

The dry run planned:

uploaded: docs/c-copy.json, docs/deleted-before.txt, docs/new.txt
skipped:  docs/b.txt, docs/c.json
deleted:  docs/a.txt, docs/lag.txt

Three of those seven actions are wrong:

• It brings deleted-before.txt back. The replica still had a file the primary deleted before the outage.
• It deletes lag.txt. The primary wrote it after the last replication, so the secondary never had it, and prune treats it as extra.
• It skips b.txt. The overwrite during the outage kept the same size, and compare: "size" can’t see a same-size change. A dry run with the default compare: "etag" did plan the upload, because both sides are MinIO and these were single-part uploads, but it still made the other two mistakes. Across providers, ETags don’t match even for identical bytes, so "size" is often the only built-in choice.

It also uploads the outage copy of new.txt over the one the user uploaded after recovery. A two-sided sync can’t tell a missed write from replication lag. Only a record of what happened during the outage can.

Reconcile from the journal

import { rename } from "node:fs/promises";

import { FilesError } from "files-sdk";
import type { FileInfo, Files } from "files-sdk";

import { JOURNAL, primary, readJournal, secondary } from "./storage";
import type { JournalEntry } from "./storage";

const dryRun = process.argv.includes("--dry-run");

const headOrNull = async (f: Files, key: string): Promise<FileInfo | null> => {
  try {
    return await f.head(key);
  } catch (error) {
    if (FilesError.wrap(error).code === "NotFound") {
      return null;
    }
    throw error;
  }
};

// The last entry per key wins: the secondary's current copy is the outcome.
const latest = new Map<string, JournalEntry>();
for (const entry of await readJournal()) {
  latest.set(entry.key, entry);
}

for (const [key, entry] of latest) {
  const [onSecondary, onPrimary] = await Promise.all([
    headOrNull(secondary, key),
    headOrNull(primary, key),
  ]);

  let action: "copy" | "delete" | "conflict" | "none" = "none";
  if (onPrimary?.lastModified && onPrimary.lastModified > entry.at) {
    action = "conflict"; // The primary took a newer write since. Decide by hand.
  } else if (onSecondary) {
    action = "copy";
  } else if (onPrimary) {
    action = "delete";
  }
  console.log(`${action.padEnd(8)} ${key}`);

  if (dryRun) {
    continue;
  }
  if (action === "copy") {
    const file = await secondary.download(key, { as: "stream" });
    await primary.upload(key, file.stream(), {
      contentType: file.contentType,
      ...(file.metadata && { metadata: file.metadata }),
    });
  } else if (action === "delete") {
    await primary.delete(key);
  }
}

if (!dryRun && latest.size > 0) {
  // Keep the applied journal as the incident record and start a fresh one.
  await rename(
    JOURNAL,
    new URL(`failover-journal.${Date.now()}.jsonl`, JOURNAL)
  );
}

The script only touches keys in the journal, and for each one it treats the secondary’s current state as the result of the outage. If the secondary has the key, the script copies it to the primary. If it doesn’t, the key was deleted during the outage, so the script deletes it from the primary. It doesn’t need to replay operations in order: an upload followed by a delete of the same key ends as a delete.

The conflict check covers the case sync got wrong. If the primary’s copy was modified after the journal entry, someone wrote to it after recovery, and the script leaves it alone. bun drill/reconcile.ts --dry-run printed:

conflict docs/new.txt
copy     docs/b.txt
delete   docs/a.txt
copy     docs/c-copy.json

Running it without --dry-run applied the three actions, left new.txt for review, and archived the journal. lag.txt and deleted-before.txt weren’t in the journal, so the script didn’t touch them.

Verify, then re-seed the replica

import { createHash } from "node:crypto";

import type { Files } from "files-sdk";

import { primary, secondary } from "./storage";

const hashes = async (f: Files) => {
  const out = new Map<string, string>();
  for await (const { key } of f.listAll()) {
    const file = await f.download(key, { as: "stream" });
    const hash = createHash("sha256");
    for await (const chunk of file.stream()) {
      hash.update(chunk);
    }
    out.set(key, hash.digest("hex"));
  }
  return out;
};

const [p, s] = await Promise.all([hashes(primary), hashes(secondary)]);
const report = {
  onlyPrimary: [...p.keys()].filter((key) => !s.has(key)),
  onlySecondary: [...s.keys()].filter((key) => !p.has(key)),
  different: [...p.keys()].filter(
    (key) => s.has(key) && p.get(key) !== s.get(key)
  ),
};
console.log(report);
process.exitCode = Object.values(report).some((keys) => keys.length) ? 1 : 0;

After the reconcile, it reported onlyPrimary: ["docs/lag.txt"], onlySecondary: ["docs/deleted-before.txt"], and different: ["docs/new.txt"]. The primary was now correct. The differences were all on the secondary: the replication lag from before the outage, plus the conflict resolved in the primary’s favor. Re-seeding the replica from the primary is now a one-way mirror with a known answer:

import { sync } from "files-sdk";

import { primary, secondary } from "./storage";

const plan = await sync(primary, secondary, {
  compare: "etag",
  prune: true,
  dryRun: process.argv.includes("--dry-run"),
});
console.log({
  uploaded: plan.uploaded,
  deleted: plan.deleted,
  errors: plan.errors,
});

The dry run planned uploads of lag.txt and new.txt and a prune of deleted-before.txt, which is exactly the drift verify.ts found. After the real run, verify.ts reported three empty lists and exited 0. compare: "etag" works here because both sides are MinIO. Between different providers, use "size" or a comparator that reads a checksum you control, and review the dry run before you prune. Migrate S3 files to R2 covers verification at bucket scale, where hashing every object costs real money.

The recovery runbook

1. During the outage, let onFailover alert you, and treat url() and signedUploadUrl() as broken for locally signed adapters.
2. When the primary returns, keep the app running. New writes land on the primary, and the reconcile’s conflict check protects them.
3. Dry-run the reconcile, review conflicts, then apply it.
4. Run the verification. Re-seed the replica from the primary with a reviewed dry run, then verify again until it exits 0.

Limits and tradeoffs

• No circuit breaker. Each call tries the primary first, so every call during an outage pays for the primary’s failure: a fast refusal, or the full timeout when the primary hangs. If that’s too slow, switch the app to a secondary-first instance when onFailover fires repeatedly, and switch back yourself.
• The journal has a gap. A crash between the secondary write and the journal append loses the entry. Verification finds those keys. Uploads that use an UploadControl go through the adapter’s resumableUpload driver, which the wrapper doesn’t intercept, so they aren’t journaled either.
• Conflict detection relies on clocks. It compares the app’s clock, which stamps the journal, with the primary’s lastModified. Clock skew between the two can hide a conflict or report a false one. Leave a margin, or flag every journaled key the primary changed after the outage started.
• The reconcile copies body, content type, and metadata. It doesn’t copy Cache-Control, because FileInfo has no field for it. Migrate S3 files to R2 shows a plugin that carries it.
• Keep the direct handles plugin-free. On the app’s instance, list encryption() or compression() before failover() in plugins, so the same stored bytes reach both backends. The reconcile’s plain primary and secondary instances then copy those stored bytes and their fsenc_ or fscmp_ metadata as they are, without decoding them.
• Conditional operations are refused on a failover instance, before any I/O. See conditional operations.
• The options you can use are limited to what both backends support. files.capabilities is the intersection of the backends, so an option the secondary lacks is refused even while the primary is healthy.
• This drill doesn’t test a secondary outage while the primary is healthy (nothing fails over, and nothing needs reconciling), or both backends down (the last error is thrown).

Troubleshooting

Every call takes about two seconds during the drill. The primary is hung rather than stopped, so each call waits for timeout before failing over. Lower the timeout, or switch instances as described under Limits and tradeoffs.

Calls hang forever when the primary hangs. No timeout is set. Set one on the Files constructor. The plugin applies it to the secondary too. See Timeouts.

Unauthorized with no failover. The primary rejected the credentials, which the plugin treats as a definitive answer. Fix the credentials. Pass a custom shouldFailover only if you really want auth failures to move traffic.

An upload fails with connect ECONNREFUSED even though failover() is installed. The body was a ReadableStream, which only goes to the primary. Pass bytes, a string, or a Blob instead.

Download links and presigned uploads fail during the outage. They were signed against the primary without a network call. Serve downloads through download() from your server, which does fail over, until the primary is back.

Multipart, progress, and unknown-length stream uploads on S3 require the optional peer dependency '@aws-sdk/lib-storage'. The reconcile streams each copy into the primary. Install @aws-sdk/lib-storage.

verify.ts keeps reporting the same keys under different. Those are conflicts, or writes that weren’t journaled. Decide which copy is right, write it to the primary, and re-seed the secondary.