Skip to content
Files SDK
Esc
↑↓navigate↵open⌘Jpreview
On this page

tiering

Route operations between a hot and a cold adapter by size, prefix, or age - reads transparently find objects and tier() moves them between stores.

The built-in tiering() plugin spreads one logical bucket across two adapters - a hot tier for what you reach for often and a cold tier for what you rarely touch. An upload lands in the tier your route function picks; every read transparently finds it again. The hot tier is the instance’s own adapter (reached through the rest of the pipeline); the cold tier is a second adapter you pass in.

It’s body-transparent: it never buffers or transforms bytes - a cross-tier copy streams straight through - so streaming, range downloads, url(), and signedUploadUrl() all keep working. It has no native dependencies and works on any pair of adapters.

import { createFiles } from "files-sdk";
import { s3 } from "files-sdk/s3";
import { tiering } from "files-sdk/tiering";

const files = createFiles({
  adapter: s3({ bucket: "hot" }), // hot tier
  plugins: [
    tiering({
      cold: s3({ bucket: "cold" }),
      // archives go cold; everything else stays hot
      route: ({ key }) => (key.startsWith("archive/") ? "cold" : "hot"),
    }),
  ],
});

await files.upload("photo.jpg", body); // → hot
await files.upload("archive/2019.zip", zip); // → cold
await files.download("archive/2019.zip"); // transparently read from cold

The route function

route is handed { key, size? } and returns "hot" or "cold". It’s called once per logical operation:

  • on upload, with size set to the body’s declared byte length when it’s known up front (a string, Blob, ArrayBuffer, or typed array - a streaming body has no declared length);
  • on every other decision (reads, delete, the destination of copy / move, signedUploadUrl, and locating an object), with size omitted.

Route by prefix and the decision is a pure function of the key, so reads land on the right tier first try with zero overhead:

route: ({ key }) => (key.startsWith("cold/") ? "cold" : "hot");

Route by size and writes go by size, but reads can’t recompute it - so turn on fallback and a read that misses the guessed tier checks the other:

tiering({
  cold,
  route: ({ size }) =>
    size !== undefined && size > 5_000_000 ? "cold" : "hot",
  fallback: true,
});

What each verb does

  • upload routes by route({ key, size }) and writes to that tier.
  • download / head / url / exists consult the routed tier (with fallback, url() signs against whichever tier holds the key).
  • delete removes the routed tier’s copy.
  • copy / move locate the source, route the destination by its key, and use a native same-tier op when both keys land in one tier - or stream the bytes across (preserving content type and metadata) when they differ. A move then deletes the source.
  • list merges a page from each tier into one result, in key order (see merged listing).
  • signedUploadUrl signs against the tier route({ key }) picks. The resulting direct upload bypasses the plugin, so it can’t be size-routed, and with fallback the other tier’s stale copy isn’t evicted.

fallback

By default routing is deterministic: every operation touches exactly the one tier route names, with no extra round-trip. That’s the right mode for prefix / key-based routing.

Set fallback: true to treat an object’s tier as discoverable rather than fixed:

  • a read that misses the routed tier retries the other tier;
  • url() first checks which tier holds the key and signs against that one - a presigning adapter mints a URL without checking the object exists, so it can’t discover a miss by itself, and a URL for the wrong tier would be a dead link;
  • delete removes the key from both tiers;
  • an upload evicts the key from the other tier, so a re-upload that flips tiers leaves exactly one copy.

Turn it on whenever the tier isn’t a pure function of the key - size-based routing, or when you move objects between tiers with tier(). The cost is at most one extra round-trip on a cold read, and one or two exists probes on each url().

Moving objects between tiers

Two methods land on the instance via extend:

await files.tierOf("photo.jpg"); // "hot" | "cold" | undefined
await files.tier("photo.jpg", "cold"); // stream it to cold, drop the hot copy

tier(key, target) is the lever for age-based transitions, which can’t be a write-time decision (a new object’s age is zero). Run it on a schedule: list, check each object’s lastModified, and tier down what’s gone cold.

for await (const file of files.listAll()) {
  const age = Date.now() - (file.lastModified ?? 0);
  if (age > THIRTY_DAYS) {
    await files.tier(file.key, "cold");
  }
}

Because tier() moves objects to a tier route wouldn’t pick from the key alone, pair it with fallback: true so reads still find what you’ve moved.

tier() runs directly against the two tiers’ adapters rather than through the instance, so hooks and the other plugins on the instance (audit(), cache(), usage(), …) don’t see the move. The stored bytes and metadata move verbatim, so body-transforming plugins keep working; invalidate caches or record audits yourself if you need them. On a read-only instance, tier() throws ReadOnly instead of moving anything (tierOf() still works).

Merged listing

list() returns objects from both tiers. It fetches a page from each, deduplicates (hot wins), sorts by key, and paginates the two tiers independently behind one composite cursor - so listAll() walks the whole namespace across both stores.

Keys come back in global key order across pages. A page may hold fewer than limit entries - entries past what both tiers have covered are held back for the next page - or up to the sum of both tiers’ limits (each tier’s limit is applied per tier).

Ordering and prefixes

  • Place it last (innermost). Body-transforming plugins like encryption() and compression() wrap tiering() and transform the op on the way in, so they apply to both tiers:

    plugins: [encryption(key), tiering({ cold, route })];

  • Address objects by caller-facing keys. The cold adapter does not receive the instance prefix, so configure its own bucket / container and avoid a client prefix on a tiering instance.

  • Instance defaults apply to both tiers. The timeout, retries, and signal you pass to the constructor govern the cold tier (and tierOf() / tier()) as well as the hot tier, so a hung cold backend is cut off rather than stalling the operation.

Things to keep in mind

  • The cold tier is a real store. Cold reads pay its latency; a hot→cold tier() or cross-tier copy transfers the bytes between adapters.
  • Presigned uploads bypass routing. A signedUploadUrl() upload lands directly in the routed tier: it isn’t size-routed, and with fallback the other tier’s stale copy isn’t evicted.
  • Conditional operations are refused. Cross-tier routing can’t preserve one native compare-and-set, so every conditional mode throws before any I/O, and files.capabilities reports every conditional flag as false.
  • Without fallback, routing must be stable per key. If an object can live in a tier the key wouldn’t route to (size-based routing, or a tier() move), enable fallback or reads will miss it.

Was this page helpful?