Skip to content
Files SDK
Esc
↑↓navigate↵open⌘Jpreview
On this page

tiering

Route operations between a hot and a cold adapter by size, prefix, or age. Uploads land in the right store, reads transparently find them again, and tier() moves objects between stores - body-transparent, no native dependencies.

The built-in tiering() plugin spreads one logical bucket across two adapters - a hot tier for what you reach for often and a cold tier for what you rarely touch. An upload lands in the tier your route function picks; every read transparently finds it again. The hot tier is the instance’s own adapter (reached through the rest of the pipeline); the cold tier is a second adapter you pass in.

It’s body-transparent: it never buffers or transforms bytes - a cross-tier copy streams straight through - so streaming, range downloads, url(), and signedUploadUrl() all keep working. It has no native dependencies and works on any pair of adapters.

import { createFiles } from "files-sdk";
import { s3 } from "files-sdk/s3";
import { tiering } from "files-sdk/tiering";

const files = createFiles({
  adapter: s3({ bucket: "hot" }), // hot tier
  plugins: [
    tiering({
      cold: s3({ bucket: "cold" }),
      // archives go cold; everything else stays hot
      route: ({ key }) => (key.startsWith("archive/") ? "cold" : "hot"),
    }),
  ],
});

await files.upload("photo.jpg", body); // → hot
await files.upload("archive/2019.zip", zip); // → cold
await files.download("archive/2019.zip"); // transparently read from cold

The route function

route is handed { key, size? } and returns "hot" or "cold". It’s called once per logical operation:

  • on upload, with size set to the body’s declared byte length when it’s known up front (a string, Blob, ArrayBuffer, or typed array - a streaming body has no declared length);
  • on every other decision (reads, delete, the destination of copy / move, signedUploadUrl, and locating an object), with size omitted.

Route by prefix and the decision is a pure function of the key, so reads land on the right tier first try with zero overhead:

route: ({ key }) => (key.startsWith("cold/") ? "cold" : "hot");

Route by size and writes go by size, but reads can’t recompute it - so turn on fallback and a read that misses the guessed tier checks the other:

tiering({
  cold,
  route: ({ size }) =>
    size !== undefined && size > 5_000_000 ? "cold" : "hot",
  fallback: true,
});

What each verb does

  • upload routes by route({ key, size }) and writes to that tier.
  • download / head / url / exists consult the routed tier.
  • delete removes the routed tier’s copy.
  • copy / move locate the source, route the destination by its key, and use a native same-tier op when both keys land in one tier - or stream the bytes across (preserving content type and metadata) when they differ. A move then deletes the source.
  • list merges a page from each tier into one result, keys sorted within the page (see merged listing).
  • signedUploadUrl signs against the tier route({ key }) picks. The resulting direct upload bypasses the plugin, so it can’t be size-routed or deduplicated.

fallback

By default routing is deterministic: every operation touches exactly the one tier route names, with no extra round-trip. That’s the right mode for prefix / key-based routing.

Set fallback: true to treat an object’s tier as discoverable rather than fixed:

  • a read that misses the routed tier retries the other tier;
  • delete removes the key from both tiers;
  • an upload evicts the key from the other tier, so a re-upload that flips tiers leaves exactly one copy.

Turn it on whenever the tier isn’t a pure function of the key - size-based routing, or when you move objects between tiers with tier(). The cost is at most one extra round-trip on a cold read.

Moving objects between tiers

Two methods land on the instance via extend:

await files.tierOf("photo.jpg"); // "hot" | "cold" | undefined
await files.tier("photo.jpg", "cold"); // stream it to cold, drop the hot copy

tier(key, target) is the lever for age-based transitions, which can’t be a write-time decision (a new object’s age is zero). Run it on a schedule: list, check each object’s lastModified, and tier down what’s gone cold.

for await (const file of files.listAll()) {
  const age = Date.now() - (file.lastModified ?? 0);
  if (age > THIRTY_DAYS) {
    await files.tier(file.key, "cold");
  }
}

Because tier() moves objects to a tier route wouldn’t pick from the key alone, pair it with fallback: true so reads still find what you’ve moved.

Merged listing

list() returns objects from both tiers. It fetches a page from each, deduplicates (hot wins), sorts by key, and paginates the two tiers independently behind one composite cursor - so listAll() walks the whole namespace across both stores.

Two things to know: a page can hold up to the sum of both tiers’ page sizes (each tier’s limit is applied per tier), and keys are sorted within a page but - because the tiers paginate independently - not globally across pages. For a single fully-ordered enumeration, list an adapter directly.

Ordering and prefixes

  • Place it last (innermost). Body-transforming plugins like encryption() and compression() wrap tiering() and transform the op on the way in, so they apply to both tiers:

    plugins: [encryption(key), tiering({ cold, route })];

  • Address objects by caller-facing keys. The cold adapter does not receive the instance prefix, so configure its own bucket / container and avoid a client prefix on a tiering instance.

Things to keep in mind

  • The cold tier is a real store. Cold reads pay its latency; a hot→cold tier() or cross-tier copy transfers the bytes between adapters.
  • Presigned uploads bypass routing. A signedUploadUrl() upload lands directly in the routed tier and isn’t size-routed or deduplicated.
  • Without fallback, routing must be stable per key. If an object can live in a tier the key wouldn’t route to (size-based routing, or a tier() move), enable fallback or reads will miss it.

Was this page helpful?