Persistence
Persistence saves JSON tracks to your storage while they are read through it, and gives the saved documents to whoever reads them, also after every writer has left.
Paths
{path} is relative to the reader's root; its project path is the same path from the project root.
| Who | Path | Track | Content |
|---|---|---|---|
| A reader | .persistence/{path} |
{track} |
The saved document of {path}'s {track}, a JSON snapshot track: its first value is the saved document, or null when none is saved; then each newly saved version |
| Anyone | {path} |
{track} |
The source: a JSON snapshot track |
| Storage | {STORAGE_PREFIX}{project path}/{track} |
The document as UTF-8 JSON, replaced at each save |
Each path segment is one key segment, percent-encoded where it holds a control character, a line break or one of \, {, }, ^, %, [, ], ", <, >, ~, #, |, *, ? and the backtick, or where it is . or ... Reading .persistence/{path} shows what {path} holds, so grant it as you grant {path}.
// Saved JSON through Persistence: notes publishes notes.json, and reading .persistence/notes keeps it
// saved in your storage and gives the saved document first, also after everyone has left.
import { type Connection, json } from 'tablebox.io';
export function write(connection: Connection) {
const notes = new json.Snapshot.Producer({ track: connection.publish('notes').createTrack('notes.json') });
return (value: unknown) => notes.update(value);
}
export async function read(connection: Connection, show: (notes: unknown) => void) {
const saved = await connection.read('.persistence/notes');
// The first value is the saved document, or null when nothing is saved yet; then each saved version.
for await (const notes of new json.Snapshot.Consumer({ track: saved.track('notes.json').subscribe() })) show(notes);
}Saving
Reading .persistence/{path} track {track} is what chooses it: Persistence then reads the source whenever it is published and saves it 2 s after it changes, at least every 10 s while it keeps changing, and at once when its publisher leaves. This lasts while the output is read; 20 s after the last reader leaves, Persistence saves what is still unsaved and stops. A reader's first value is the whole saved document, whether or not the source is there.
Storage
Persistence writes to an S3-compatible bucket of yours, named by the project's variables:
| Variable | Meaning |
|---|---|
STORAGE_BUCKET |
The bucket, required |
STORAGE_ACCESS_KEY_ID, STORAGE_SECRET_ACCESS_KEY |
The credentials, required |
STORAGE_ENDPOINT |
An https:// S3-compatible endpoint; AWS S3 for the region when unset |
STORAGE_REGION |
The region; us-east-1 when unset |
STORAGE_PREFIX |
Prepended to every key as written; empty when unset |
The saved objects are plain JSON: read them with any S3 client. An object you replace is what Persistence gives the next time it starts reading that path. Egress records to the same storage, and Aggregation saves its merged documents through Persistence, under keys starting with .aggregation/.
Failures
- A required storage variable is missing: the track is refused with
missing-variable; reading again after the variable is set works. - Storage fails: Persistence retries with growing delays while it runs; a first value waits for the first successful read, and later versions for the next successful save.
STORAGE_ENDPOINT's host is or resolves to an address inside a network rather than on the internet (loopback, private, link-local and the other special-purpose ranges, also when an IPv6 address carries one): the track is refused withbad-request.- The source is no JSON snapshot track: the track is refused with
bad-request. - An instance stops: reads end, and reading again reaches the instance that takes over; changes after the last save are saved again if the source is still published, and lost (at most 10 s of them) if it has gone.
- Nobody reads
.persistence/{path}for 20 s while the source keeps changing: later changes are saved once someone reads it again.