Schema Migration

Canonicalization of Rune's event journal and pins store: old-dialect data is migrated ONCE into the canonical orchestration schema, and the legacy writers delegate to the canonical stores (no dialect bazaar).

The migration contract

Old fixtures migrate once. old_fixture_to_new is idempotent by event_id: a re-run converts only the not-yet-converted entries, so a partial or interrupted migration resumes cleanly.
The new store rejects unmigrated dialects. DurableEventStore.append raises UnmigratedDialectError for any old-dialect dict (legacy payload/source/ channel/authority_class/task_id fields, schema_version < 1, or a garbage "canonical"-shaped dict) with an actionable message pointing at migrate.old_fixture_to_new().
Legacy writers delegate to canonical records. event_bus.emit and pins.pin write canonical EventEnvelope / PinRecord rows through the canonical stores — the CLI/pins.py surfaces are projections, never a second dialect.

The one-dialect rule

There is exactly ONE event dialect the store accepts: the canonical EventEnvelope (event_id, schema_version, event_type, partition_key, payload_hash, timestamp). Old data must pass through the migration before it can be appended. The migration CLI is the sanctioned path for converting a legacy JSONL fixture; the legacy event_bus.emit shim is the sanctioned path for NEW writes (it produces canonical records directly).

The migration CLI

python -m rune.orchestration.migrate_cli <fixture> [--store DIR]
python -m rune.orchestration.migrate_cli --pins <legacy.db> <out.json>

Fixture form: converts an old-dialect JSONL fixture to the canonical store (default <fixture dir>/events; --store DIR overrides the event store directory) and prints MIGRATED N. A re-run prints MIGRATED 0 — migration is idempotent.
--pins form: converts legacy SQLite wing='pins' chunks to a canonical PinRecord JSON array and prints PINS MIGRATED N.
Both exit 0 on success. A malformed / non-dict fixture line fails with a readable error (exit 1) instead of a traceback.
The CLI works with only /usr/bin:/bin on PATH — it is invoked via python -m, never the shell.

Evidence store (X1 pin discharge)

The canonical envelope carries no payload body (payload_hash is one-way). So old_fixture_to_new ALSO writes each migrated raw line to the EvidenceStore keyed by its payload_hash (digest == payload_hash — a 1:1 link). A store-only consumer can recover the original body:

from rune.orchestration.evidence_store import EvidenceStore
body = EvidenceStore(<fixture dir>/evidence).get(rec["payload_hash"])

EvidenceStore.put hashes the same canonical bytes (sha256: + sha256(json.dumps(raw, sort_keys=True))) as the event record's payload_hash, so the evidence key and the event record always agree.

source.channel extraction

Real event_bus data nests the channel INSIDE source (source='{"channel": "chat"}' or a dict). old_fixture_to_new extracts source.channel when present, falling back to the top-level channel and then conversation — without this, real data would be silently repartitioned.

Legacy pins conversion (X4 C7 discharge)

The legacy hybrid-store writer is gone, so existing pins live in a SQLite chunks table (text/wing/room columns) as multi-line text blobs:

[topic] guitar course
[open question] when?
[outcome] launched
[status] open
[context] ctx

legacy_pins_to_new(legacy_db, out_json) reads the wing='pins' chunks, parses this [key] value format, and writes a JSON array of canonical PinRecord dicts (kind='pin', item_id=topic, owner='pins', state=open|resolved, details = the question/outcome/context/reason blob, sort_keys=True). Run it once before/after shipping so existing pins are not orphaned.

Verify

PYTHONPATH=src:. python3 -m pytest tests/test_orchestration_migration_canary.py -q
PYTHONPATH=src:. python3 -m pytest tests/ -q --ignore=tests/test_message_widgets.py
python3 src/rune/canary_schema_migration.py   # SCHEMA MIGRATION CANARY PASSED

The canary walks the whole path in a tempdir: migrate once (1), idempotent re-run (0), canonical store rejects a fresh old-dialect append with the typed error, and the legacy event_bus.emit shim is proven WIRED through the canonical store (a user.message emitted via the legacy API replays back from a DurableEventStore at the same tempdir).

4.4 KiB Raw Blame History