AZ Prep: standalone canonical preparation and storage

Original independent Rust code, licensed MIT OR Apache-2.0 at your option. This package is its own Cargo workspace. It does not depend on any Lightbridge crate. The eventual GPL AZ-Mods plugin is separate and is not implemented here. No native firmware is deployed by this project. No credentials, device access, network discovery, cloud account, or network connection are required at runtime. All tests use authored WAV signals, synthetic bytes and metadata, never hardware captures.

Implemented

• A project-owned SQLite schema v2 (az-prep.sqlite3, application ID AZPR) using bundled SQLite, foreign keys, WAL, FULL synchronization and transactions. Schema v1 upgrades transactionally to v2 without rewriting old immutable JSON payloads. Other schema/application versions at that filename are refused. This is an independent canonical library, not a rekordbox-compatible database.
• SHA-256 content-addressed local audio import: opaque file bytes are streamed through a 64 KiB buffer to a temporary file, synced and checksum-verified, then published without replacing an existing object. Identical bytes deduplicate. An existing object is checked before reuse. Import preserves opaque bytes; analysis is a separate, explicit operation on the canonical object.
• Persistent playlists with explicit ordinal positions, duplicate tracks, names, and optimistic revision preconditions. Creation requires None/new; updates require the exact existing revision. Invalid references roll back the write.
• Immutable analysis revisions with parent links, origin labels, tempo, beat grids, cues, musical key, independent meter, downbeats, labelled phrase spans, loops, variable-tempo segments and a separate manual override snapshot. Optional detailed analysis artifacts retain analyzer provenance and confidence. Reanalysis preserves active manual overrides. A manual field takes priority over the analysis field. save_corrections replaces the entire override snapshot: None clears an override, while an empty grid/cue array explicitly overrides to empty. Older versions remain available. Comparison includes both stored versions and a list of changed effective fields; activation uses a track generation CAS. Activation always increments the generation, including switching back, so stale writers cannot bypass the precondition through an ABA sequence.
• Additive, transactional AZ Prep canonical JSON v2 metadata export/import, preserving full detailed artifacts. Legacy v1 imports remain supported; a v1 document containing populated v2 musical fields or a detailed artifact is rejected, requiring an explicit v2 declaration. The format identifier is az-prep.canonical; use a filename such as my-set.az-prep.canonical.json. Exports contain no audio bytes, transfer paths, or native database records. Import requires the expected current library revision, validates unique IDs, track/playlist/analysis references, ownership of active revisions and parents, acyclic history, valid finite analysis values, and SHA-256/size of every referenced local audio object. Immutable analysis ID differences, lower entity generations, and divergent equal generations conflict. A higher incoming generation explicitly selects the incoming track/playlist state under the library CAS; absent local entities and history are retained. library_revision in the document is provenance, not permission to rewind the local library revision. A successful merge increments the local library revision once. Transfer journaling does not change that canonical metadata revision.
• Persistent per-file local transfer jobs, checkpointed copying, prefix and final SHA-256 checks, explicit retry/restart/cancel, error recording, verified publication, and recovery after an interrupted journal commit. A library transaction serializes workers across connections/processes. Destinations are reserved case-insensitively within a library. Jobs never replace a final pathname. Only one portable ASCII filename is accepted, excluding traversal, directory components, drive/ADS syntax, dotfiles and DOS reserved aliases.
• Explicit resource policy: the library receives caller-supplied PlaybackState. Any positive active_decks pauses heavy imports, canonical object validation, transfer creation/copying/verification/publication. Streaming operations poll the callback before each buffer. Transfer pause preserves the checkpoint; a paused import discards its temporary file and can be restarted. Metadata edits and status reads remain available. There is no inferred playback state.
• CLI commands for the operations above, with JSON output and actionable errors. Playback pause exits with code 3; other failures exit with code 1.

Local audio analysis

Library::analyze_track(track_id, expected_revision, AnalysisConfig, callback) uses the independent analysis/ crate to stream-decode MP3, FLAC and PCM WAV from the canonical content-addressed object. The callback is FnMut(&AnalysisProgress) -> AnalysisWorkState { playback, control }; any active deck overrides Continue/Throttle and pauses the operation. Continue, bounded Throttle, Pause and Cancel are supported. Paused/cancelled work returns no revision and can be restarted from the beginning. No real decks are discovered.

The source SHA-256/byte count must match the Track. The complete-preparation API rejects truncated duration/sample coverage instead of activating a partial result. The default analysis duration limit is 1800 seconds; longer sources require an explicitly larger configuration within the worker's hard bounds. Hashing, decoding, feature analysis, report validation and the final pressure callback run before acquiring a SQLite write transaction. Saving checks the expected track generation again under the transaction and retains every active manual field. A competing edit wins normally; stale background work cannot overwrite it. Scheduling is cooperative and not a realtime/latency guarantee.

AnalysisRevision.details is optional for compatibility with externally supplied and old analyses. Real results contain decoded media facts, source hash, heuristic tempo/key confidence, actual onset-supported beat candidates, local tempo windows, generic structural novelty candidates, analyzer version, methods, limitations, the full config and its SHA-256 digest. Corrections preserve the artifact; replacing the underlying analysis with externally supplied values creates a new revision without an unrelated old artifact. analysis-get and canonical v2 export/import retain the entire artifact.

Beat candidates can have gaps; the legacy beat_grid_seconds projection does not turn these into a complete measured DJ grid. The full artifact identifies onset support. Meter, bar phase, downbeats and semantic phrase labels remain unknown/empty in this analyzer. Structural energy-change candidates are not automatically converted into cues or semantic phrases. Confidence is heuristic, not calibrated probability, and no rekordbox accuracy/equivalence is claimed. Manual/imported time signatures, downbeats, phrases and loops can be stored and corrected independently. No waveform rendering or preparation UI is included.

See AZ Prep Analysis for the exact bounded algorithms, worker controls, failure handling and limits. Original AZ Prep code remains MIT/Apache-2.0. Symphonia 0.6.1 is an MPL-2.0 dependency, with its own notices and source-availability obligations when distributed; it is not relicensed by the original code's license. No dependency source is vendored or modified, no model downloads run, and no native media database is opened/written.

Local layout and recovery contract

LIBRARY/
  az-prep.sqlite3       # canonical metadata and durable transfer journal
  az-prep.sqlite3-wal   # SQLite-managed, when present
  az-prep.sqlite3-shm   # SQLite-managed, when present
  objects/<sha256>     # immutable audio bytes; no extension implied
  staging/             # import temporary files
DESTINATION/
  .az-prep-stage-<job-id>/payload
  chosen-filename      # appears only after complete size/hash verification

Library and destination roots must be application-controlled local directories. The destination must exist and be outside the library. Symlinked roots, object files, stage directories and payload files are rejected where checked. Paths are canonicalized; no snapshot-supplied absolute audio paths are used. This is not a sandbox against another local process replacing directory components, mounts or files concurrently. SQLite locking coordinates this library's workers, not unrelated writers or a different library. Keep unrelated tools from editing these roots; use private permissions. Live SQLite backups must use SQLite-aware backup/snapshot procedures; copying only the main database with a live WAL is not a backup feature provided here.

Each copy step writes at most 64 KiB, calls sync_all, then commits the new offset. Before resuming, bytes beyond the last committed offset are truncated and committed prefixes are checked against the source. Thus an interrupted or partial/full-disk write does not skip uncommitted bytes. A shorter or corrupted committed prefix fails closed and requires explicit restart. Prefix and final verification use bounded buffers and poll playback, but can scan a whole prefix/file within one step; step wall time is not bounded. Repeated prefix scans deliberately favor correctness over throughput and can be quadratic in file size. These calls allocate and block and belong on a background worker, never a frame/audio evaluation thread. No latency, CPU or jitter improvement is claimed.

After the full checksum passes, a separate ready journal checkpoint precedes publication. Publication atomically renames the stage to the chosen final filename without replacing any existing entry. Linux uses renameat2(RENAME_NOREPLACE) through rustix; Windows uses MoveFileExW with MOVEFILE_WRITE_THROUGH, without MOVEFILE_REPLACE_EXISTING or MOVEFILE_COPY_ALLOWED. Staging and final paths must be on the same filesystem. Hard links are not required, including for audio import and canonical JSON export. Linux kernels/filesystems lacking no-replace rename return an error; other operating systems explicitly return Unsupported. There is no check-then-overwrite, hard-link or partial-file copy fallback. Target filesystem acceptance (including FAT/exFAT) still needs operator validation. This is a local file-copy foundation, not a native AZ USB export implementation.

If the process exits after publication but before complete commits, recovery accepts only a complete durable byte checkpoint and a size/hash-verified regular destination. The renamed payload need not remain; an absent empty stage directory also permits recovery. An existing stage is rechecked before publication/adoption. Failed/paused jobs with a full checkpoint and verified final file first recover to ready, then complete. A matching late arrival can likewise be adopted without replacing it. Any other collision fails and leaves existing bytes intact. Restart refuses an existing destination. Complete stages are never opened for writing, and no copy phase truncates a stage while a final pathname exists, preserving legacy recovery where stage and final may already share a hard link.

failed jobs can be retried with another step after resolving an I/O error or freeing disk space; they retain their reservation and durable offset. Integrity failures generally require restoring the source and transfer-restart to discard the checkpoint. If SQLite itself cannot commit an error (for example the disk also filled its WAL), the caller receives the database error and the previous journal checkpoint remains authoritative. A job becomes complete only after verified publication; metadata does not advertise partial output.

Cancellation releases the reservation and never removes a final file. Stage payloads are moved on successful publication; empty stage directories and cancelled/legacy stage files are retained, and no automatic garbage collector runs. A collision adopted as identical can also leave its stage payload behind. A crash before initial job commit can leave an orphan stage directory; an import can leave an unreferenced object after file publication but before metadata commit. They are never automatically adopted/deleted. Partial import temp files may survive a process kill and need manual cleanup. Do not edit retained legacy stages: they can share the final inode.

File contents are synced and SQLite uses FULL synchronization. Directory entries in both affected parents are synced on Unix after publication. Windows requests write-through for the move, but Rust std directory synchronization is not implemented here, so crash recovery is provided subject to filesystem persistence; power-loss durability of Windows directory entries is not guaranteed. Hostile storage, media removal, filesystem corruption, free-space reservation, cross-filesystem batch atomicity, automated cleanup, and whole-library backup are outside this slice. Keep source audio until transfer verification succeeds.

CLI examples

Run from this directory; <track-id> and <job-id> are returned in JSON. The final integer for heavy commands is the caller's active-deck count.

cargo run -- init ./local/library
cargo run -- import-audio ./local/library C:/music/example.wav 0
cargo run -- playlist-put ./local/library my-set new "My set" <track-id>
cargo run -- playlist-get ./local/library my-set
cargo run -- playlist-put ./local/library my-set 1 "My reordered set" <track-id>
cargo run -- analysis-save ./local/library <track-id> 1 external-analysis ./analysis.json
cargo run -- analyze ./local/library <track-id> 1 0
cargo run -- corrections-save ./local/library <track-id> 2 ./overrides.json
cargo run -- analysis-compare ./local/library <analysis-id-1> <analysis-id-2>
cargo run -- analysis-activate ./local/library <track-id> 3 <analysis-id-1>
cargo run -- canonical-export ./local/library ./local/my-set.az-prep.canonical.json
cargo run -- status ./local/library
cargo run -- canonical-import ./local/library ./local/my-set.az-prep.canonical.json <current-library-revision> 0
cargo run -- transfer-create ./local/library <track-id> C:/exports track.wav 0
cargo run -- transfer-step ./local/library <job-id> 0 100
cargo run -- transfer-list ./local/library
cargo run -- transfer-verify ./local/library <job-id> 0
cargo run -- transfer-restart ./local/library <job-id>
cargo run -- transfer-cancel ./local/library <job-id>

Create C:/exports yourself before transfer-create; it is an explicit local destination, not an auto-detected device. The filename is caller-selected; the bytes are copied unchanged and no extension/codec conversion is performed. transfer-step defaults to one phase/chunk; its optional count is capped at 100000, and it stops when complete/cancelled. The CLI's deck count is a snapshot for that invocation. A live integration must use the library callback to report changing playback state; no command monitors real decks.

Analysis/overrides JSON uses this shape (all fields may be omitted or null):

{
  "tempo_bpm": 123.0,
  "beat_grid_seconds": [0.5, 1.0, 1.5],
  "cues": [{"seconds": 3.5, "label": "Manual cue"}],
  "key": {"tonic": "Db", "mode": "minor"},
  "meter": {"numerator": 3, "denominator": 8},
  "downbeat_seconds": [0.5, 2.0, 3.5],
  "phrases": [{"start_seconds": 0.5, "end_seconds": 8.0, "label": "Reviewed span", "provenance": "manual"}],
  "loops": [{"start_seconds": 1.0, "end_seconds": 3.0, "label": "Manual loop"}],
  "tempo_segments": [{"start_seconds": 0.0, "end_seconds": 8.0, "tempo_bpm": 123.0, "confidence": 1.0, "provenance": "manual"}]
}

All musical fields are optional: missing/null inherits the underlying analysis; an empty vector explicitly clears that field. Replacing the whole correction snapshot with all fields missing clears the overrides. Key modes support major/minor, the seven church modes (major/ionian uses major, minor/aeolian uses minor) and unknown; key pitch spellings use common naturals/sharps/flats. Unknown overall key/meter is represented by a missing/null field.

Grid/downbeat positions must be finite, nonnegative and strictly increasing; cue positions must be finite/nonnegative; tempo must be finite/positive. A meter has numerator 1..32 and denominator a power of two in 1..64. Intervals require finite nonnegative start < end. Phrase and tempo spans are ordered/nonoverlapping; alternative loops may overlap but are strictly sorted by (start,end). When a detailed artifact gives source coverage, all stored coordinates must fit it. Confidence is finite in [0,1]; an unknown tempo segment uses null BPM.

Detailed artifacts validate source identity, schema, config digest, rate/channels, frame/duration consistency, resource limits, confidence and event order before storage/import. Limits are 100000 timestamps per grid/event list, 8192 cue/region/ boundary/segment entries per list, 128 tags/warnings/method strings per list, 12 MiB per detailed report, 14 MiB per revision and 16 MiB per canonical input. Canonical entity lists and playlist entries are capped at 100000 each. Strings are bounded and NUL-free. These are bounded metadata inputs, not unlimited serialization of arbitrary analyzer output.

analyze TRACK_ID EXPECTED_REVISION ACTIVE_DECKS [CONFIG_JSON] accepts an optional full AnalysisConfig JSON file. The CLI deck count is a snapshot; live callers must update playback state through the callback. Configuration controls resource limits and optional key/local-tempo/structure work; it does not imply accuracy. The CLI caps input metadata JSON at 16 MiB. JSON export is staged/synced and published without replacing an existing file. To merge a snapshot into another library, first import the matching audio objects locally, then supply that library's current revision. Import never opens a native library database.

Upcoming / blocked / not implemented

• No validated bar/downbeat/semantic phrase recognition, commercial-analyzer accuracy parity, waveform UI or automatic semantic cue placement is claimed.
• Native record writers upcoming: neither export.pdb nor exportLibrary.db writing exists. No native database reader/migration, rekordbox compatibility, native media export, or hardware validation is claimed.
• No firmware deployment, AZ-Mods plugin, hardware transport, deck playback, authorization/arming of physical output, or device I/O exists here.
• No credential storage, cloud/network features, multi-file atomic transfer batches, validated target-media compatibility, or automatic orphan cleanup is implemented.

Verification commands

From the containing checkout root:

cargo fmt --manifest-path projects/trara-az/az-prep/Cargo.toml --all --check
cargo clippy --manifest-path projects/trara-az/az-prep/Cargo.toml --workspace --all-targets -- -D warnings
cargo test --manifest-path projects/trara-az/az-prep/Cargo.toml --workspace

The integration suite additionally uses generated WAV clicks/silence to cover real analysis, source integrity, complete-coverage rejection, detailed artifact persistence/canonical v2 roundtrip, nondestructive schema v1 migration, legacy canonical v1 validation, new musical override priority/coordinate checks, playback pause/cancellation and callback-time CAS conflicts. It also covers playlist persistence/order/duplicates, competing writers, stale/ABA revisions, manual priority/history, invalid timing data, pause/stale playback input changes, empty/exact/partial copy boundaries, path traversal/aliases/collisions, corrupted/missing prefixes, checksum mismatch, resumed copying, concurrent no-replace publication, legacy hard-link recovery, rename-before-journal recovery with a missing stage, failed/paused recovery, temporary-file cleanup, synthetic SQLite commit failures, canonical validation/conflicts/rollback after SQL writes, schema refusal and CLI workflows. Unix also tests symlink rejection. No real disk was filled; interruption/partial suffixes and journal failures are synthetic fault fixtures. Full-disk, power-cut and target-filesystem acceptance remain operator validation work, not completed hardware tests. Run the commands above for the current checkout; the containing project's verification record reports scope.