Skip to content

Persistence and Artifact Storage

Mozaiks is a framework with a durable build pipeline, not a stateless prompt wrapper. Persistent storage is therefore a first-class runtime contract.

Core Rule

  • Durable persistence is required for Studio and the workflow-owned build sequence, factory_app, multi-stage build workflows, refinement, and revision history.
  • In-memory execution is only acceptable for smoke tests, demos, or simple non-builder workflows that do not need upstream artifacts.
  • MongoDB is the canonical persistence backend today.

Ownership Layers

Mozaiks persistence is divided into three scopes.

1. Runtime State

Framework-owned operational data used by the runtime itself:

  • ChatSessions
  • AG2StreamEvents
  • AG2StreamHeads
  • GeneralChatSessions
  • GeneralChatCounters
  • RuntimeUsageEvents

This data supports session continuity, workflow execution state, AG2 stream bootstrap/replay, and reconnectable UI state.

Current implemented workflow-run persistence contract:

  • ChatSessions is run metadata and UI-state projection, not canonical execution history.
  • Its WorkflowStatus is 0 (in progress, including pauses), 1 (completed), or 2 (failed). Failure writes failed_at, duration, and a session-version increment before failure hooks or terminal UI delivery, and clears pending input without discarding artifacts. A terminal status cannot be overwritten by another outcome. Only completed sessions satisfy workflow prerequisites.
  • Failed sessions reject websocket reconnection and execution input. Both completed and failed sessions reject direct orchestration re-entry. Pauses, admission denials, lease loss, and cancellation do not mark a session failed. Retrying a failed workflow requires a new session, not replay of that run.
  • A closed chat-scoped AG2 channel also blocks opening a replacement channel, even for older session projections left at 0. This uses AG2's existing WAL, not Factory build receipts or a second runtime state store. A receipt-only historical failure with no terminal AG2 state cannot be inferred generically; this change does not migrate historical records or inspect product outcomes.
  • AG2 run history is persisted separately through the AG2 stream storage adapters and is the source of truth for execution re-entry and UI replay.
  • While the backend process still owns a paused AG2 workflow channel, user replies continue that live AG2 channel first. Persisted AG2 events are the canonical replay and restart fallback when the live Hub/channel handle is no longer available.
  • AG2 agent-turn, LLM-call, tool-call, and HITL telemetry is emitted through AG2 beta TelemetryMiddleware as OpenTelemetry spans.
  • LLM token usage is emitted by Mozaiks AG2 1.0 usage middleware as chat.usage_delta events and stored in RuntimeUsageEvents. This ledger is measurement-only. It does not enforce entitlements, quotas, pricing, or hosted billing.
  • Reconnectable workflow UI state lives under ChatSessions.workflow_ui_state with:
  • schema_version
  • last_artifact
  • pending_input_request
  • tool_calls
  • On startup, the runtime backfills pre-migration top-level workflow UI fields such as last_artifact and pending_input_request into workflow_ui_state and removes the old top-level fields. Runtime readers should not depend on those pre-migration top-level fields.
  • The current source of truth for this runtime contract is mozaiksai/core/data/persistence/persistence_manager.py, mozaiksai/core/workflow/execution/run_bootstrap.py, mozaiksai/core/transport/run_replay.py, mozaiksai/hosts/runtime.py, mozaiksai/hosts/platform.py, and the focused tests tests/test_persistence_initial_messages.py, tests/test_orchestration_seed_persistence.py, tests/test_run_replay.py, tests/test_runtime_websocket_contract.py, and tests/test_platform_chat_meta_contract.py.

Runtime usage surfaces:

  • OSS app creators query /api/admin/usage in Studio/Admin for app-scoped or workspace-scoped token totals.
  • Hosted products such as Mozaiks App use the same /api/admin/usage route and may also forward summary events to their Refinement Engine.
  • Generated app end users query /api/me/usage from the profile surface. The response combines measured runtime usage with usage-limit metadata declared in app/config/subscriptions.yaml when the app is a SaaS app.
  • MozaiksPay and other billing providers consume these measurements through app-owned facade modules. Runtime usage events are not the billing authority.

2. Builder Artifacts

Framework-owned pipeline artifacts produced and consumed by factory_app:

  • BuilderConcepts
  • BuilderBuildPlans
  • DesignDocuments
  • ThemeCaptures
  • DataContracts
  • DatabaseMigrations
  • WorkflowExports
  • LLMConfig

These collections hold the durable handoff between workflow stages such as ValueEngine, DesignDocs, AgentGenerator, and AppGenerator.

Versioned BuildRecord documents in ArtifactVersions use the unique key (app_id, build_family, build_key, version_number). Counters in ArtifactVersionCounters use (app_id, build_family, build_key). Store initialization installs these indexes for canonical records and removes only the known obsolete unique indexes over artifact_kind and artifact_key. Existing documents are not deleted or rewritten. An unexpected index definition or conflicting canonical data fails initialization and requires operator review.

2b. Platform Connector Metadata

Platform-owned, app-scoped connector metadata used by the visible Integrations/Admin surfaces and workflow integration helpers:

  • AppConnectors

This collection stores sanitized connector state only. Raw API keys, OAuth client secrets, refresh tokens, and other secrets do not belong in MongoDB builder artifacts.

2c. Connector Secret Vault

Durable connector secrets are a separate framework-owned backend:

  • default contract: mozaiksai.core.secrets.connector_vault
  • default provider mode: MOZAIKS_CONNECTOR_SECRET_BACKEND=auto

Backends selected by auto:

Condition Backend
AZURE_KEY_VAULT_NAME is set AzureKeyVaultConnectorVaultBackend
No Azure vault configured MongoConnectorVaultBackend (default)

get_secret reports status: found | not_found | error alongside its existing success and secret_value fields. Only found carries a usable secret; not_found means the backend confirmed that the named secret is absent. Azure requires the Key Vault SecretNotFound error code; an unrelated 404, unavailable vault, authentication failure, transport failure, or unreadable encrypted value is error. Callers that create a new secret after a read must proceed only on not_found and must treat a missing status as an error.

MongoConnectorVaultBackend stores Fernet-encrypted secrets in the ConnectorSecrets collection in the same MongoDB instance. Encryption key priority:

  1. MOZAIKS_CONNECTOR_SECRET_KEY — explicit 32-byte URL-safe base64 or hex key
  2. Derived via HMAC-SHA256 from SECRET_KEY
  3. Dev-only deterministic fallback with a loud warning (not for production)

To generate an explicit key: 

python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

Rules:

  • MongoDB (Connectors collection) stores connector metadata, status, timestamps, and ownership only — never raw secrets.
  • ConnectorSecrets collection stores encrypted secret values managed by MongoConnectorVaultBackend. Each record is uniquely keyed by (scope, scope_id, normalized_service) where scope is app or workspace. These are framework-internal records, not app data.
  • Azure secret names include the scope kind and a digest of the normalized full prefix and complete connector identity. Reads verify the latest identity tags. Stores and name-wide deletes verify the tags on every existing version and fail closed when version listing is unavailable or any version is unowned. Azure needs secret get and list permissions for these mutations.
  • Distinct deployments using Mongo connector secrets need separate Mongo instances; the Mongo key does not include the Azure name prefix, and the fixed system database ignores a URI database suffix.
  • The vault backend requires an explicit scope on every save, read, and delete. Unqualified older records are never used as a fallback.
  • Before upgrading an environment with saved connectors, run the private metadata-only inventory described in the workspace integration guide. Old records need operator review; ambiguous records need credential re-entry. Retained unqualified records keep the inventory unready until separately reviewed cleanup, even after scoped credentials have been re-entered.
  • Azure Key Vault remains the recommended backend for production deployments that already operate Key Vault infrastructure.
  • A connector is active when secret_available: true (secret stored in vault). It is metadata_only only when the save itself failed — not simply because no external vault is configured. Old unqualified secret metadata does not count as ready after the scoped vault contract is installed. A connector saved under a different active backend is also unready until its credential is configured in the selected backend.

3. App Business Data

App-owned product data managed by generated or hosted modules:

  • projects
  • tasks
  • audit_logs
  • notifications
  • other module-owned collections

This data is not builder metadata. It belongs to app module boundaries and must be declared by the app's backend contracts.

Canonical Namespace

Framework-owned persistence uses a single system database:

  • database: mozaiksai

Active code should not introduce hardcoded framework database names outside the canonical mozaiksai namespace. Older local database names are not part of the clean OSS contract.

Builder Artifact Flow

The build pipeline depends on durable artifact handoff:

  1. ValueEngine writes concept and planning artifacts.
  2. DesignDocs reads those artifacts and writes design contracts.
  3. AgentGenerator and AppGenerator read the design artifacts.
  4. Refinement and revision flows read prior versions and migration history.

This is why persistence is required for the real builder experience.

Staged Filesystem Output

Generated app bundles are staged on disk under:

generated/apps/{app_id}/{build_id}/app

That staged bundle is separate from Mongo persistence:

  • Mongo stores build/runtime metadata and durable workflow handoff artifacts.
  • the filesystem stores the generated app bundle itself.

Promotion into a runnable workspace is an explicit later step.

Data Contracts and Revisions

Database evolution is a first-class generated artifact, not an implicit side effect of handler code.

  • DesignDocs owns the typed data_contract
  • AppGenerator stages data/contract.json
  • Refinement Runs may stage data/migrations/{migration_id}.json
  • generated module repos use backend/schemas.py for typed document shapes and backend/repo.py for persistence operations
  • the runtime injects ctx.persistence into module actions when app_id exists; generated repo code uses ctx.persistence.collection(module_id, collection_name) and must not require ctx.db
  • the runtime loads data/contract.json during app load; missing intent is allowed for non-persistent apps, while invalid JSON or invalid shape fails app loading
  • the runtime applies and rereads declared indexes idempotently, reports readiness only after exact materialized verification, and applies only additive migration files from data/migrations/*.json
  • migration states are recorded in mozaiksai.AppDatabaseMigrations

For collections declaring per_user or per_workspace tenancy and an owner_field, runtime persistence applies the authenticated principal's user or workspace identity to every read, count, update, delete, and aggregation. Inserts receive that exact owner field automatically; a supplied conflicting value is rejected. Updates cannot move a row to another owner. This boundary covers both generated canonical actions and model-authored repositories without relying on their filters or use of policy.py.

The runtime captures the token-validated HTTP or socket principal; Page Ask dispatch carries the same immutable identity. A registered host module_scope_resolver can return verified_workspace_id only after verifying membership for that authenticated actor. Omission preserves the signed token's workspace claim; explicit None revokes workspace ownership. Plain workspace_id, action inputs, requested scope, and mutable context fields never grant ownership. The assertion is accepted only from the registered host hook. The hook's verified_tenant_id assertion keys module entitlement lookups only; persistence never reads it, and stored rows keep the dispatch tenant.

Auth-disabled local, development, and test hosts use an explicit logged development principal with stable workspace development. This authority is unavailable in deployed environments. Generated apps with any per_user or per_workspace collection deterministically set app.json.authRequired=true and receive the canonical auth scaffold, even when model-authored app intent omits auth or requests a public app.

Module execution binds and revokes ownership authority around each dispatch. Each collection operation uses that active principal, so a cached handle cannot retain an earlier caller's ownership. Expired scopes and cached handles used from another app fail closed. Host-created standalone persistence contexts may supply an explicit principal for host operations. app_wide and ownerless collections receive no additional owner filter.

Repos call ctx.persistence.collection(module_id, collection_name) using the declared collection name. A declared entity value resolves to the same physical collection and ownership policy; unknown references fail closed. A filter that repeats the principal's own owner or app identity adds no condition and is not matched twice, so owned upserts may name the owner field.

Platform modules a host mounts into a workspace, such as Studio's built-in modules, are bound by their own bundle's data/contract.json rather than the workspace's. Each dispatch's persistence is composed for the dispatching module: app modules keep the workspace contract and cannot reach platform collections, and a platform module reaches only the collections declared for it. See Data Contracts.

Aggregations operate on the caller's owned rows. Foreign-collection stages and aggregation writes cannot bypass the boundary, and admin permissions do not implicitly disable it. Cross-owner behavior is unavailable through module persistence until an explicitly declared, auditable runtime access contract is implemented. Generated code must not substitute raw Mongo access. In mixed apps, literal access to an owned physical collection is rejected. Unowned aliases remain usable through a bounded Mongo-compatible facade that supports their existing operations and safe aggregation cursors; it prevents foreign-collection stages and database introspection from exposing owned rows. Each method binds its call against the installed driver's contract for that operation: primary arguments (filter, update, document, pipeline and so on) by position or by the driver's keyword but not both, plus only the options the driver defines for that operation (such as sessions, comments, sorts, upserts, index hints, time limits, skip/limit for counts, and allowDiskUse/batchSize for aggregations). Any other option, a surplus positional argument, or a raw query command envelope in a read filter is refused before the driver builds a command. An aggregation pipeline, on this facade and on owner-scoped collections, is copied once into plain documents, and that copy is both what the stage validator checks and what the driver sends. Bounded handles keep a module's mistakes inside its own collections; they are not a sandbox against hostile code running in the host process, which can reach the database by other means. Apps with no owned collections retain raw alias behavior. Scoped shared collections must declare their owning surface; missing ownership fails loading. Index changes on owned collections remain host startup work: module calls cannot create collection-wide indexes, including TTL indexes that delete other owners' rows.

Generated-module admission rejects private persistence attributes, raw driver imports, client/context constructors, and request-principal binding internals, including ordinary aliases. Generated modules resolve declared aliases through app_data_from_context(ctx), without overriding its contract or app root; direct literal_collection calls and constructing AppData are rejected. Startup declarations grant no raw-driver or ownership exemption; generated workers follow the same module admission rules. Provider database SDK mechanics belong in declared services/adapters/database/ integrations, which cannot own app persistence authority. Account-data handlers receive their database through the account-data protocol and have no raw-client exemption.

The target contract is:

  • additive changes can be applied deterministically
  • destructive changes require explicit review
  • migration history must stay linked to app artifact versions

Supported generated app migration operations today are:

  • ensure_collection
  • ensure_index

The runtime does not execute arbitrary migration code, drop collections, delete fields, rename fields, or rewrite documents as part of generated app migrations.

Generated-app database startup policy is controlled by MOZAIKS_DATABASE_STARTUP_POLICY:

  • best_effort is the default for additive migrations. Migration failures are logged and startup continues.
  • declared index inspection, mismatch, creation, or verification failures always fail startup; an app cannot report healthy against an incompatible persistence contract.
  • required is recommended for production persistent generated apps so migration failures also fail startup.

Apps with no data/contract.json, with no declared indexes, or running locally under best_effort without a configured Mongo connection keep the non-persistent startup path and do not require Mongo index readiness. A Mongo connection, required policy, or production environment enables the readiness gate. Index comparison uses names, ordered keys, and all canonical materialized options. Runtime application is additive only and never drops an index; definition changes require an explicit operator compatibility migration.

App business data database names are resolved from an injected adapter value, then MOZAIKS_APP_DATABASE_NAME, then MOZAIKS_APPS_DATABASE, then mozaiks_apps.

Account export and deletion dispatch registered module AccountDataHandler implementations through the platform's /api/account/export and /api/account routes. The routes use the authenticated principal's app and user identity and the existing app-data database accessor; they do not read the workflow runtime's persistence manager. App-data alias consumers additionally honor MOZAIKS_APP_DATA_DATABASE_NAME before the app database settings above. Handlers resolve their own collection contracts and enforce app/user ownership. Factory's onboarding module exercises this path with canonical generated collection names; an alias manifest is not required merely to resolve the account database.

For a module owning per_user collections, code renders backend/account_data_handler.py from the data contract. Its constructor takes persistence: the account routes build the requesting account's MongoPersistenceContext from the loaded app's data contract (app.state.data_contract) and the authenticated principal, so every export and delete query is app- and owner-scoped by the runtime exactly like a module action's ctx.persistence, and the handler also filters by the owner field. Handlers whose constructor takes db still receive the app database. For any other module declaring user_data_scope, AppGenerator's ServiceAgent owns the handler; the materializer preserves its ServiceOutput.python_files content, rather than generating a deletion policy from module stubs. The existing account-data file contract and hook supply raw Motor API guidance and the current mozaiksai.core.runtime.persistence.naming.collection_name_for signature. For a repo using ctx.persistence.collection(module_id, collection_name), the handler must use those same IDs with collection_name_for(app_id=app_id, ...) and omit app_slug, matching the standard executor. Literal collections, aliases, external bindings, and custom contexts retain their declared storage contract; the generated-name helper is not a universal alias resolver.

Raw database access does not inject scope: deletion and export queries must include app identity and the user's ownership fields, even in an app-specific physical collection. Preserve the module's existing deletion or anonymization policy. Account exports must also be JSON-safe before reaching JSONResponse: exclude storage _id or stringify ObjectId values, and encode datetimes as ISO 8601 strings, including nested values. The contract's export example is tested against real BSON values and JSON serialization; this verifies guidance and materialization, not arbitrary model-generated implementations.

Migration history records use in_progress, applied, and failed. The mozaiksai.AppDatabaseMigrations collection also acts as the migration lock: the runtime atomically claims a migration by inserting an in_progress record for (app_id, migration_id) before operations run. The collection has a unique index on (app_id, migration_id) so concurrent startup instances cannot both claim the same migration.

Failed records include error type/message and failed operation details. Existing in_progress or failed records block automatic retry until an operator clears or repairs the history record. in_progress means another instance is applying the migration or a previous instance crashed after claiming it. The first-pass runtime does not take over expired locks; operators must inspect the history record and repair or clear it deliberately. Production persistent apps should run with MOZAIKS_DATABASE_STARTUP_POLICY=required so migration lock conflicts fail startup instead of being treated as healthy.

Migration health is inspectable through the read-only runtime helper get_migration_health_report(). The report returns summary counts, migration items, has_blockers, and has_unknown_statuses. failed and in_progress records are operational blockers; applied records are healthy; unknown statuses are surfaced for operator review. The helper does not repair, clear, retry, or mutate migration records. Operators should inspect this report when startup logs or required-mode startup failures mention migration application or claim failures.

The CLI exposes the same read-only report:

mozaiks migrations status --app-id app_123
mozaiks migrations status --status failed --json

Options:

  • --app-id: filter to one app.
  • --status: filter to one migration status.
  • --limit: maximum rows, default 100.
  • --database-name: migration history database override for diagnostics.
  • --json: print the exact report as JSON.

Exit codes:

  • 0: no blockers and no unknown statuses.
  • 1: failed/in-progress blockers or unknown statuses exist.
  • 2: configuration, Mongo connection, or report loading error.

The command does not print Mongo connection strings or credentials. It does not repair, clear, retry, mutate migration records, or take over locks.

Real Mongo Smoke

Normal CI does not require MongoDB for generated-app persistence. The real Mongo-backed smoke is opt-in and validates the production adapter path:

$env:MONGO_URI="mongodb://localhost:27017"
$env:MOZAIKS_RUN_REAL_MONGO_TESTS="1"
python -m pytest tests/test_runtime_persistence_real_mongo.py

The smoke creates a dedicated test app database named mozaiks_persistence_test_{random} by default and drops it during cleanup. To use an explicit test database name, set MOZAIKS_TEST_APP_DATABASE_NAME to a dedicated database whose name contains test. Do not use production credentials or production database names for this smoke.

Generated module layering for app business data:

  • handler.py dispatches only
  • service.py orchestrates business logic and calls repo methods
  • repo.py uses ctx.persistence.collection(module_id, collection_name)
  • policy.py provides optional ownership preflight from the immutable persistence principal
  • schemas.py defines typed shapes and pure helpers

Generated modules must not call get_mongo_client() directly, use ctx.db, or hardcode database names. Do not generate backend/models.py, backend/database/schema.json, or backend/database/seed.json.

Implementation Rules

  • Framework-owned builder artifact persistence should flow through BuilderArtifactStore, not raw collection access in workflow tools.
  • App-scoped connector metadata should flow through a connector service/store, not raw collection access in workflow tools.
  • Durable connector secrets should flow through the connector vault backend, not MongoDB collections or generated module code.
  • Do centralize framework DB and collection names in shared runtime constants.
  • Do route workflow tools through artifact-aware persistence helpers where possible.
  • Do keep runtime state, builder artifacts, and app business data logically separate.
  • Do keep connector metadata separate from app business collections and builder artifact collections.
  • Do fail fast when the Studio host or the builder is launched without durable persistence configured.
  • Do not teach workflows or docs that removed database names are canonical.
  • Do not treat persistence as optional for the builder journey.