Source Validation Framework¶

The source validation framework plans framework-detected commands against a workspace to measure source health before and after code changes. It is intentionally opt-in: command execution requires confirm_execution=True and a trusted injected runner. The default host subprocess does not run repository commands in any auth mode until OS isolation is available. After confirmation, the Studio and refinement-harness entry points currently return static checks with required commands marked as skipped.

Related documents:

• App Intelligence Plane
• Framework Detection
• Source Import Contracts

Purpose¶

Validation answers: "Does this source still compile, lint, and test after the last staged change?" It feeds into:

• The App Intelligence Plane's AppContextVersion.validation_summary
• Studio's App Intelligence panel health display
• The refinement harness as a post-edit quality signal

Entry Points¶

Against the current App Intelligence context¶

result = await run_current_app_source_validation(
    app_id="my-app",
    allowed_kinds=["lint", "test"],
    confirm_execution=True,
)

Reads framework detection from the current AppIntelligenceSnapshot (or the latest index job as a fallback) and resolves the workspace root from the latest AppIntelligenceIndexJob.

Against an explicit workspace¶

result = run_app_source_validation(
    app_id="my-app",
    workspace_root="/path/to/workspace",
    framework_detection=detection_dict,
    confirm_execution=True,
)

This direct API accepts a specific workspace root and staged overlay files. The current refinement-harness tool instead uses the App Intelligence context.

Execution Model¶

framework_detection
    → plan_app_source_validation_commands()   # build ordered command plan
        → filter by allowed_kinds
        → reject long-running (dev, start)
        → reject unsafe command strings
        → reject disallowed executables
        → reject out-of-workspace working directories
        → cap at max_commands (default 4)
    → disposable workspace copy (tempfile)
    → apply overlay_files (staged changes)
    → execute planned commands in order only through a trusted injected runner
      (default host subprocess is blocked)
    → if no commands ran → fallback checks

Workspace copy¶

By default (copy_workspace=True), the runner creates a temporary directory and copies the workspace into it before executing any command. The copy excludes:

• Cache and build output directories: .cache, .git, .next, .pytest_cache, build, coverage, dist, node_modules, venv, etc.
• Secrets and credential files: .env, .env.local, .env.production, .npmrc, .pypirc, id_rsa, id_dsa
• Compiled artefacts: *.pem, *.key, *.pyc, *.pyo

The temporary directory is cleaned up after the run regardless of outcome. The copy keeps ordinary relative writes out of the original workspace. It does not isolate a command process from the host filesystem or network: code can use absolute paths. Run commands from untrusted repositories only in an OS sandbox with those boundaries.

Overlay files¶

overlay_files is a dict[str, str] of relative path → file content. The runner writes these into the isolated copy before executing commands. This allows the harness to validate staged changes without touching the original workspace.

Limits: 200 files maximum, 2 MB total content. Paths are validated with the same _safe_relpath() check used by source imports — no traversal out of the workspace is possible.

Command Security Model¶

Commands go through four checks before execution:

1. Shell metacharacter rejection — commands containing ;, |, &, `, $, >, <, \r, or \n are rejected outright.

2. Argv parsing — shlex.split(command, posix=True) must succeed.

3. Executable allowlist — only these executables may appear as argv[0]:

composer   mypy     next     npm      npm.cmd
php        pnpm     pnpm.cmd pytest   python
python3    ruff     tsc      vite     yarn
yarn.cmd

python and python3 are remapped to sys.executable at runtime. Other executables are resolved via shutil.which().

1. Working directory containment — working_directory must resolve to a path inside the workspace root via Path.relative_to().

Commands that fail any check are marked skipped with a machine-readable skip_reason, not failed. This keeps aggregate status accurate. Allowed tools such as Python and npm can run repository code, so the executable allowlist is a command-selection limit rather than an execution sandbox. The default runner marks otherwise runnable commands skipped with host_command_execution_requires_isolation even when the caller confirms execution. This also applies in no-auth local and open development modes. Static fallback checks may run, but required commands remain incomplete, so the result cannot be passed. A caller-injected runner is an internal Python seam, unavailable through Studio requests and workflow tool schemas. Its owner must provide OS isolation before using it for untrusted source.

Command Kinds¶

Default kind order: install → lint → typecheck → test → build. Commands run in this order when planned. Within a kind, higher-confidence candidates run first.

Fallback Checks¶

When no framework commands could be run (either none were planned or all were skipped), the runner falls back to deterministic static checks:

Fallback checks are also run when command results are all skipped.

Fallback checks never replace real framework validation. They provide a minimum-viable health signal for apps where validation commands are unavailable.

Result Schema¶

class AppSourceValidationResult(BaseModel):
    schema_version: "mozaiks.app_source_validation.v1"
    app_id: str
    validation_status: "passed" | "failed" | "skipped" | "warning"
    source: "app_intelligence_context" | "explicit_workspace"
    execution_mode: "isolated_workspace_copy" | "direct_workspace" | "not_executed"
    framework_detection_available: bool
    primary_framework_id: str | None
    primary_framework_label: str | None
    selected_kinds: list[str]
    planned_commands: list[AppValidationCommandPlanItem]
    command_results: list[AppValidationCommandResult]
    fallback_checks: list[AppValidationFallbackCheckResult]
    workspace_root_present: bool
    overlay_file_count: int
    started_at: str   # ISO 8601 UTC
    completed_at: str
    duration_ms: int
    warnings: list[str]

Aggregate status rules¶

Environment During Validation¶

The runner passes only the host PATH and Windows process essentials (PATHEXT, SystemRoot, WINDIR, COMSPEC). It points home, configuration, and temporary-directory variables at the disposable workspace copy. It also sets:

• PYTHONUSERBASE to the host interpreter's user package base, so installed Python validators remain importable without passing the host profile as home. This path can reflect an operator-supplied PYTHONUSERBASE and allows access to packages installed there; it is not filesystem isolation
• CI=true
• MOZAIKS_APP_VALIDATION=1

This signals to frameworks that they are running in a non-interactive validation context. Host runtime configuration and credentials are not passed to the child process. This environment limit does not replace an OS sandbox. Network commands that require a host proxy or custom CA setting may fail until that configuration is supplied through a separate trusted runner policy.

Skipping Execution Safely¶

When confirm_execution=False (the default), and runnable commands exist, the result is returned with execution_mode="not_executed" and validation_status="skipped". The planned command list is still populated so callers can inspect what would have run.

This makes it safe to call the planner to preview what validation would do without actually executing anything.

Refinement Harness Integration¶

The refinement harness calls source validation through factory_app/refinement_harness/tools/app_validation.py. The tool:

1. Reads the current framework detection from App Intelligence.
2. Uses the workspace root from the current App Intelligence index job.
3. Forwards its caller's confirm_execution choice (default False) to run_current_app_source_validation(); it does not supply overlay files.
4. Returns the result as a structured tool output for the LLM checkpoint.

The LLM uses validation results as a signal when deciding whether to accept a staged change or route it to a review checkpoint. Confirming execution in this tool currently does not run repository commands: command readiness stays unverified until an isolated runner is connected.

Implementation Map¶

Contributor Rules¶

• Do not add framework-specific logic to the validation runner. Executable detection and command emission belong in framework_detection.py.
• Review additions to _ALLOWED_EXECUTABLES carefully. The allowlist limits which command starts, but several entries can execute repository code.
• Do not remove the workspace copy step for performance. The isolated copy ensures staged overlay files cannot be left behind in the real workspace.
• Do not pass copy_workspace=False from production callers. The direct workspace mode exists for testing only.
• All operator-supplied paths (overlay file keys, working_directory) must pass _safe_relpath() before any filesystem operation.