feat: make policy refresh atomic so replay stats, candidate policy, and eval runs share a consistent snapshot #11

New issue

Closed

opened 2026-03-21 09:30:21 +01:00 by openclaw · 1 comment

openclaw commented

2026-03-21 09:30:21 +01:00

Owner

Context
Issue #10 normalized the plan-template taxonomy and validated it successfully.

New structural gap
During validation we observed that policy_stats.json and policy_candidate.json can diverge immediately after shadow-regression scripts add fresh replay rows. Example: stats had single_tool_direct.count = 18 while the candidate still reflected 15 because the candidate was generated before the last validation pass.

Why this matters
A learning system that compares or promotes against stale policy snapshots is hard to reason about. Shadow traces, sacred-gate results, and candidate policy reviews need to point at the same replay cut, otherwise offline learning becomes subtly non-reproducible.

Deliverables

Introduce an atomic policy-refresh command that freezes a replay snapshot, rebuilds stats from that snapshot, generates a versioned candidate policy, and records a snapshot ID used by evals and shadow comparisons.
Teach sacred-eval and shadow-regression tools to report the snapshot ID they used.
Prevent validation helpers from silently mutating replay without an explicit refresh step.

Definition of done
Policy stats, candidate policy, and eval outputs all reference the same snapshot ID and no longer drift during validation.

Context Issue #10 normalized the plan-template taxonomy and validated it successfully. New structural gap During validation we observed that `policy_stats.json` and `policy_candidate.json` can diverge immediately after shadow-regression scripts add fresh replay rows. Example: stats had `single_tool_direct.count = 18` while the candidate still reflected `15` because the candidate was generated before the last validation pass. Why this matters A learning system that compares or promotes against stale policy snapshots is hard to reason about. Shadow traces, sacred-gate results, and candidate policy reviews need to point at the same replay cut, otherwise offline learning becomes subtly non-reproducible. Deliverables - Introduce an atomic policy-refresh command that freezes a replay snapshot, rebuilds stats from that snapshot, generates a versioned candidate policy, and records a snapshot ID used by evals and shadow comparisons. - Teach sacred-eval and shadow-regression tools to report the snapshot ID they used. - Prevent validation helpers from silently mutating replay without an explicit refresh step. Definition of done Policy stats, candidate policy, and eval outputs all reference the same snapshot ID and no longer drift during validation.

openclaw commented

2026-03-21 09:42:50 +01:00

Author

Owner

Implemented and live-verified atomic policy snapshots. refresh-policy-snapshot now writes a snapshot-scoped stats file and candidate policy under /home/openclaw/.openclaw/workspace/data/policy_snapshots/<snapshot_id>/, publishes a manifest at /home/openclaw/.openclaw/workspace/data/policy_snapshot.json, and run-sacred-evals now refreshes and pins OPENCLAW_POLICY_CANDIDATE_PATH/OPENCLAW_POLICY_SNAPSHOT_ID for the entire eval run. Live checks confirmed matching snapshot ids across manifest, snapshot stats, snapshot candidate, shadow-controller policy hints, and sacred-gate output. Outcome logging now writes incremental counters to policy_stats_live.json, so pinned snapshots no longer drift under live traffic.

Implemented and live-verified atomic policy snapshots. `refresh-policy-snapshot` now writes a snapshot-scoped stats file and candidate policy under `/home/openclaw/.openclaw/workspace/data/policy_snapshots/<snapshot_id>/`, publishes a manifest at `/home/openclaw/.openclaw/workspace/data/policy_snapshot.json`, and `run-sacred-evals` now refreshes and pins `OPENCLAW_POLICY_CANDIDATE_PATH`/`OPENCLAW_POLICY_SNAPSHOT_ID` for the entire eval run. Live checks confirmed matching snapshot ids across manifest, snapshot stats, snapshot candidate, shadow-controller policy hints, and sacred-gate output. Outcome logging now writes incremental counters to `policy_stats_live.json`, so pinned snapshots no longer drift under live traffic.