# scripts/ reorganization plan — 2026-05-02

*Track 1 Bucket B fix #4 — read-only proposal. Nothing moved or deleted yet. Approve before executing.*

## Summary

The `~/aaronai/scripts/` directory currently holds **41** `.py`/`.sh` files. Reading the listing it is hard to tell which files are live workers and which are completed-experiment artifacts. The proposed split:

| Bucket | Count | Destination |
|---|---|---|
| Production (stay) | 11 | `scripts/` |
| Experimental (move) | 28 | `scripts/experiments/` (already exists, holds 4 files; will hold 32) |
| Deprecated (move) | 2 | `scripts/deprecated/` (new) |
| `.bak*` to delete | 19 | git history is the durable record |
| Uncertain | 0 | n/a |

After execution, `ls scripts/*.py scripts/*.sh` should return only the 11 production files plus the two subdirectories.

## Reference checks performed

Before producing this plan I grepped:
- `subprocess` calls inside `api.py` for paths under `scripts/`
- `import` and string-path references inside every production script
- `ExecStart=` lines across every `aaronai-*.service` in `/etc/systemd/system/`
- The user crontab for any line invoking a `scripts/` path

**Findings:**
- The only scripts referenced from `api.py` are `ingest.py` (line 43, `INGEST_SCRIPT`), `dream.py` (lines 661 and 1111), `ingest_conversations.py` (line 1127), and `corpus_integrity.py` (line 934, `CORPUS_INTEGRITY_SCRIPT`).
- `api.py` (line 937) and `corpus_integrity.py` (line 29) reference the data file `~/aaronai/experiments/tier1_migration_state.json` — that path is the **state file** in `~/aaronai/experiments/`, not the script. Moving `tier1_migration.py` does not break either reader.
- No production script imports or shells out to any experimental file.
- All eight `aaronai-*.service` units' `ExecStart` lines point at production scripts only.
- The user crontab references `backup.sh` and `aaronai-watcher` (a service) — no experimental files.

So the reorganization is safe at the reference level for every file in section B (experiments), C (deprecated), and D (delete). No moves change a runtime code path.

---

## A — PRODUCTION (stay in `scripts/`)

These 11 files are constraint-locked or referenced by an active runtime mechanism. None moves.

| File | Why it stays |
|---|---|
| `api.py` | `aaronai.service` ExecStart; long-running FastAPI backend; APScheduler. |
| `dream.py` | `aaronai-dreamer.service` ExecStart; called by APScheduler in `api.py`; called by `/api/dreamer/run`. |
| `watcher.py` | `aaronai-watcher.service` ExecStart; Stage 1 of the encoding pipeline. |
| `stage2_worker.py` | `aaronai-stage2.service` ExecStart. |
| `stage3_worker.py` | `aaronai-stage3.service` ExecStart (service is currently stopped, but the unit is enabled and the file is the unit's ExecStart). |
| `graphiti_service.py` | `aaronai-graphiti.service` ExecStart. |
| `ingest.py` | `INGEST_SCRIPT` constant in `api.py`; `/api/reindex` shells out to it. |
| `ingest_conversations.py` | `aaronai-index-conversations.service` ExecStart **and** APScheduler `ingest_job` in `api.py`. |
| `corpus_integrity.py` | `CORPUS_INTEGRITY_SCRIPT` constant in `api.py`; `/api/corpus/reconcile` shells out to it. |
| `st_embedder.py` | Imported by `graphiti_service.py` at sidecar startup (`SentenceTransformerEmbedder`). |
| `backup.sh` | User crontab `0 3 * * *` daily snapshot of `memory.md`, `settings.json`, `conversations.db`. |

---

## B — MOVE TO `scripts/experiments/`

28 files. None is referenced by any production code, systemd unit, or cron job.

For brevity, the "Why" column gives the experiment identity — full per-file write-ups are in the inventory's Phase 1 experimental table. The "Referenced by" column is the result of the grep against api.py / systemd ExecStart lines / cron / production scripts; "(none in production)" means no production code references it.

| Current path | Action | Why | Referenced by |
|---|---|---|---|
| `scripts/audit_expansion_draw.py` | move → `scripts/experiments/` | Type-aware stratified draw for n=20 audit expansion (sample-construction tool for `base_class_audit_rerun.py`). | (none in production) |
| `scripts/base_class_test.py` | move → `scripts/experiments/` | Base-class enrichment OOP framing experiment, n=20. | (none in production) |
| `scripts/base_class_validation.py` | move → `scripts/experiments/` | Base-class enrichment validation, n=50. | (none in production) |
| `scripts/base_class_audit_rerun.py` | move → `scripts/experiments/` | Base-class n=8 paired-extraction audit. | (none in production) |
| `scripts/briefing_generator_v2.py` | move → `scripts/experiments/` | Experiment 002b — briefing v2; validated 96% Mistral structural pattern. | (none in production) |
| `scripts/briefing_test.py` | move → `scripts/experiments/` | Experiment 002 — briefing v1; superseded by v2. | (none in production) |
| `scripts/cascade_test.py` | move → `scripts/experiments/` | Entity-drafter cascade n=20; falsified. | (none in production) |
| `scripts/cascade_optimization_test.py` | move → `scripts/experiments/` | Optimized entity-drafter cascade n=30; confirmed entity-drafter cascade is dead. | (none in production) |
| `scripts/consistency_test.py` | move → `scripts/experiments/` | Experiment 001 — Mistral 3-pass consistency, n=50. | (none in production) |
| `scripts/consistency_test_v2.py` | move → `scripts/experiments/` | Experiment 003 — entity-only consistency with corrected sampling. | (none in production) |
| `scripts/cost_test_graphiti_bulk.py` | move → `scripts/experiments/` | Bulk endpoint cost test, n=50. | (none in production) |
| `scripts/cost_test_graphiti_bulk_retry.py` | move → `scripts/experiments/` | Retry of failed bulk batches (pre-MAX_QUEUED_QUERIES bump). | (none in production) |
| `scripts/cost_test_graphiti_bulk_retry2.py` | move → `scripts/experiments/` | Second retry attempt, smaller batches. | (none in production) |
| `scripts/cost_test_graphiti_migration.py` | move → `scripts/experiments/` | Single-episode migration cost test, n=50. | (none in production) |
| `scripts/e1_select_sample.py` | move → `scripts/experiments/` | E1 sample selection. | (none in production) |
| `scripts/e1_run_cascade.py` | move → `scripts/experiments/` | E1 cascade orchestration (initial). | (none in production) |
| `scripts/e1_run_cascade_corrected.py` | move → `scripts/experiments/` | E1 corrected (custom_extraction_instructions path). | (none in production) |
| `scripts/e1_per_source_predicates.py` | move → `scripts/experiments/` | E1 per-source predicate count, corrected metric. | (none in production) |
| `scripts/e1_compare_metrics.py` | move → `scripts/experiments/` | E1 A vs B metrics comparison. | (none in production) |
| `scripts/e14_select_sample.py` | move → `scripts/experiments/` | E1.4 stratified sample selection (n=30). | (none in production) |
| `scripts/e14_run_cascade.py` | move → `scripts/experiments/` | E1.4 cascade orchestration. | (none in production) |
| `scripts/e14_per_source_predicates.py` | move → `scripts/experiments/` | E1.4 per-source predicate diversity. | (none in production) |
| `scripts/e16_rate_purity.py` | move → `scripts/experiments/` | E1.6 domain-purity human rating UI; surfaced taxonomic-mismatch finding. | (none in production) |
| `scripts/e16_analyze.py` | move → `scripts/experiments/` | E1.6 Spearman correlation against E1.4. | (none in production) |
| `scripts/e2_resolution_check.py` | move → `scripts/experiments/` | E2 entity-resolution diagnostic on six test entities. | (none in production) |
| `scripts/e2_alias_followup.py` | move → `scripts/experiments/` | E2 alias follow-up (Aaron AI variants etc.). | (none in production) |
| `scripts/e2_source_diversity.py` | move → `scripts/experiments/` | E2 episode count per entity. | (none in production) |
| `scripts/token_measurement_test.py` | move → `scripts/experiments/` | Experiment 005 — token reduction measurement. | (none in production) |

`scripts/experiments/` already contains four files (`e1_8_eval.py`, `e1_8_taxfree_cascade.py`, `e1_9_retroactive.py`, `e3_dreamer_substrate.py`); after the move it holds 32. **No collisions** between current `scripts/` filenames and existing `scripts/experiments/` filenames — verified by the file lists.

---

## C — MOVE TO `scripts/deprecated/`

Two files. New directory `scripts/deprecated/` is created. Per the user constraint on tier1, both are flagged.

| Current path | Action | Why | Referenced by |
|---|---|---|---|
| `scripts/consolidator_v0_1.py` | move → `scripts/deprecated/` | The reframe doc explicitly identifies "consolidator-as-separate-system" as the architectural mistake (its function moves into the dream phase). The 0.1 calibration findings (2026-04-29) showed alias-resolution-from-graph-features-alone has structural problems on this corpus that threshold tuning cannot address. Bespoke decision dissolves the layer. | (none in production); `scripts/consolidator_v0_1.py.bak` is in section D. |
| `scripts/tier1_migration.py` | move → `scripts/deprecated/` | One-shot completed 2026-04-30 (1,205 sources, 4,990 nodes, 22,289 edges). Under the bespoke decision the substrate this migrated **to** is being replaced; re-running the script against the bespoke substrate would not be the right move. **Flag (per Tier1 constraint):** the script's state file at `~/aaronai/experiments/tier1_migration_state.json` IS still consumed — `corpus_integrity.py:29` and `api.py:937` read it for the "graphiti coverage" report. **Moving the script does not affect the state file** (the state file lives in `~/aaronai/experiments/`, not `~/aaronai/scripts/`). The reader-vs-writer separation makes this safe. | (none in production); state file `~/aaronai/experiments/tier1_migration_state.json` consumed by `corpus_integrity.py` + `api.py`, not the script itself |

---

## D — DELETE (`.bak*` files)

19 files. Git history is the durable record of every prior version. Removing `.bak*` files is a cleanup, not a loss.

For each: action is `rm`. None is referenced by any production path.

| File | Approximate purpose |
|---|---|
| `scripts/api.py.bak.20260501-001427` | Pre-CV-pinning-strip / pre-F1 snapshot. |
| `scripts/consolidator_v0_1.py.bak` | Pre-0.1.5-patch (Jaccard, before containment metric). |
| `scripts/corpus_integrity.py.bak.20260501-021703` | Pre-F14 truncation snapshot. |
| `scripts/dream.py.bak` | Older dreamer (pre v1.1 score-band). |
| `scripts/dream.py.bak.20260501-002209` | Pre-F1 dreamer. |
| `scripts/graphiti_service.py.bak` | Pre-bulk-saga sidecar. |
| `scripts/graphiti_service.py.bak.20260501-185619` | Mid-rollback snapshot. |
| `scripts/graphiti_service.py.bak.20260502-022307` | Mid-rollback snapshot (rolled-back work). |
| `scripts/ingest.py.bak.20260501-004131` | Pre-F14 truncation snapshot. |
| `scripts/stage2_worker.py.bak.20260501-171928` | v2.0 → v2.1 transition. |
| `scripts/stage2_worker.py.bak.20260501-172531` | v2.1 patch step. |
| `scripts/stage2_worker.py.bak.20260501-185942` | v2.1 patch step. |
| `scripts/stage3_worker.py.bak.20260501-050354` | Pre-saga-split. |
| `scripts/stage3_worker.py.bak.20260501-050453` | Pre-saga-split. |
| `scripts/stage3_worker.py.bak.20260501-050719` | Pre-saga-split. |
| `scripts/stage3_worker.py.bak.20260501-173233` | Mid-v2.1. |
| `scripts/stage3_worker.py.bak.20260501-190357` | v2.1 final. |
| `scripts/watcher.py.bak` | Pre-in-process refactor (2026-04-30). |
| `scripts/watcher.py.bak.20260501-004131` | Pre-F14 truncation snapshot. |

Stage 3 alone has five `.bak` versions; Stage 2 has three. Both are visible in `git log` for the corresponding production files — no information is lost.

---

## E — UNCERTAIN

None. Every file in `scripts/` is classified above. The grep against api.py / systemd / cron / production scripts produced clean answers for each.

The `scripts/__pycache__/` directory exists and contains `.pyc` for `api`, `corpus_integrity`, `dream`, `ingest`, `stage3_worker`, `st_embedder`, `watcher` (notably no `.pyc` for `stage2_worker.py`). Not part of this plan, but Python regenerates `.pyc` on next import — `__pycache__/` is safe to remove at any time and has no bearing on the moves above. **Recommended but not in this plan: `rm -rf scripts/__pycache__/` after the moves complete, so stale entries for moved files don't linger.**

---

## Execution-step preview (NOT executed in this turn)

For when the plan is approved, the proposed mechanic is:

```bash
mkdir -p ~/aaronai/scripts/deprecated/

# Section B — 28 moves to scripts/experiments/
git mv scripts/{audit_expansion_draw,base_class_test,base_class_validation,base_class_audit_rerun, \
                briefing_generator_v2,briefing_test, \
                cascade_test,cascade_optimization_test, \
                consistency_test,consistency_test_v2, \
                cost_test_graphiti_bulk,cost_test_graphiti_bulk_retry,cost_test_graphiti_bulk_retry2,cost_test_graphiti_migration, \
                e1_select_sample,e1_run_cascade,e1_run_cascade_corrected,e1_per_source_predicates,e1_compare_metrics, \
                e14_select_sample,e14_run_cascade,e14_per_source_predicates, \
                e16_rate_purity,e16_analyze, \
                e2_resolution_check,e2_alias_followup,e2_source_diversity, \
                token_measurement_test}.py scripts/experiments/

# Section C — 2 moves to scripts/deprecated/
git mv scripts/consolidator_v0_1.py scripts/tier1_migration.py scripts/deprecated/

# Section D — 19 deletes
rm scripts/*.bak*

# Section E recommendation (post-move)
rm -rf scripts/__pycache__/
```

`git mv` keeps git history. After execution, a single commit with a body listing each move and delete (no Co-Authored-By trailer) would land the change.

---

## What this plan does NOT do

- Does not modify `api.py`, `corpus_integrity.py`, `tier1_migration.py`, or any other code. The `MIGRATION_STATE` path in `corpus_integrity.py:29` and the matching constant in `api.py:937` continue to point at `~/aaronai/experiments/tier1_migration_state.json` — unchanged by the move.
- Does not modify any systemd unit. Every `ExecStart` continues to point at a `scripts/<production>.py` path that remains valid.
- Does not touch the user crontab.
- Does not touch `~/aaronai/db/` (separate decision flagged in inventory; ChromaDB-era 550M directory).
- Does not delete `scripts/__pycache__/` (recommendation only).
- Does not touch the four files already in `scripts/experiments/` (`e1_8_eval.py`, `e1_8_taxfree_cascade.py`, `e1_9_retroactive.py`, `e3_dreamer_substrate.py`).

## Awaiting approval

Tell me to proceed and I will execute Sections B → C → D in order, then run `git status` and `git diff --stat` so you can review before the commit. No commit will be made until you give the second go-ahead.