docs/: track Track 1 inventory and reorg plan

These are working artifacts of the 2026-05-02 Track 1 stabilization work. Versioning them alongside the code keeps the operational narrative coherent and gives future sessions clear reference docs. The inventory document includes the cross-repo verification finding on share_time — captured at the document level so future sessions don't repeat the same dead-code mischaracterization.
2026-05-03 00:00:16 +00:00
parent 4b520b2bc2
commit ec67e19b4f
2 changed files with 1008 additions and 0 deletions
@@ -0,0 +1,194 @@
+# scripts/ reorganization plan — 2026-05-02
+
+*Track 1 Bucket B fix #4 — read-only proposal. Nothing moved or deleted yet. Approve before executing.*
+
+## Summary
+
+The `~/aaronai/scripts/` directory currently holds **41** `.py`/`.sh` files. Reading the listing it is hard to tell which files are live workers and which are completed-experiment artifacts. The proposed split:
+
+| Bucket | Count | Destination |
+|---|---|---|
+| Production (stay) | 11 | `scripts/` |
+| Experimental (move) | 28 | `scripts/experiments/` (already exists, holds 4 files; will hold 32) |
+| Deprecated (move) | 2 | `scripts/deprecated/` (new) |
+| `.bak*` to delete | 19 | git history is the durable record |
+| Uncertain | 0 | n/a |
+
+After execution, `ls scripts/*.py scripts/*.sh` should return only the 11 production files plus the two subdirectories.
+
+## Reference checks performed
+
+Before producing this plan I grepped:
+- `subprocess` calls inside `api.py` for paths under `scripts/`
+- `import` and string-path references inside every production script
+- `ExecStart=` lines across every `aaronai-*.service` in `/etc/systemd/system/`
+- The user crontab for any line invoking a `scripts/` path
+
+**Findings:**
+- The only scripts referenced from `api.py` are `ingest.py` (line 43, `INGEST_SCRIPT`), `dream.py` (lines 661 and 1111), `ingest_conversations.py` (line 1127), and `corpus_integrity.py` (line 934, `CORPUS_INTEGRITY_SCRIPT`).
+- `api.py` (line 937) and `corpus_integrity.py` (line 29) reference the data file `~/aaronai/experiments/tier1_migration_state.json` — that path is the **state file** in `~/aaronai/experiments/`, not the script. Moving `tier1_migration.py` does not break either reader.
+- No production script imports or shells out to any experimental file.
+- All eight `aaronai-*.service` units' `ExecStart` lines point at production scripts only.
+- The user crontab references `backup.sh` and `aaronai-watcher` (a service) — no experimental files.
+
+So the reorganization is safe at the reference level for every file in section B (experiments), C (deprecated), and D (delete). No moves change a runtime code path.
+
+---
+
+## A — PRODUCTION (stay in `scripts/`)
+
+These 11 files are constraint-locked or referenced by an active runtime mechanism. None moves.
+
+| File | Why it stays |
+|---|---|
+| `api.py` | `aaronai.service` ExecStart; long-running FastAPI backend; APScheduler. |
+| `dream.py` | `aaronai-dreamer.service` ExecStart; called by APScheduler in `api.py`; called by `/api/dreamer/run`. |
+| `watcher.py` | `aaronai-watcher.service` ExecStart; Stage 1 of the encoding pipeline. |
+| `stage2_worker.py` | `aaronai-stage2.service` ExecStart. |
+| `stage3_worker.py` | `aaronai-stage3.service` ExecStart (service is currently stopped, but the unit is enabled and the file is the unit's ExecStart). |
+| `graphiti_service.py` | `aaronai-graphiti.service` ExecStart. |
+| `ingest.py` | `INGEST_SCRIPT` constant in `api.py`; `/api/reindex` shells out to it. |
+| `ingest_conversations.py` | `aaronai-index-conversations.service` ExecStart **and** APScheduler `ingest_job` in `api.py`. |
+| `corpus_integrity.py` | `CORPUS_INTEGRITY_SCRIPT` constant in `api.py`; `/api/corpus/reconcile` shells out to it. |
+| `st_embedder.py` | Imported by `graphiti_service.py` at sidecar startup (`SentenceTransformerEmbedder`). |
+| `backup.sh` | User crontab `0 3 * * *` daily snapshot of `memory.md`, `settings.json`, `conversations.db`. |
+
+---
+
+## B — MOVE TO `scripts/experiments/`
+
+28 files. None is referenced by any production code, systemd unit, or cron job.
+
+For brevity, the "Why" column gives the experiment identity — full per-file write-ups are in the inventory's Phase 1 experimental table. The "Referenced by" column is the result of the grep against api.py / systemd ExecStart lines / cron / production scripts; "(none in production)" means no production code references it.
+
+| Current path | Action | Why | Referenced by |
+|---|---|---|---|
+| `scripts/audit_expansion_draw.py` | move → `scripts/experiments/` | Type-aware stratified draw for n=20 audit expansion (sample-construction tool for `base_class_audit_rerun.py`). | (none in production) |
+| `scripts/base_class_test.py` | move → `scripts/experiments/` | Base-class enrichment OOP framing experiment, n=20. | (none in production) |
+| `scripts/base_class_validation.py` | move → `scripts/experiments/` | Base-class enrichment validation, n=50. | (none in production) |
+| `scripts/base_class_audit_rerun.py` | move → `scripts/experiments/` | Base-class n=8 paired-extraction audit. | (none in production) |
+| `scripts/briefing_generator_v2.py` | move → `scripts/experiments/` | Experiment 002b — briefing v2; validated 96% Mistral structural pattern. | (none in production) |
+| `scripts/briefing_test.py` | move → `scripts/experiments/` | Experiment 002 — briefing v1; superseded by v2. | (none in production) |
+| `scripts/cascade_test.py` | move → `scripts/experiments/` | Entity-drafter cascade n=20; falsified. | (none in production) |
+| `scripts/cascade_optimization_test.py` | move → `scripts/experiments/` | Optimized entity-drafter cascade n=30; confirmed entity-drafter cascade is dead. | (none in production) |
+| `scripts/consistency_test.py` | move → `scripts/experiments/` | Experiment 001 — Mistral 3-pass consistency, n=50. | (none in production) |
+| `scripts/consistency_test_v2.py` | move → `scripts/experiments/` | Experiment 003 — entity-only consistency with corrected sampling. | (none in production) |
+| `scripts/cost_test_graphiti_bulk.py` | move → `scripts/experiments/` | Bulk endpoint cost test, n=50. | (none in production) |
+| `scripts/cost_test_graphiti_bulk_retry.py` | move → `scripts/experiments/` | Retry of failed bulk batches (pre-MAX_QUEUED_QUERIES bump). | (none in production) |
+| `scripts/cost_test_graphiti_bulk_retry2.py` | move → `scripts/experiments/` | Second retry attempt, smaller batches. | (none in production) |
+| `scripts/cost_test_graphiti_migration.py` | move → `scripts/experiments/` | Single-episode migration cost test, n=50. | (none in production) |
+| `scripts/e1_select_sample.py` | move → `scripts/experiments/` | E1 sample selection. | (none in production) |
+| `scripts/e1_run_cascade.py` | move → `scripts/experiments/` | E1 cascade orchestration (initial). | (none in production) |
+| `scripts/e1_run_cascade_corrected.py` | move → `scripts/experiments/` | E1 corrected (custom_extraction_instructions path). | (none in production) |
+| `scripts/e1_per_source_predicates.py` | move → `scripts/experiments/` | E1 per-source predicate count, corrected metric. | (none in production) |
+| `scripts/e1_compare_metrics.py` | move → `scripts/experiments/` | E1 A vs B metrics comparison. | (none in production) |
+| `scripts/e14_select_sample.py` | move → `scripts/experiments/` | E1.4 stratified sample selection (n=30). | (none in production) |
+| `scripts/e14_run_cascade.py` | move → `scripts/experiments/` | E1.4 cascade orchestration. | (none in production) |
+| `scripts/e14_per_source_predicates.py` | move → `scripts/experiments/` | E1.4 per-source predicate diversity. | (none in production) |
+| `scripts/e16_rate_purity.py` | move → `scripts/experiments/` | E1.6 domain-purity human rating UI; surfaced taxonomic-mismatch finding. | (none in production) |
+| `scripts/e16_analyze.py` | move → `scripts/experiments/` | E1.6 Spearman correlation against E1.4. | (none in production) |
+| `scripts/e2_resolution_check.py` | move → `scripts/experiments/` | E2 entity-resolution diagnostic on six test entities. | (none in production) |
+| `scripts/e2_alias_followup.py` | move → `scripts/experiments/` | E2 alias follow-up (Aaron AI variants etc.). | (none in production) |
+| `scripts/e2_source_diversity.py` | move → `scripts/experiments/` | E2 episode count per entity. | (none in production) |
+| `scripts/token_measurement_test.py` | move → `scripts/experiments/` | Experiment 005 — token reduction measurement. | (none in production) |
+
+`scripts/experiments/` already contains four files (`e1_8_eval.py`, `e1_8_taxfree_cascade.py`, `e1_9_retroactive.py`, `e3_dreamer_substrate.py`); after the move it holds 32. **No collisions** between current `scripts/` filenames and existing `scripts/experiments/` filenames — verified by the file lists.
+
+---
+
+## C — MOVE TO `scripts/deprecated/`
+
+Two files. New directory `scripts/deprecated/` is created. Per the user constraint on tier1, both are flagged.
+
+| Current path | Action | Why | Referenced by |
+|---|---|---|---|
+| `scripts/consolidator_v0_1.py` | move → `scripts/deprecated/` | The reframe doc explicitly identifies "consolidator-as-separate-system" as the architectural mistake (its function moves into the dream phase). The 0.1 calibration findings (2026-04-29) showed alias-resolution-from-graph-features-alone has structural problems on this corpus that threshold tuning cannot address. Bespoke decision dissolves the layer. | (none in production); `scripts/consolidator_v0_1.py.bak` is in section D. |
+| `scripts/tier1_migration.py` | move → `scripts/deprecated/` | One-shot completed 2026-04-30 (1,205 sources, 4,990 nodes, 22,289 edges). Under the bespoke decision the substrate this migrated **to** is being replaced; re-running the script against the bespoke substrate would not be the right move. **Flag (per Tier1 constraint):** the script's state file at `~/aaronai/experiments/tier1_migration_state.json` IS still consumed — `corpus_integrity.py:29` and `api.py:937` read it for the "graphiti coverage" report. **Moving the script does not affect the state file** (the state file lives in `~/aaronai/experiments/`, not `~/aaronai/scripts/`). The reader-vs-writer separation makes this safe. | (none in production); state file `~/aaronai/experiments/tier1_migration_state.json` consumed by `corpus_integrity.py` + `api.py`, not the script itself |
+
+---
+
+## D — DELETE (`.bak*` files)
+
+19 files. Git history is the durable record of every prior version. Removing `.bak*` files is a cleanup, not a loss.
+
+For each: action is `rm`. None is referenced by any production path.
+
+| File | Approximate purpose |
+|---|---|
+| `scripts/api.py.bak.20260501-001427` | Pre-CV-pinning-strip / pre-F1 snapshot. |
+| `scripts/consolidator_v0_1.py.bak` | Pre-0.1.5-patch (Jaccard, before containment metric). |
+| `scripts/corpus_integrity.py.bak.20260501-021703` | Pre-F14 truncation snapshot. |
+| `scripts/dream.py.bak` | Older dreamer (pre v1.1 score-band). |
+| `scripts/dream.py.bak.20260501-002209` | Pre-F1 dreamer. |
+| `scripts/graphiti_service.py.bak` | Pre-bulk-saga sidecar. |
+| `scripts/graphiti_service.py.bak.20260501-185619` | Mid-rollback snapshot. |
+| `scripts/graphiti_service.py.bak.20260502-022307` | Mid-rollback snapshot (rolled-back work). |
+| `scripts/ingest.py.bak.20260501-004131` | Pre-F14 truncation snapshot. |
+| `scripts/stage2_worker.py.bak.20260501-171928` | v2.0 → v2.1 transition. |
+| `scripts/stage2_worker.py.bak.20260501-172531` | v2.1 patch step. |
+| `scripts/stage2_worker.py.bak.20260501-185942` | v2.1 patch step. |
+| `scripts/stage3_worker.py.bak.20260501-050354` | Pre-saga-split. |
+| `scripts/stage3_worker.py.bak.20260501-050453` | Pre-saga-split. |
+| `scripts/stage3_worker.py.bak.20260501-050719` | Pre-saga-split. |
+| `scripts/stage3_worker.py.bak.20260501-173233` | Mid-v2.1. |
+| `scripts/stage3_worker.py.bak.20260501-190357` | v2.1 final. |
+| `scripts/watcher.py.bak` | Pre-in-process refactor (2026-04-30). |
+| `scripts/watcher.py.bak.20260501-004131` | Pre-F14 truncation snapshot. |
+
+Stage 3 alone has five `.bak` versions; Stage 2 has three. Both are visible in `git log` for the corresponding production files — no information is lost.
+
+---
+
+## E — UNCERTAIN
+
+None. Every file in `scripts/` is classified above. The grep against api.py / systemd / cron / production scripts produced clean answers for each.
+
+The `scripts/__pycache__/` directory exists and contains `.pyc` for `api`, `corpus_integrity`, `dream`, `ingest`, `stage3_worker`, `st_embedder`, `watcher` (notably no `.pyc` for `stage2_worker.py`). Not part of this plan, but Python regenerates `.pyc` on next import — `__pycache__/` is safe to remove at any time and has no bearing on the moves above. **Recommended but not in this plan: `rm -rf scripts/__pycache__/` after the moves complete, so stale entries for moved files don't linger.**
+
+---
+
+## Execution-step preview (NOT executed in this turn)
+
+For when the plan is approved, the proposed mechanic is:
+
+```bash
+mkdir -p ~/aaronai/scripts/deprecated/
+
+# Section B — 28 moves to scripts/experiments/
+git mv scripts/{audit_expansion_draw,base_class_test,base_class_validation,base_class_audit_rerun, \
+                briefing_generator_v2,briefing_test, \
+                cascade_test,cascade_optimization_test, \
+                consistency_test,consistency_test_v2, \
+                cost_test_graphiti_bulk,cost_test_graphiti_bulk_retry,cost_test_graphiti_bulk_retry2,cost_test_graphiti_migration, \
+                e1_select_sample,e1_run_cascade,e1_run_cascade_corrected,e1_per_source_predicates,e1_compare_metrics, \
+                e14_select_sample,e14_run_cascade,e14_per_source_predicates, \
+                e16_rate_purity,e16_analyze, \
+                e2_resolution_check,e2_alias_followup,e2_source_diversity, \
+                token_measurement_test}.py scripts/experiments/
+
+# Section C — 2 moves to scripts/deprecated/
+git mv scripts/consolidator_v0_1.py scripts/tier1_migration.py scripts/deprecated/
+
+# Section D — 19 deletes
+rm scripts/*.bak*
+
+# Section E recommendation (post-move)
+rm -rf scripts/__pycache__/
+```
+
+`git mv` keeps git history. After execution, a single commit with a body listing each move and delete (no Co-Authored-By trailer) would land the change.
+
+---
+
+## What this plan does NOT do
+
+- Does not modify `api.py`, `corpus_integrity.py`, `tier1_migration.py`, or any other code. The `MIGRATION_STATE` path in `corpus_integrity.py:29` and the matching constant in `api.py:937` continue to point at `~/aaronai/experiments/tier1_migration_state.json` — unchanged by the move.
+- Does not modify any systemd unit. Every `ExecStart` continues to point at a `scripts/<production>.py` path that remains valid.
+- Does not touch the user crontab.
+- Does not touch `~/aaronai/db/` (separate decision flagged in inventory; ChromaDB-era 550M directory).
+- Does not delete `scripts/__pycache__/` (recommendation only).
+- Does not touch the four files already in `scripts/experiments/` (`e1_8_eval.py`, `e1_8_taxfree_cascade.py`, `e1_9_retroactive.py`, `e3_dreamer_substrate.py`).
+
+## Awaiting approval
+
+Tell me to proceed and I will execute Sections B → C → D in order, then run `git status` and `git diff --stat` so you can review before the commit. No commit will be made until you give the second go-ahead.