From 4204806c80543a9c40ad5b5f5e1fa6623b3de477 Mon Sep 17 00:00:00 2001
From: Aaron Nelson <aaron@aaronnelson.studio>
Date: Mon, 4 May 2026 03:24:51 +0000
Subject: [PATCH] conversations.db, sessions.db: enable WAL, add message index;
 update backup.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Both databases ran with journal_mode=delete — every write rewrote the
rollback journal per transaction. WAL eliminates the journal-rewrite and
lets readers run without blocking writers.

Index on messages(conversation_id, timestamp DESC) is preventive — only
280 rows today, but the access pattern (load conversation history in
order) is exactly what a composite index serves, and we don't want to
re-revisit this when the table grows.

backup.sh updated in the same commit because WAL changes the on-disk
layout: a bare `cp` of just the .db file can miss recently-committed
transactions that still live in the -wal sidecar, and can race with
concurrent writes to produce a torn file. Switched to the SQLite Online
Backup API via python3 -c "...src.backup(dst)..." — same mechanism as
the sqlite3 CLI's `.backup` (which isn't installed on this host),
handles WAL correctly without forcing a checkpoint, and is non-locking
from the writer's perspective. Verified backup integrity_check returns
ok and row counts match.

Note: synchronous=NORMAL was considered but deferred — it's a
per-connection PRAGMA, and applying it correctly requires a connect
helper that wraps every sqlite3.connect() call site in api.py (~14
sites). Out of scope for this commit; tracked as a follow-up. WAL alone
delivers the journal-rewrite elimination and reader/writer concurrency
improvements; the additional fsync reduction from synchronous=NORMAL is
a smaller marginal win on top.

Confirmed via concurrency audit that api.py is the sole writer to both
databases. ingest_conversations.py and dream.py are read-only consumers
of conversations.db; nothing else touches sessions.db.
---
 scripts/api.py    | 3 +++
 scripts/backup.sh | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/scripts/api.py b/scripts/api.py
index 67fdba2..bd4415a 100644
--- a/scripts/api.py
+++ b/scripts/api.py
@@ -124,6 +124,7 @@ SESSIONS_DB = str(Path.home() / "aaronai" / "sessions.db")
 def _init_sessions():
     conn = sqlite3.connect(SESSIONS_DB)
     conn.execute("CREATE TABLE IF NOT EXISTS sessions (token TEXT PRIMARY KEY, created_at TEXT)")
+    conn.execute("PRAGMA journal_mode=WAL")
     conn.commit()
     conn.close()
 
@@ -182,6 +183,8 @@ def init_conversations_db():
         timestamp TEXT NOT NULL,
         FOREIGN KEY (conversation_id) REFERENCES conversations(id)
     )''')
+    c.execute("PRAGMA journal_mode=WAL")
+    c.execute("CREATE INDEX IF NOT EXISTS idx_messages_conv_ts ON messages(conversation_id, timestamp DESC)")
     conn.commit()
     conn.close()
 
diff --git a/scripts/backup.sh b/scripts/backup.sh
index 81cd736..6a51cbb 100755
--- a/scripts/backup.sh
+++ b/scripts/backup.sh
@@ -6,7 +6,7 @@ mkdir -p "$BACKUP_DIR"
 # Copy critical files
 cp ~/aaronai/memory.md "$BACKUP_DIR/memory-$DATE.md"
 cp ~/aaronai/settings.json "$BACKUP_DIR/settings-$DATE.json"
-cp ~/aaronai/conversations.db "$BACKUP_DIR/conversations-$DATE.db"
+python3 -c "import sqlite3, sys; src = sqlite3.connect('$HOME/aaronai/conversations.db'); dst = sqlite3.connect('$BACKUP_DIR/conversations-$DATE.db'); src.backup(dst); dst.close(); src.close()"
 
 # Keep only last 7 days
 find "$BACKUP_DIR" -name "*.md" -mtime +7 -delete