ALL MEMOS Download .docx

Newsletter System — Master Audit

Date: 2026-05-22 · Trigger: "newsletters sent with older data" + consolidate all processes.

How the system actually works (verified)

Root causes of "older data" (4 found)

1. Watchdog re-send loop (PRIMARY — already fixed today). verify_and_fix (hourly) → watchdog cleared the send sentinel + re-ran generators every hour → the same issue went out up to 6×/day (the "Claude Opus 4.7" repeats). Fixed: watchdog now retries/alerts at most once/day and never before the scheduled hour.

2. Weekly source windows on a DAILY newsletter. Generators pull github.com/trending/...?since=weekly and use 7-day (week_ago) windows. Since newsletters are now daily, the same weekly-trending items repeat every day for a week → reads as stale.

3. No freshness guard. Generators send whatever gather() returns — even if every source fetch failed or returned cached/old items. There is no "minimum N fresh stories or skip" gate.

4. Research indexer gap. titan-newsletter-research-daily (05:00 UTC) produced no research-index-2026-05-22.jsonl today (latest is 5/21). Doesn't directly stale the newsletters (they use live gather()), but it stales Genius/Scout and signals the indexer is failing silently.

Fix + consolidation plan

A. Freshness guard (kills stale sends): in the shared send path, require ≥N items with a recency signal; if not met, skip the send (log + sentinel-skip) rather than mail stale/empty content. Openclaw "0 items" should be a skip, not a force-retried failure.

B. Daily windows: switch ?since=weeklydaily and 7-day→24–48h windows so daily issues carry fresh items.

C. Consolidate 3 generators → 1 (cleanup-plan Theme 2): agentic_ai, claude, openclaw are ~90% identical (same gather, same template/ledger calls, same entity/headline/summary mapping). Collapse into one parameterized newsletter.py --slug <agentic-ai|claude|openclaw> with a per-slug source config. Merge newsletter_templates{,_contrast,_extra} → one; newsletter_to_audio+newsletter_to_polly → one. Net: ~20 newsletter scripts → ~6.

D. Indexer reliability: make the 05:00 indexer alert on failure (once/day) and never let a missing index silently degrade downstream.

E. Process consolidation (scheduled tasks): the newsletter trio + newsletter-research-daily + agent-stack-daily + watchdog → one orchestrated daily pipeline: indexer → generate (per slug, freshness-gated) → audio → archive → watchdog (verify only). One cron chain instead of 5+ overlapping tasks.

Order of execution (safe)

1. ✅ Watchdog loop (done — biggest visible fix).

2. Add freshness guard + daily windows (dry-run each slug before live).

3. Consolidate 3 generators → 1 parameterized module (dry-run parity check vs current output).

4. Merge templates + audio paths.

5. Collapse the scheduled tasks into one pipeline; re-register.

6. Indexer failure alerting.

Guardrail: every generator change is dry-run tested (--dry-run renders without SES) and parity-checked against current output before going live. No blind edits to a system that emails real subscribers.