Cron Data Ingestion Pipeline

A reusable architecture pattern used across 4+ cron jobs in this Hermes instance. The same design powers commute tracking, workout logging, nutrition consolidation, and deal monitoring.

The Pattern

┌─────────────┐     ┌──────────────┐     ┌──────────┐     ┌──────────┐     ┌───────────┐
│  Cron Tick  │────▶│  no_agent    │────▶│  Detect  │────▶│  Write   │────▶│  Deliver  │
│  (recurring)│     │  Script Runs │     │  Change? │     │  to Vault│     │  (stdout) │
└─────────────┘     └──────────────┘     └──────────┘     └──────────┘     └───────────┘
                          │                    │
                          ▼                    ▼
                   Poll external API    Compare against
                   (Google Routes,       last known state
                    Hevy, Amazon)        (~/.hermes/cache/)

Key Design Rules

1. no_agent=True is the default — When the script finds nothing new, stdout is empty → nothing is sent. Zero LLM calls, zero delivery cost on quiet ticks.
2. Script owns the structured output — The script generates properly formatted markdown (frontmatter, tables, breadcrumb links), commits, and pushes. No LLM involved in data transformation.
3. State file prevents duplicate work — Each pipeline maintains a JSON state file at ~/.hermes/cache/<pipeline>-last-state.json tracking the last-processed ID or timestamp.
4. Timezone pinning — All date-sensitive scripts explicitly set os.environ["TZ"] = "America/Los_Angeles" + time.tzset() at the top. A cron job fired at “midnight Pacific” runs at 07:00 UTC, and date.today() would return the wrong day without this.
5. HTML comments as insertion anchors — When appending rows to an existing markdown table, scripts search for a stable HTML comment anchor (e.g. <!-- commute rows added here -->) rather than relying on fragile text-matching.
6. Auto-commit is safe — Deterministic writes (data transformations, not editorial decisions) are automatically git add -A && git commit && git push. Commit format: chore: <domain> log — YYYY-MM-DD.
7. Every new note gets an inbound link — Before writing, scripts update the domain MOC with a wikilink. No orphan pages.

Instance Comparison

Pipeline	Script	API	Vault Destination	Frequency	State File
Commute tracker	check-commute.sh	Google Routes API	content/commute/commute-log.csv	30 min, M-F	Timestamp
Hevy workout watchdog	hevy-workout-watchdog.py	Hevy API	content/fitness/workout-YYYY-MM-DD.md	15 min	Workout ID
Nutrition consolidation	consolidate-food-log.py	(reads daily logs)	fitness/nutrition/trends.md + Nutrition-Home	Midnight daily	Date
Liberty 5 Pro deal watcher	check-liberty5pro-deals.py	Amazon/product page	Telegram only (alert)	11:00 UTC daily	Price threshold

When to Use This Pattern vs Agent-Driven Cron

Use case	Best fit
Polling an API for new data	✅ no_agent=True pipeline
Formatting structured data into markdown	✅ no_agent=True pipeline
Threshold alerts (price drop, new workout)	✅ no_agent=True pipeline
Summarizing/interpreting data (daily briefing)	❌ Agent-driven cron
Drafting reports with reasoning	❌ Agent-driven cron
Tasks with branching logic based on content	❌ Agent-driven cron

Interaction with Vault CI/CD

Every pipeline that writes to the vault triggers a Git push, which in turn triggers the GitLab CI → Cloudflare Pages deployment. The deployment is cached (~20–30s build time on cache hit), so frequent small pushes (15–30 min) are acceptable.

See Also

• System Architecture — where this pattern fits in the overall system
• Cron Design Decisions — why 15 min vs 30 min, why no_agent vs agent
• Garden: Hermes Bot Infrastructure — running config and cron schedules