I needed to know when an AI system became worse—not simply when a test failed.
Lighthouse is a lightweight, local-first evaluation framework for real-world AI systems. It is designed to test product outcomes in systems such as Synapse, Harbor, and Sentinel instead of relying on academic benchmarks or intuition alone.
The problem
Personal AI systems change constantly: prompts are edited, models are swapped, routing logic evolves, retrieval is tuned, and tools are added. Without a repeatable evaluation layer, those changes can create silent regressions.
The risk is not abstract. A local assistant can route to the wrong domain, call the wrong tool, return a less useful answer, retrieve the wrong file, or become slower after a seemingly small change.
Design & implementation
Lighthouse treats AI evaluation like a product feedback loop. A dataset describes real scenarios, a runner calls the target system, an evaluator applies checks to the response, and reporters produce readable output.
The current MVP targets Synapse first. It supports JSON datasets, a local HTTP runner, deterministic text checks, latency checks, JSON reports, and Markdown reports.
The supported checks are intentionally simple: contains, exact_match, regex, not_contains, and latency_ms_max. That keeps the system transparent enough to trust while still making regressions visible.
Current state
V0.1 focuses on a minimal eval loop that can run a dataset in one command, detect failures, measure latency, and produce readable output.
The roadmap expands from Synapse toward Harbor and Sentinel, then toward historical comparisons, tag-level metrics, failure clustering, diagnostics, optional judge layers, CI integration, trend analysis, and experiment comparison.
What it does
- Local-first evaluation framework for practical AI systems
- JSON datasets built from real product scenarios
- Local HTTP runner for target systems
- Deterministic checks for text output and latency
- JSON and Markdown reports for each run
- Roadmap toward multi-system evals across Synapse, Harbor, and Sentinel
Why I built it
Lighthouse makes AI iteration less blind. It gives local AI products a repeatable way to detect regressions, compare behavior after changes, and move from "did it pass?" toward "why did it fail?" and eventually "what should be improved next?"
