I needed to know when an AI system became worse—not simply when a test failed.
Lighthouse is a lightweight, local-first evaluation framework for real-world AI systems. It is designed to test product outcomes in systems such as Synapse, Harbor, and Sentinel instead of relying on academic benchmarks or intuition alone.
The problem
Personal AI systems change constantly: prompts are edited, models are swapped, routing logic evolves, retrieval is tuned, and tools are added. Without a repeatable evaluation layer, those changes can create silent regressions.
A local assistant can route to the wrong domain, call the wrong tool, retrieve the wrong file or become slower after a change.
Design & implementation
A dataset describes real scenarios, a runner calls the target system, and an evaluator checks the response and records the results.
The current MVP targets Synapse first. It supports JSON datasets, a local HTTP runner, deterministic text checks, latency checks, JSON reports, and Markdown reports.
The supported checks are contains, exact_match, regex, not_contains, and latency_ms_max. Each failed check can be traced back to the scenario and response.
Current state
V0.1 focuses on a minimal eval loop that can run a dataset in one command, detect failures, measure latency, and produce readable output.
The roadmap expands from Synapse toward Harbor and Sentinel, then toward historical comparisons, tag-level metrics, failure clustering, diagnostics, optional judge layers, CI integration, trend analysis, and experiment comparison.
What it does
- Local-first evaluation framework for practical AI systems
- JSON datasets built from real product scenarios
- Local HTTP runner for target systems
- Deterministic checks for text output and latency
- JSON and Markdown reports for each run
- Roadmap toward multi-system evals across Synapse, Harbor, and Sentinel
