Synapse

What works today

Synapse is an active personal system, not a public hosted product. Its working capabilities extend from everyday interaction to longer-running tasks:

• Voice and messaging: Telegram and an ESP32 voice endpoint connect conversation to service tools, approved home-control routines and spoken responses.
• Multimodal conversation: real messaging tests cover photo follow-ups, exact transcription, audio understanding and short-video analysis, including reuse of an image across turns.
• Public research: a dedicated Hermes specialist searches, reads source passages and produces referenced reports. Missions retain their objective, evidence and state for follow-up, cancellation or resumption.
• Development to deployment: An idea expressed in conversation can become a working application — built, tested and made available by the agent.
• Operational investigation: Hermes can use documentation to find the right service, inspect its live state and explain what it found. This workflow has been tested on real infrastructure; broader autonomy remains a work in progress.

What I designed and built

I designed the interaction and architecture, built the orchestration and integrations, and deployed the system across dedicated hosts.

I integrated Hermes for execution and specialist research. Synapse keeps its native tool loop for interaction and orchestration; not every request becomes an agent run.

• Interaction: voice and messaging share tools for retrieval, local control and specialist services.
• Execution: Hermes handles development in isolated workspaces and deploys previews to a separate Docker host.
• Inference: a stable gateway connects clients to the distributed model cluster; speech and multimodal processing run on dedicated nodes.

Research with continuity

Research can be quick and focused or deeper, with an additional review. The specialist reads sources, compares findings and keeps supporting references, uncertainties and contradictions visible.

Follow-up questions carry the recent subject forward without treating conversation history as evidence. Missions keep their progress and sources so the work can continue. Telegram notification follow-ups and switching from quick to deep research are still being validated in everyday use.

Routing media to the right model

Audio requests go to two different specialised models: a speech-recognition model for exact transcription, or a multimodal model for understanding and summarisation.

Images and videos are validated centrally and passed to tools through an opaque asset reference. Conversations retain lightweight, expiring references, so follow-up questions can reanalyse the same media without another upload.

Explicit replies select the intended media; conversation history stores references, not binary files.

Control and reliability

The system asks for clarification when a reference is ambiguous and gives explicit requests priority over older media. Hermes runs locally as a non-root user, with development, inference and preview hosting separated. Human approvals still matter: remote access can allow administrative changes, so this separation is not a security sandbox.

When the multimodal node is unavailable, Synapse reports it rather than substituting a text-only answer. The independent transcription service can remain available.

Automated regression tests complement real-world checks. Media expires, video analysis is limited to 15 seconds, and multimodal availability depends on the workstation. Visual identification can still be wrong; references and self-review help inspection, but do not guarantee factual accuracy. Improving that reliability is ongoing work.