Agent Runtime Architecture

This document describes the recommended way to extend the agent runtime without turning loop.ts back into a god file.

Design Principles

1. Keep loop.ts as the orchestrator

loop.ts should read like a pipeline. It assembles context, calls stage modules in order, decides whether to continue or short-circuit, and handles top-level errors.

The loop should not own command-specific logic, file processing details, tool policy text, or toolkit activation rules.

2. Prefer typed stage boundaries

Each stage accepts a small input object and returns a named result object:

• resolveAgentRuntimeContext(...)
• buildInboundUserMessages(...)
• handlePendingImageEdit(...)
• runImageKnowledgeBypass(...)
• executeAgentRun(...)
• resolveAgentReply(...)

3. Keep tools atomic

A tool is the smallest model-callable capability. A toolkit is a code-owned bundle of tools with grouped permission semantics and shared policy text.

4. Put policy close to runtime

Tool behavior constraints belong in tooling/runtime.ts or tooling/toolkits/<name>.ts, not hardcoded inside loop.ts.

5. Commands are not tools

Commands (/start, /status) are explicit user control-plane actions. Tools (knowledge_search, run_sql) are model-invoked capabilities. Keep these systems separate.

Architecture

src/lib/agent/
  loop.ts                 → top-level orchestration
  channel-session.ts      → channel/session lookup
  runtime-context.ts      → tool assembly, skills, memories, system prompt, MCP
  execution.ts            → generateText execution, step handling
  media-search.ts         → image knowledge bypass stage
  tools.ts                → builtin tools + sub-app tools
  commands/
    index.ts              → parseCommand + dispatchCommand
    handlers/*.ts         → one file per command
  media/
    download.ts           → file download abstraction
    messages.ts           → user message construction
    image-edit.ts         → pending image-edit intercept
  tooling/
    catalog.ts            → client-safe tool/toolkit metadata
    runtime.ts            → tool filtering, policy sections, tool directives
    registry.ts           → toolkit runtime registry
    toolkits/*.ts         → toolkit policies and activation rules
    tools/*.ts            → extracted tool implementations

Integration Pipeline

Incoming event → channel-session → commands → image-edit → messages
→ runtime-context → media-search → tooling/runtime → execution → persist + send reply

Where New Logic Should Go

Naming Conventions

• resolve* — state/context assembly
• build* — derived prompts/messages/config
• handle* — interceptors or side-effecting branches
• execute* — model/tool runs
• parse* — syntax extraction