Skip to content

Context and SQLite Store

tether.context

State Plane: ContextAssembler, Ephemeral Observation Folding, and Token Estimator.

ContextAssembler

Assembles KV-cache-friendly prompts with deterministic observation folding.

build_system_prompt(toolbox_index, skill_instructions=None)

Build the immutable system prompt prefix once per session.

assemble(*, system_prompt, goal, turns, observations, pins, current_step)

Assemble the message array while preserving the static prefix and folding old outputs.

estimate_tokens(text)

Fast deterministic token approximation (~4 chars per token).

estimate_messages_tokens(messages, tools=None)

Estimate total tokens across messages and tool schema payload.

elide_active_observation(obs_id, raw_output, max_chars=2400)

Slice oversized active observations into Head + Tail in L1.

tether.store

SQLite WAL state persistence and FTS5 BM25 RAG chunk store for Tether.

SQLiteStore

ACID SQLite WAL state and FTS5 BM25 RAG store.

close()

Close the current thread's cached SQLite connection.

index_text(session_id, source_type, source_id, text, max_chunk_chars=600)

Chunk and index text into the SQLite FTS5 chunks_fts table.

search_chunks(query, session_ids, source_types=None, source_id=None, top_k=3)

Run a BM25-ranked FTS5 query across the specified session(s).

get_project_session_ids()

Return GLOBAL_SESSION_ID plus all COMPLETED session IDs for cross-session RAG.

search_project_chunks(query, *, source_types=None, source_id=None, top_k=3)

Run a BM25 FTS5 query across project pins (__project__) and COMPLETED sessions.

get_session_digest(session_id)

Build a compact ~50-token digest card for a session (avoids flooding parent context).

extract_friction_trace(session_id)

Deterministic failed -> correction/fix lines from a session's turns (for retro).

get_stats(*, model=None, skill=None, limit=100)

Aggregate per-model telemetry, token cost, and ACI friction offenders.

format_stats_report(*, model=None, skill=None, limit=100)

Render the ultra-compact tether stats engineering scorecard and top offenders.

canonical_tool_command(raw_cmd)

Normalize a raw tool command into its canonical <tool> or <tool> <subcommand> key.

chunk_text(text, max_chars=600)

Split markdown or plain text into ~150-token chunks along headings and paragraphs.

Structured history replay

When the provider sets structured_history = True (Ollama, OpenAI-compatible, Anthropic), ContextAssembler(structured=True) replays each past action as an assistant tool_calls message (act with cmd and a bounded stdin) followed by a tool message holding the (folded) observation, instead of act('...') text and Observation: user text that small models imitated into real arguments (#38). stdin is replayed as a real prefix (400 chars, then …), never a <N chars> placeholder that could be copied literally. A call and its result are one unit when trimming to max_context_tokens; an action with no result yet, and tool turns with no preceding action (pipeline digests), fall back to the textual form. The system prompt and tools array are unchanged (Rule 1).

Observation folding and small models

Raw tool outputs stay in the prompt for \(K=2\) turns, then fold into a one-line tombstone (see Built-in Tools). For small models this has consequences (#27), seen in the course-builder example:

  • Re-viewing files loops, then invents data. The model runs edit view, the output folds two turns later, it views again, and eventually writes from memory.
  • What worked: one compact joined view, used right before the write, that ends with the exact next call. The course-source <module> tool returns a module's plan and chosen videos in one output and finishes with NOW WRITE THIS MODULE: act("edit write part-N.md", ...), so the data is still in the prompt when the write happens.
  • Write long outputs in small pieces. One big write dropped a whole section. The writer skill writes one part-N.md per module with edit write, then a course-build tool concatenates them into course.md.