Halo AI SDK

A Cache-First LLM Agent Framework

src/index.ts

typescript

// 🧠 Initialize Cache-First Agentic Loop
import { Halo, StablePrefix } from "@halo-ai/core";
import { deepseek } from "@halo-ai/adapters";

const halo = new Halo({
  model: deepseek("deepseek-chat"),
  prefix: new StablePrefix({
    system: "You are a highly accurate codebase analysis assistant.",
    tools: { getCodeContext, searchFiles },
  }),
});

const stream = await halo.run({
  prompt: "Optimize this algorithm and find best practices...",
  // Automatic prefix caching for instant responses in multi-turn chat
  cache: "first",
});

⚡

Dynamic Prefix Caching

Automatically recognize and lock prompt prefixes, eliminating duplicate token computation. Measured on a live DeepSeek 10-turn tool loop: 86.7% warm cache hit-rate and 51% lower prompt cost vs a cache-hostile baseline (see benchmarks/).

🛠️

Multi-Model Adapters

Out-of-the-box support for DeepSeek, OpenAI, and Anthropic. Standardize differences in provider-level caching under a unified high-level API.

🩹

Self-Repairing Tools

Built-in smart strategies automatically catch and fix invalid JSON or missing parameters from model outputs, guaranteeing loop resilience.

🌊

Real-Time Streaming

Full-stream event pipeline feeds Agent thinking traces, decision logs, and tool execution state to your frontend application.

⚡ From API Wrapper to Production-Ready Agent Architecture

Traditional frameworks only concatenate Prompts. Halo is built from the ground up to solve latency and cost pain points in enterprise-scale agent loops.

❌ Ordinary API Frameworks

⚠️Expensive Token Cost: Historical contexts accumulate linearly, forcing the model to re-evaluate all tokens on every turn.
⚠️Unpredictable Latency: As Context Windows grow, model inference queues and processing times stretch exponentially.
⚠️Fragile Execution: Crashes immediately when encountering minor model output formatting errors without any self-healing.

✨ Halo AI Core Primitives

✅Prefix Locking & Fast Caching: Stable prefixes (system prompts, tool definitions) stay cached for near-zero latency responses.
✅Precision Context De-noising: Integrated truncation policies maintain clean context sizes, shedding irrelevant history.
✅Self-Healing Pipelines: Intercept and repair invalid tool call arguments dynamically to preserve execution integrity.

📦 Modular Package Design

Halo follows the Unix philosophy, separating agent complexity into distinct, high-cohesion packages

packages/core

@halo-ai/core

Core Agent Engine

Manages the main Agent execution loop, context lifecycles, stable prefixes, and event stream lifecycles.

packages/adapters

@halo-ai/adapters

Multi-Provider Adapters

Provides native bindings for DeepSeek, OpenAI, and Anthropic, mapping raw API models to standard Cache-First interfaces.

packages/strategies

@halo-ai/strategies

Resilience Strategies

Provides truncation policies (Truncate Strategy) and tool calling self-healing rules (Basic Repair Strategy) to prevent loops from failing.

Ready to build high-performance agents?

Say goodbye to slow response times and heavy token bills. Start building cache-first agent systems optimized for production today.

Start My Journey