Skip to main content
← All Eidos Patterns

Proxy Pattern

Provide a surrogate or placeholder for another object to control access to it.

The Problem​

You're calling an LLM API from your app. Every call costs money and takes time. The same question asked twice shouldn't cost twice. Users spam the button. And when the primary provider goes down, your app crashes.

The naive approach:

async function askLLM(question: string): Promise<string> {
  const response = await fetch("https://api.openai.com/v1/chat", {
    method: "POST",
    body: JSON.stringify({ prompt: question }),
  });
  return response.json(); // $0.003 every single time
}

// "What is the capital of France?" asked 10 times = 10 API calls = $0.03
// Provider goes down? App crashes.
// User spams? You burn through your rate limit.

No cache. No rate limit. No fallback. Every call hits the API, burns money, and prays the provider stays up.

The Solution​

Wrap the function with proxy layers. Same interface, three layers of protection:

import { memoize, throttle } from "@pithos/core/arkhe";
import { withFallback } from "@pithos/core/eidos/strategy/strategy";

// 1. Cache: same question = instant response, $0.000
const cachedAsk = memoize(askLLM);

// 2. Rate limit: max 1 call per second
const rateLimitedAsk = throttle(cachedAsk, 1000);

// 3. Fallback: if primary fails, try backup silently
const resilientAsk = withFallback(rateLimitedAsk, askBackupLLM);

// Consumer code is identical — just a function call
const answer = await resilientAsk("What is the capital of France?");
// First call:  1.2s, $0.003 — cache miss
// Second call: 2ms,  $0.000 — cache hit ⚡

Three Pithos utilities, three proxy layers. The consumer never knows about caching, rate limiting, or failover.

Live Demo​

Ask questions to a simulated LLM and watch the proxy in action: cache hits, rate limits, and provider failover.

Code
/**
 * LLM Proxy demo: memoize from Pithos for caching,
 * manual rate-limit, and fallback provider.
 */

import { memoize } from "@pithos/core/eidos/proxy/proxy";
import type { ProxyLogEntry, ProxyStats } from "./types";
import { RESPONSES, DEFAULT_RESPONSE, PRIMARY_COST, FALLBACK_COST, RATE_LIMIT_WINDOW } from "./data";

// Re-export for consumers
export type { ProxyLogEntry, ProxyStats } from "./types";
export { PRESET_QUESTIONS } from "./data";

async function simulateCall(question: string, delayMs: number): Promise<string> {
  await new Promise((r) => setTimeout(r, delayMs));
  return RESPONSES[question] ?? DEFAULT_RESPONSE;
}

export function createLLMProxy() {
  let simulateFailure = false;
  let lastCallTime = 0;
  let logId = 0;

  const cachedPrimaryCall = memoize(
    (question: string) => simulateCall(question, 800 + Math.random() * 600),
  );

  const cachedBackupCall = memoize(
    (question: string) => simulateCall(question, 1200 + Math.random() * 800),
  );

  const stats: ProxyStats = {
    logs: [],
    totalCost: 0,
    totalSaved: 0,
    cacheHits: 0,
    rateLimitHits: 0,
    fallbackHits: 0,
  };

  function addEntry(entry: ProxyLogEntry) {
    stats.logs = [entry, ...stats.logs];
  }

  async function ask(question: string): Promise<{ entry: ProxyLogEntry; stats: ProxyStats }> {
    const now = Date.now();
    const elapsed = now - lastCallTime;

    // Rate limit
    if (lastCallTime > 0 && elapsed < RATE_LIMIT_WINDOW) {
      const entry: ProxyLogEntry = {
        id: ++logId, question, type: "rate-limited", provider: null,
        duration: 0, cost: 0,
        response: `Rate limited. Try again in ${Math.ceil((RATE_LIMIT_WINDOW - elapsed) / 1000)}s.`,
        timestamp: now,
      };
      stats.rateLimitHits++;
      addEntry(entry);
      return { entry, stats: { ...stats } };
    }

    lastCallTime = now;
    const start = performance.now();

    // Primary or fallback
    const useFallback = simulateFailure;
    const call = useFallback ? cachedBackupCall : cachedPrimaryCall;
    const cost = useFallback ? FALLBACK_COST : PRIMARY_COST;

    const response = await call(question);
    const duration = Math.round(performance.now() - start);
    const isCacheHit = duration < 50;

    const entry: ProxyLogEntry = {
      id: ++logId, question,
      type: isCacheHit ? "cache-hit" : useFallback ? "fallback" : "cache-miss",
      provider: isCacheHit ? null : useFallback ? "backup" : "primary",
      duration, cost: isCacheHit ? 0 : cost, response, timestamp: now,
    };

    if (isCacheHit) {
      stats.cacheHits++;
      stats.totalSaved += cost;
    } else {
      stats.totalCost += cost;
      if (useFallback) stats.fallbackHits++;
    }

    addEntry(entry);
    return { entry, stats: { ...stats } };
  }

  return {
    ask,
    setSimulateFailure: (v: boolean) => { simulateFailure = v; },
    getSimulateFailure: () => simulateFailure,
    reset: () => {
      Object.assign(stats, { logs: [], totalCost: 0, totalSaved: 0, cacheHits: 0, rateLimitHits: 0, fallbackHits: 0 });
      lastCallTime = 0;
      logId = 0;
    },
    stats,
  };
}
Result
The complete source code is available on GitHub

Real-World Analogy​

A credit card is a proxy for your bank account. It provides the same "pay for things" interface, but adds access control (credit limit), logging (transaction history), and can work offline (signature-based transactions).

When to Use It​

  • Cache expensive computations or API calls
  • Rate-limit access to external services
  • Add fallback/retry logic without changing consumer code
  • Add logging or metrics transparently
  • Lazy-load resources on first access

When NOT to Use It​

If the function is cheap, fast, and reliable, a proxy adds overhead for no benefit. Don't cache a function that returns Date.now() or wrap a pure synchronous calculation in a rate limiter.

Proxy Variants in Arkhe​

Arkhe provides several proxy utilities:

import { memoize, once, throttle, debounce, lazy, guarded } from "@pithos/core/arkhe";

// Caching proxy — cache results by arguments
const cached = memoize(expensiveCalculation);

// Single-execution proxy — run only once
const initialize = once(loadConfig);

// Rate-limiting proxies
const throttled = throttle(saveToServer, 1000);  // max once per second
const debounced = debounce(search, 300);         // wait for pause in calls

// Lazy initialization proxy
const config = lazy(() => loadExpensiveConfig());

// Conditional execution proxy
const adminOnly = guarded(deleteUser, (user) => user.isAdmin);

API​

These functions are from Arkhe and re-exported by Eidos: