Run the smoke-test task from §7 again, but this time treat it as a real experiment: record tool-call count, total wall-clock time, total tokens in/out, and total cost for each of the four providers. Produce one row of the matrix per provider. This is Forge Lab Log Entry 003.01.

#!/usr/bin/env bash
# Forge Lab — Experiment 01 · First Contact
set -euo pipefail
cd ~/forge/lab/smoke

PROMPT="Read util.py, then write test_util.py with one pytest case for add()."
OUT=~/forge/lab/exp01-$(date +%Y%m%d-%H%M)
mkdir -p "$OUT"

for p in openai gemini deepseek ollama; do
  echo "=== $p ===" | tee -a "$OUT/log.txt"
  git reset --hard -q
  openclaw provider switch $p > /dev/null

  START=$(date +%s)
  openclaw task "$PROMPT" --quiet
  END=$(date +%s)

  openclaw session show --last --json > "$OUT/$p.json"
  echo "  wall_seconds: $((END - START))" >> "$OUT/log.txt"
done

# Quick matrix from the JSON traces
jq -r '[.provider, .tool_calls|length, .tokens.total, .cost_usd, .duration_ms] | @tsv' \
  "$OUT"/*.json | column -t

• Output file test_util.py exists and runs pytest -q cleanly for each provider
• Tool-call counts within ~1.5× of each other across providers (large drift = a prompt the model misreads)
• Cost on Ollama is exactly zero (this is the canary — non-zero means a misrouted call)
• Each session JSON saved to $OUT/<provider>.json for later replay

Five tasks of increasing complexity, run across all four providers, recorded in one matrix. The deliverable is a 5×4 grid of cells with cost, tool-call count, and a quality score. This becomes the reference table the rest of the lab cites — and the table the community does not yet have for OpenClaw + DeepSeek V4 as of this week.

The practitioner's value is publishing a matrix others can reproduce. Every cell needs exact provider + model + date — drift in any of those invalidates comparison.

Each task gets two scores, judged independently on a 0–3 scale. Correctness: did the output do what was asked? Reasoning independence: did the model figure out the task on its own, or did it require operator nudging mid-loop? Recording these separately is what makes provider comparison interesting — a model can be highly correct only when heavily guided, and that should not score the same as a model that just gets it right.

DeepSeek V4 ships in two sizes with very different price points. The hypothesis to test: V4-Flash is cost-equivalent to local inference but quality-equivalent to a frontier API on agentic-coding tasks. The control is GPT-5.4 (frontier reference); the contrast is V4-Pro (the high-effort variant, where DeepSeek's max-reasoning mode lives). Three providers, the same five tasks from Experiment 02, with cost-per-completed-task as the headline metric.

#!/usr/bin/env bash
# Forge Lab — Experiment 03 · DeepSeek V4 Head-to-Head
set -euo pipefail
cd ~/forge/lab/scratch

declare -a PROVIDERS=("openai" "deepseek" "deepseek-pro")
declare -a TASKS=(T1 T2 T3 T4 T5)

OUT=~/forge/lab/exp03-$(date +%Y%m%d)
mkdir -p "$OUT"

for p in "${PROVIDERS[@]}"; do
  openclaw provider switch "$p" > /dev/null
  for t in "${TASKS[@]}"; do
    for run in 1 2 3; do                          # 3 runs per cell, take median
      git reset --hard -q
      openclaw task "$(cat prompts/$t.txt)" --quiet
      openclaw session show --last --json > "$OUT/${p}_${t}_run${run}.json"
    done
  done
done

# Build the matrix — median wall_ms and median cost_usd per (provider, task)
python3 ~/forge/lab/aggregate_matrix.py "$OUT" > "$OUT/matrix.tsv"

• H1 — V4-Flash completes T1–T3 at quality scores within 0.5 of GPT-5.4-mini, at <15% the cost
• H2 — V4-Pro completes T5 (multi-step) at higher independence score than V4-Flash, justifying the price step
• H3 — On T4 (web-grounded) the gap closes — citation discipline is more about prompting than model size
• H4 — V4 reasoning_effort=max produces measurably better T3 (fix-the-bug) results than effort=high, but at >3× latency

The pivot question of the lab: at what task complexity does local inference (Gemma 4 26B on Ollama) stop being the right answer compared to a cheap API (DeepSeek V4-Flash)? The hypothesis is not that one wins outright. The hypothesis is that there is a clean cutoff in task complexity above which the API saves you more time than the local model saves you in dollars.

"""Forge Lab — Experiment 04 · Local vs API crossover analysis.
Reads session JSONs from experiments 02–03 and finds the task at which
the time-saved by API exceeds the dollars-spent on API.
"""
import json, glob, statistics
from pathlib import Path

DEV_HOURLY_USD = 90     # lab operator's effective hourly rate
SECONDS_PER_USD = 3600 / DEV_HOURLY_USD

def load_runs(provider, task):
    files = glob.glob(f"~/forge/lab/exp03-*/{provider}_{task}_run*.json")
    return [json.loads(Path(f).expanduser().read_text()) for f in files]

for task in ["T1", "T2", "T3", "T4", "T5"]:
    local = load_runs("ollama", task)
    api   = load_runs("deepseek", task)

    local_s = statistics.median(r["duration_ms"] / 1000 for r in local)
    api_s   = statistics.median(r["duration_ms"] / 1000 for r in api)
    api_$   = statistics.median(r["cost_usd"]   for r in api)

    time_saved = local_s - api_s
    breakeven  = api_$ * SECONDS_PER_USD       # seconds the API has to save to pay for itself
    verdict    = "API wins" if time_saved > breakeven else "local wins"

    print(f"{task}: local {local_s:.1f}s · api {api_s:.1f}s · ${api_$:.4f} · {verdict}")

Move out of toy tasks. Pick a real, ~500-line Python or Swift project from your own work — small enough that one OpenClaw session can hold it in mind, big enough that no single tool call fixes everything. Run a substantial refactor against it on each provider. The goal is not just correctness — it is to count how many individually approved actions collectively constitute scope creep. This is the Agency Paradox composition problem, made measurable.

• Replace all blocking I/O calls in a small async Python service with proper async equivalents
• Extract a module's tightly coupled tests into property-based tests (Hypothesis or SwiftCheck)
• Migrate a UIKit view controller to SwiftUI while preserving public API and state semantics
• Replace bespoke logging with structured logging (structlog or os_log) end-to-end

After each provider's session, classify every tool call into one of three buckets: in-scope (directly serves the stated refactor), adjacent (related cleanup the model decided to do), out-of-scope (formatting, renames, dependency edits, comment additions the operator did not request). The ratio of these buckets is the composition footprint.

Synthesize Experiments 01–05 into a single allocation table: for each task class, which provider is the right default? The output is a one-page decision aid you keep open while working — not a benchmark report, a working tool. The matrix is reproducible, dated, and tied to the price sheets in effect on the day of the run.

• Median cost-per-task from Experiment 02 (5 tasks × 4 providers)
• Composition footprint from Experiment 05 (5 providers, real refactor)
• Provider price sheets, dated to the run day, archived as PDFs in ~/forge/lab/pricing/
• Operator hourly rate (used to convert wall time to operator cost)

Tags for this document: OpenClaw DeepSeek V4 Gemma 4 Apple Silicon Agency Paradox Provider Matrix 2026-04