Engineered for Autonomous Computer-Use Evaluation

The Production Harness for
Computer Use AI Agents

Bridging high-speed Model Context Protocol (MCP) software APIs, isolated Docker sandbox runtimes, and token-optimized multimodal vision with strict ground-truth verification.

Launch Simulator Config Generator Architecture Guide
100% Deterministic Resets
84% Token Savings vs Pure GUI
24x Faster via StoryTold MCP
Zero Host Destruction Risk

Interactive Harness Execution Simulator

Experience real-time ReAct computer use execution across the Shell, GUI desktop display, and StoryTold WordCraft MCP server.

Display Mode:
Step 1 of 5
PTY Shell Session (Windowed) Bash 5.2
sandboxuser@nexus:~$ ls -la inbox/
total 12 -rw-r--r-- 1 sandboxuser staff 98 Oct 24 inv_001.txt -rw-r--r-- 1 sandboxuser staff 102 Oct 24 inv_002.txt -rw-r--r-- 1 sandboxuser staff 95 Oct 24 inv_003.txt
Virtual Display (:99 1024x768)
StoryTold WordCraft Desktop State: Synchronized
StoryTold WordCraft Document Editor

Q3 Engineering Architecture Memo

Executive Summary

NexusCUH delivers hybrid shell, GUI, and MCP orchestration with high reliability.

[1] Save
[2] Format
[3] Title Area
Model Context Protocol (MCP) JSON-RPC StdIO Pipe
Cognitive Router Decision:
Level 1: MCP Deterministic Tool (Fastest)
MCP JSON-RPC Packet: 
{
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "name": "wordcraft__new_document",
    "arguments": { "title": "Q3 Engineering Architecture Memo" }
  }
}
Step Telemetry & Tokens:
Prompt Tokens: 340
Visual Tokens: 0 (Saved via MCP)
Duration: 18.4ms
Ground-Truth: ASSERT_OK

Engineered for Reliability in Mind

How NexusCUH solves the fatal flaws of naive screenshot-clicking wrappers through hierarchical orchestration.

Tri-Modal Cognitive Router

Standard computer-use agents rely exclusively on screenshots and mouse clicks ($1,600$ visual tokens per step, hallucinated pixel coordinates, high latency). NexusCUH implements a 4-tier cognitive priority hierarchy:

  • Priority 1: MCP Software Tools — Executes deterministic API actions on desktop software (StoryTold WordCraft, SheetCraft, SQL) with 0 visual tokens and 100% precision.
  • Priority 2: Direct FileSystem Tools — Atomic read, write, append, and search avoiding shell escaping bugs.
  • Priority 3: Stateful Shell Session — Background PTY execution with windowed head/tail output truncation.
  • Priority 4: Desktop GUI Fallback — Perceptual frame diffing deduplicates unchanged screens to preserve context.
router.py 
# Cognitive Routing Hierarchy
PRIORITY = {
    "mcp": 1,         # 100% Deterministic, 0 Visual Tokens
    "filesystem": 2,  # Atomic I/O, Safe Quoting
    "shell": 3,       # Batch processing, CLI tools
    "computer": 4     # GUI clicks & visual verification
}

Model Context Protocol (MCP) Bridge

Direct native integration with StoryTold desktop applications (WordCraft, DeckCraft, ArtCraft) and arbitrary enterprise software.

  • JSON-RPC 2.0 Stdio/SSE Engine — Asynchronous process management with multiplexed requests.
  • Dynamic Tool Translation — Automatically maps MCP tool schemas into Anthropic, OpenAI, and Gemini function signatures.
  • AST & Layout Parity — Directly query document tree structures and assert rendering parity.
client.py 
# MCP Client Tool Call Protocol
res = await mcp_client.call_tool(
    "wordcraft__type_text",
    {
        "text": "Executive Architecture Summary",
        "paragraph_type": "heading1"
    }
)
# Returns structured AST state and layout confirmation

Zero-Trust Sandboxed Execution

Complete containerized isolation preventing any host destruction or credential leaks.

  • Xvfb Virtual Display — Headless X11 server at display :99 with 1024x768 resolution.
  • HTML5 noVNC Live Stream — Watch the agent interact with desktop windows live in your browser on port 6080.
  • Deterministic Rollbacks — $O(1)$ pristine snapshot restoration before every benchmark run.
docker run 
# Launch isolated Docker sandbox container
docker run -d \
  --name nexus_sandbox_runtime \
  -p 5900:5900 -p 6080:6080 \
  -v ./workspace:/home/sandboxuser/workspace \
  -e DISPLAY=:99 \
  nexus-computer-use-sandbox:latest

Ground-Truth Multi-Modal Verification

Exit code zero does NOT mean a task succeeded. NexusCUH evaluates assertions against actual disk and application state.

  • File Hash & JSON Assertions — Verifies SHA-256 signatures, row counts, and JSON schemas.
  • MCP State Assertions — Queries application internal AST rather than relying on visual guesses.
  • Cryptographic Evidence Logs — Produces diagnostic proof of execution for audits.
verifier.py 
verifier = MultiModalVerifier([
    FileExistsAssertion("reports/summary.json"),
    FileJsonAssertion("reports/summary.json", {"total": 412.50}),
    MCPStateAssertion("wordcraft", "inspect_document", {}, check_paragraphs)
])
report = await verifier.verify(sandbox, mcp_manager)

Production Safety & Guardrails

Multi-layered defenses against runaway loops, massive token bills, and prompt injections.

  • Command AST Guard — Blocks destructive shell commands (rm -rf /, fork bombs, disk formats).
  • Prompt Injection Sanitizer — Quarantines adversarial instructions detected in files or OCR text.
  • Sliding-Window Loop Detector — Intervenes if identical actions repeat without environment progress.
  • Resource & USD Budget Enforcer — Caps steps, time, tokens, and dollar expenditures.
loop_detector.py 
# Automatic intervention on cyclic behavior
is_loop, warning = loop_detector.record_step(
    tool_name="computer",
    tool_args={"action": "left_click", "coordinate": [200, 300]}
)
if is_loop:
    # Force agent to pivot strategy

Interactive Configuration Builder

Configure your deployment backend, LLM model, MCP servers, and safety guardrails. Export ready-to-run YAML, Python, or CLI commands.

Max Step Budget: 30 steps
Max USD Spend: $5.00 
# Generated Config will appear here

Evaluation Benchmarks & Performance

Empirical comparison: Pure Vision-Based GUI vs Hybrid NexusCUH (MCP + Shell + GUI).

Token Footprint Comparison (Tokens / Task)

-84% Reduction

Execution Latency Comparison (Seconds)

24x Acceleration
Benchmark Task Category Pass Rate Steps Tokens Avg Duration Verification Method
task_01_receipt_processing Shell & Filesystem 100% (PASS) 2 380 0.01s JSON Schema + Row Aggregation
task_02_wordcraft_mcp_workflow MCP Desktop Software 100% (PASS) 4 720 0.04s WordCraft AST + Parity Sync
task_03_gui_form_interaction Desktop GUI 100% (PASS) 3 1,850 0.22s Display State + Log Regex
task_04_error_resilience Fault Tolerance 100% (PASS) 3 610 0.03s File Permissions + Ground Truth

Developer Quickstart

Install, launch the sandbox, and run your first computer use task in under 60 seconds.

1

Install with uv

Clone the repository and install dependencies in an isolated virtual environment:

bash 
git clone https://github.com/subasah/computer-use-harness.git
cd computer-use-harness
uv venv && source .venv/bin/activate
uv pip install -e ".[dev]"
2

Configure Environment

Copy the environment template and configure your provider API key:

bash 
cp .env.example .env
# Set ANTHROPIC_API_KEY=your_key or use offline mock
3

Run Task or Benchmark

Execute any task or run the full evaluation suite with rich telemetry:

bash 
# Run evaluation suite
nexus eval --output results.md

# Inspect connected MCP tools
nexus mcp

# Run custom task
nexus run -t "Create memo in WordCraft" -p mock