jonathan.costin
← All projects

Benchmark

factobot

A benchmark that runs an AI agent in a real, headless Factorio game, holds it to what a human player can do, and scores it on game time to a milestone.

  • Rust
  • Lua
  • Python
  • bubblewrap
  • RCON
Source on GitHub
factobot architecture: the agent runs in a bubblewrap sandbox and reaches the game only through the fac CLI, which talks to the Rust daemon over a JSON socket; the daemon drives the headless Factorio server over RCON, and the playtest runner reads milestones on a separate socket.bubblewrap sandboxagentany harnessfac CLIRustfac daemonRustFactorioheadless + Lua modplaytest runnerPythonJSONRCONdev socketstartsScore: game ticks to each milestone, against human reference times.
ArchitectureFrom the repo's README

How it's built

  • A Rust daemon supervises the Factorio server, talks to it over RCON and serves a versioned JSON-lines protocol on two Unix sockets, one for the agent and one for the harness.
  • The agent reaches the game only through the fac CLI. It runs inside bubblewrap with its own PID and network namespaces and every capability dropped, so it can't see the harness, the repo or the host.
  • A Lua mod holds the agent to what a player can do: walking time, reach and the explored map. The game keeps running while the model thinks.
  • The harness reads milestones from the game in ticks and scores them against human reference times.

Why

The game runs at normal speed and never pauses for the agent, so time spent thinking is game time lost. An early version paused between commands, which made thinking free and the times unrealistic. Results record game ticks and wall-clock time separately.