## `shkolnik/browser-use-benchmarks` — investigation summary ### What the project is A small public GitHub project whose sole job is to be a **builder service** that produces prebuilt, self-contained Docker images for the standard **web-agent benchmarks** — MiniWoB++, WebShop, WebArena, and VisualWebArena — plus the reproducible build pipeline behind them. It publishes 8 images to GHCR (e.g. `webarena-shopping`, `webarena-reddit`, `webarena-gitlab`, `vwa-classifieds`, `webshop-server`, `miniwob-server`). ([README](https://github.com/shkolnik/browser-use-benchmarks/blob/main/README.md)) **Why it exists:** the benchmark *tasks* are easy to get, but the *websites they run against* are not. Upstream, each benchmark environment is a mix of multi-gigabyte `docker save` tars on flaky university mirrors, Google-Drive datasets, and multi-step setup scripts that patch containers after boot — standing up the full set from scratch takes days and is hard to reproduce. This project packages each benchmark website as a single pullable image: self-contained (all data baked in, no runtime network fetches), reproducible (sha256-pinned datasets, Dockerfiles never touch the network), relocatable (rewrites links via `HTTP_HOST`/`HTTP_PORT`), and health-checked. ([README](https://github.com/shkolnik/browser-use-benchmarks/blob/main/README.md)) ### What changed recently (Aug 14–15, 2026 — the active development window) The commit history shows intense, focused engineering on the build pipeline's correctness and scale: - **"Media path" work** — a new mechanism to handle huge dataset archives (19–116 GB per image) by *demuxing* them into bucket tars without unpacking, routing large data (Magento, Wikipedia ZIM, OSRM routing, Nominatim, Reddit/classifieds photos, map tiles) around the build, and writing bucket tars concurrently with per-phase timing. ([commits](https://github.com/shkolnik/browser-use-benchmarks/commits/main/)) - **Build anchoring & provenance** — stamping the commit an image was built from into the image, scoping rebuilds by what each image was actually built from, and promoting `:latest` only after the attestation exists. ([commits](https://github.com/shkolnik/browser-use-benchmarks/commits/main/)) - **Ordering/robustness fixes** — enqueueing the image matrix longest-first, draining tar listings instead of closing pipes, clearing world-writable bits instead of refusing archives, and keeping backup tars out of the partitioned tree. ([commits](https://github.com/shkolnik/browser-use-benchmarks/commits/main/)) - **A new `push_images` subcommand** so a run can be timed without publishing. ([commits](https://github.com/shkolnik/browser-use-benchmarks/commits/main/)) ### Why it matters — the technical substance The design docs are unusually rigorous, and the "why it matters" is best captured by the **silent-failure traps** the project documents and defends against ([build-data-path.md](https://github.com/shkolnik/browser-use-benchmarks/blob/main/docs/build-data-path.md)): - **`du` is not the size of a layer.** A layer is a tar (512-byte headers + padding), and `du` is wrong in both directions — measured 4× over on many small files, and it reports **0 K for a 64 MB sparse file**, which can produce a layer the registry refuses at push time, after the whole image is built. - **A zero-member tar is not an archive to docker** — `ADD` drops it as a literal file and the build goes green with a wrong image. - **`ADD --chown` is silent when it can't resolve a name** — everything lands `root:root` and the build exits clean; the first sign is Postgres refusing to start at run time. - **`COPY --from` commits the whole archive; `rm` afterwards reclaims nothing** (whiteout layers keep the bytes). - **An audit of an extracted tree audits the extractor** — `tar x` applies umask, drops setgid, and makes ownership host-dependent, so a passing audit only proves "acceptable on this host." The project's **registry-size probe** ([registry-limits.md](https://github.com/shkolnik/browser-use-benchmarks/blob/main/docs/registry-limits.md)) empirically settled real infrastructure questions: GHCR accepts ≥100 GiB single images on a free-tier account, and Docker Hub accepts a **128.89 GB** single image with ~10 GiB layers (refuting the reported 100 GB per-image cap and the 5 GB S3-multipart layer ceiling), so mirroring the ~280 GB fleet needs no rechunking. The per-account total remains an open question. ### Broader context This sits at the center of the fast-moving **AI browser-agent evaluation** space. WebArena is the canonical self-hostable web-agent benchmark (1.6k stars, 244 forks) whose own README warns users to "setup your own WebArena websites" for correct evaluation — exactly the painful, multi-step process this project automates ([web-arena-x/webarena](https://github.com/web-arena-x/webarena)). It also connects to the broader **browser-use** ecosystem — the popular open-source framework (79k+ GitHub stars) for letting LLMs drive real browsers ([browser-use/browser-use](https://github.com/browser-use/browser-use)) — and to newer derivatives like WebArena Verified and WebArena-Lite that keep pushing reproducible evaluation forward. ### Strongest sources - [README.md](https://github.com/shkolnik/browser-use-benchmarks/blob/main/README.md) — what the project is, the image list, why it exists - [docs/design.md](https://github.com/shkolnik/browser-use-benchmarks/blob/main/docs/design.md) — full architecture, cache-invalidation model, "the pin IS the key" - [docs/build-data-path.md](https://github.com/shkolnik/browser-use-benchmarks/blob/main/docs/build-data-path.md) — the measured silent-failure traps (the technical heart of why this is hard) - [docs/registry-limits.md](https://github.com/shkolnik/browser-use-benchmarks/blob/main/docs/registry-limits.md) — empirical GHCR/Docker Hub size-limit findings - [Commits on main](https://github.com/shkolnik/browser-use-benchmarks/commits/main/) — the Aug 14–15, 2026 change window - [web-arena-x/webarena](https://github.com/web-arena-x/webarena) — the upstream benchmark this packages - [browser-use/browser-use](https://github.com/browser-use/browser-use) — the broader agent framework ecosystem **Bottom line:** this is a small but technically deep infrastructure project that turns the notoriously painful, days-long, non-reproducible setup of web-agent benchmarks into a set of pullable, health-checked Docker images — and its docs contain genuinely valuable, measured engineering knowledge about building and shipping multi-hundred-GB container images correctly.