Boxes have no mortar either. Whether a stack stays up depends on where each box lands, and that call is
usually made by eye, with the next box already in hand.
99,800
nonfatal cases with days away from work in US transportation and warehousing (2020)
12,100
of them “struck by object or equipment”, a category broader than falling stacks
65
fatal injuries from “contact with objects and equipment” in 2020 (62–75 a year, 2016–2020)
Protocol locked in git (8a953e4) before the first trial · numbers from data/eval_report/
BUILT WITH OPENCV 5 + AWS
No mortar. No black box.Every joint is visible.
Fixed rules decide
GREEN Under 10%: place it
YELLOW 10–50%: a supervisor approves, takes the safer spot, or rejects
RED Over 50%: refuse and suggest a safer spot
Overrides need a name and a reason, and never change the decision
No language model makes the safety call
OpenCV 5
Calibrated camera, ArUco markers, pose averaged over 10 frames.
MuJoCo
150 to 600 simulated placements per check, with 1 cm position and 2 cm load uncertainty.
AWS Lambda
Same decision code as the laptop agent. Supervisor choices logged in CloudWatch.
111 tests · 25 locked trials · every decision logged
The camera measures.The simulation estimates.The rules decide.The supervisor signs.
HOW IT WORKS
Five checks before the next stone.One loop per box.
A laptop camera watches the stack. For every new box, Drystone runs the same loop, and what the camera sees changes what happens next: a shifted box is re-checked, a fallen box is re-planned.
01PerceiveOpenCV 5, 10 frames
02TwinMarker poses + box sizes
03Simulate150–600 MuJoCo placements
04Decideplace it · ask · move it
05CheckOK · OFF_PLAN · FELL
AFTER A RED
RED is a detour, not a dead end.Not +16 cm. Try +10.
Every RED comes with the nearest safer spot. Margin, overrides and what happens after a fall are rules MARGIN, OVERRIDE, NOT VISIBLE and FELL in the table below.
You see boxes.Drystone sees a wall.
Each 80 mm marker gives a box’s position and rotation. With the measured box sizes, that is enough to rebuild the stack and test the next box before anyone lifts it.
Ten rules, written as code.
Physics gives a probability. These rules turn it into an action, the same way every time.
Rule
When
What happens
GREEN
P(fall) under 10%
Place it: guide the box in
YELLOW
P(fall) 10–50%
Ask: a supervisor approves, takes the safer spot, or rejects
RED
P(fall) over 50%
Move it: refuse and suggest a safer spot
MARGIN
Every suggested spot
Must stay GREEN 1 cm further out
MORE SIMS
Near 10% or 50%
Simulate more (150 → 300 → 600); if still unclear, take the safer zone
OVERRIDE
Going ahead on RED
Name (2–60 characters) and reason (5–300 characters), logged; the zone never changes
RETAKE
Base marker not visible, markers change between frames, a box jitters more than 1 cm or is turned more than 12°
Up to 3 fresh readings, then ask the person
NOT VISIBLE
Placed box not seen, stack below intact
Ask: fell, or not placed yet?
FELL
Box gone after placement
Pick it up, plan the same box again
SETUP GATE
Evaluation mode, setup off by more than 1.5 cm
Refuse until fixed
The whole system, on one page.
Laptop agent with OpenCV 5, AWS Lambda running the same decision code, CloudWatch logs and a zero-spend budget alarm.
ARCHITECTURE
Webcam1280×720 · ChArUco calibration
OpenCV 5ArUco + pose · 10 frames, median
Digital twinstack by contact · box catalogue
MuJoCoMonte Carlo · 150–600 simulations
Agent rulesGREEN/YELLOW/RED · 1 cm margin
Supervisorapprove · suggest · override + reason
Box placedby a person
Check againOK/OFF_PLAN/FELL
↺ read the stack again
--aws · AWS ap-southeast-1
AWS Lambda + Function URL · same decision code · container image · 2048 MB · 60 s
Pick a real photo from our test stack, choose where the next box goes, and the same code that runs the laptop agent decides, live on AWS Lambda.
The first check can take up to 10 s while the server wakes up. Supervisor choices are logged without your IP address.
Try a placement
placeholder image
markerstacksuggested spotnext box
Where should the next box go?cm from the box below: − left, + right
0 cm
GREEN <10% · place itYELLOW 10–50% · askRED >50% · move it
DECISION
Pick a spot with the slider or an example, then press Check.
How it decided
total — ms
EVIDENCE
25 real stacks.Locked before the first box moved.
The test plan, all 25 setups and the pass criteria went into git before any trial ran (commit 8a953e4). We ran all 25 and report every result.
Locked criteria
Verdict: FAIL on 1 of 5 locked criteria. C2b missed by one fall (12 against a band of 6–11). No box fell in the GREEN zone.
Criterion
What it asks
Result
C1
No GREEN placement falls
0 of 8 fell PASS
C2a
At least 85% right on clear cases (P < 10% or > 70%)
14 of 14 PASS
C2b
Total falls inside the model’s 90% band
12 fell; 8.5 predicted (band 5.6–11.5) FAIL
C3
Brier score at most 0.20
0.125 PASS
C4
At least 20 valid trials
25 trials with an outcome PASS
Where the wall cracks.Reported, not patched.
YELLOW is riskier than its number
4 of 9 YELLOW placements fell, while the model gave them 25% on average. With a centre-of-mass margin of +0.5 to +2.5 cm, 4 of 8 trials fell (T16, T17, T01, T25). Treat YELLOW as more dangerous than its number says.
No quiet fixes
After the run we searched simulator settings for one that closes the gap. None did, so Drystone stays exactly as tested.
A simple rule is just as accurate
A static tipping check also gets 84% right on these trials. Drystone’s edge is calibration (Brier 0.125 vs 0.160), not more hits.
The agent, under test.
25/25placements checked correctly afterwards
12/12falls detected (7/7 in rehearsal)
4setups refused until fixed
0retakes, 0 fallbacks
0.6 cmmean placement error of boxes that stood; 10 of 13 within 1 cm
7.6 smedian decision time on AWS Lambda (round trip, 25 decisions)
Limits and responsible use.
Tested on three identical 1 kg boxes (29 × 17 × 19.4–19.6 cm), one floor, one laptop camera.
Every box needs a printed ArUco marker.
A decision aid prototype, not a certified safety device.
Lay the first stone.Start with zero install. End with a real camera.
Pinned versions throughout (Python 3.12, OpenCV 5.0, MuJoCo 3.15). Commands are copied from README.md and aws/README.md, run from the repository folder, one command per line.
Where Windows and macOS/Linux differ, both are shown. Each command was run in a fresh clone on Windows cmd.
No install
The Demo tab runs the full decision on AWS Lambda with three real photos.
Then, on any system (requirements-dev.txt includes requirements.txt and adds pytest):
pip install -r requirements-dev.txt
python tools/web_local.py
Then open http://127.0.0.1:8000/ (the address it prints; another port: --port 8080).
The full agent: camera + markers
You need a webcam, printed ArUco markers (DICT_4X4_50, 80 mm, IDs 0–2), a ChArUco board on something rigid and flat, and boxes listed in config/boxes.json. Measure what you printed. A calibration is only used if it passes the built-in quality check.
Calibrate your camera. Measure one printed square and use your own name. Match the board to the blue outline on screen; it saves by itself and moves to the next one. About 14 targets. Then the verdict is PASS or RETRY. PASS needs an RMS error of 1 px or less, a focal length uncertainty of 3% or less, and 9 of 12 image regions covered. PASS saves data/camera_mycam.yaml; on RETRY the terminal says what to fix. data/camera_laptop.yaml is our own webcam; calibrate yours.
Replace 30 with the square you measured (ours: 29.8).
Keys in the calibration window: N skip target · B back one target · C calibrate now · Q quit
For a clean reading: print on matte paper, tape it flat to something hard, light the board from the side (a lamp behind the camera shines back off a tilted board), and keep your fingers off the squares. The hints speak to you behind the board: “pull the top edge toward you” means that edge goes away from the camera. If one target will not match, N skips it.
Without the outlines: add --auto (a frame is saved whenever the board is held still at a new position).
Keys in the agent window: A / D move candidate 1 cm · SPACE decide / check · X cancel · Q quit
Optional: decisions on your own AWS Lambda instead of the laptop. Deploy it first (next section); the commands to run the agent with it are at the end of that section.
Your own AWS Lambda
Build the container, push it to Amazon ECR, and put a Lambda Function URL in front of it. Region ap-southeast-1; the commands read your AWS account ID from the AWS CLI. Full steps are in aws/README.md.
Before you start: Docker Desktop (running) and AWS CLI v2.
First-time setup
Billing and Cost Management → Budgets → create a zero-spend budget with an email alert.
Connect the AWS CLI: an access key of your own IAM user with ECR and Lambda permissions, region ap-southeast-1.
Build, tag and push the image: the build → tag → push commands under Update (up to docker push).
Lambda console → Create function → Container image. Function name: drystone (the Update commands use --function-name drystone). Image: drystone-service with your tag; architecture x86_64. Configuration → General: memory 2048 MB, timeout 60 s.
Configuration → Function URL: auth type NONE, CORS allow POST. Keep the URL in DRYSTONE_URL, not in the repository.
Update (every new version, tagged with the commit hash)
Run the agent with your Lambda (optional, after the first-time setup). The laptop keeps the camera and the window; each decision is computed on your Lambda, and if AWS fails the laptop computes it itself. The Lambda image uses the calibration in data/camera_laptop.yaml, so put yours in that file before you build it.
Windows (cmd)
for /f %i in ('aws lambda get-function-url-config --function-name drystone --region ap-southeast-1 --query FunctionUrl --output text') do set DRYSTONE_URL=%i
Run the test suite, and rebuild the evaluation summary from the published per-trial table, data/eval_report/trials.csv. It writes data/eval_report/from_trials/summary.md (ignored by git), the same as our summary.md except its first line. Our raw logs stay on our laptop (their photos show people).
python -m pytest tests -q
111 tests, about 3 minutes (runs real physics simulations).