Testing and Verification

The environment modules (Sky, Weather and Forests) are checked by three kinds of tests beyond their unit tests: the sky’s look-up tables against an independent reference, golden screenshots of the whole stack, and a long time-lapse soak. They live with the engine’s Python tests (python/tests in the engine source) and the samples they drive (python/samples). Every GL test renders offscreen in a subprocess of its own, so a crash or a driver error fails one test, not the run.

The targets are Windows and Linux on OpenGL 4.5. The tests also run on a macOS development machine (OpenGL 4.1), and on Linux without a GPU under Xvfb, where Mesa’s llvmpipe provides OpenGL 4.5; that is how the engine’s CI runs them.

The sky’s look-up tables

test_sky_luts.py renders the transmittance table and the multiple-scattering table of acron.sky, reads them back from the GPU and compares them with a double-precision model of the same Earth written in plain Python from Hillaire’s paper (_sky_reference.py):

  • the transmittance, as optical depth, within 3 % at a grid of texels (the shader’s 40 steps under-sample the low aerosol layer on steep rays; that is the 3 %), and the zenith transmittance at the ground in closed form (0.936, 0.864 and 0.759 in red, green and blue);

  • the CPU twin (SkyModel, which lights the DirectionalLight) against the same reference and against the GPU table;

  • the multiple-scattering table against a brute-force integral (within 4 %), and its energy bound;

  • a hazy, humid look reaching the tables through the frame values.

The finest brute force takes about two minutes; it runs when ACRON_SLOW_TESTS=1 is set:

cd python/tests
pytest test_sky_luts.py
ACRON_SLOW_TESTS=1 pytest test_sky_luts.py

Golden screenshots

test_nature_goldens.py renders a fixed scene of the whole stack (samples/nature_stack.py: the sky with its probe and shadows, the frame pipeline, clouds, volumetric fog, a forest edge, grass and three grey boxes) for every combination of

  • four hours: dawn (6:00), noon, golden hour (18:30) and night (23:30), on 21 June at Zagreb;

  • four weathers: fair, overcast, storm and fog;

  • two quality presets: low and high.

That is 32 pictures of 320 x 180, each rendered at that size on a fixed 30 Hz clock with the day clock stopped and every seed fixed, after 16 frames, with shadow cascades of 384 texels a side (the presets’ 1024 and 2048 are for 1080p: the same shadow texels a pixel). The fog’s froxel grid and the forest’s levels of detail follow the small screen as they would any window. The first cell renders alone and fills the prototype, impostor and blue-noise caches; the rest render four at a time (ACRON_GOLDEN_JOBS), and on Mesa’s software rasterizer, llvmpipe, each gets its share of the cores (LP_NUM_THREADS). On a four-core machine under llvmpipe the first cell takes about 25 s and the others 24 to 30 s each. Each picture is compared with its golden by two measures:

  • SSIM of the brightness over 8 x 8 blocks: the mean must be at least 0.95;

  • the CIE76 colour difference of the two pictures averaged down to 80 x 45: a mean of at most 1.2 and a 95th percentile of at most 3.0.

The marks are two to three times what rendering the same cell a frame or two later changes (the wind moves the grass, the clouds drift), and a camera exposure of +0.5 EV, or the volumetric fog switched off in the fog weather, fails them. The test proves this again on every run against the goldens.

Pictures differ between renderers by far more than that, so the goldens are kept per platform, in tests/goldens/nature/<key>/. The key is made from the system, the GL version and the renderer, for example linux-gl4.5-llvmpipe or darwin-gl4.1-apple-m4-pro. When there are no goldens for the machine’s key, the test skips and says how to make them.

To bless the goldens of a machine (the first time, or after an intended change of the look), render them, look at them, and commit the directory:

ACRON_BLESS_GOLDENS=1 pytest test_nature_goldens.py

ACRON_GOLDEN_KEY sets the key by hand, and ACRON_GOLDEN_DIR reads and writes the goldens somewhere other than tests/goldens/nature. ACRON_GOLDEN_JOBS (4 by default) sets how many cells render at once. A failing cell’s picture is kept in a temporary directory, which the failure message names, or in ACRON_GOLDEN_FAILURES.

The scene also renders a single cell from the command line:

python samples/nature_stack.py --screenshot out.png --size 320x180 \
    --shadow-resolution 384 --hour 18.5 --weather storm --preset low

The time-lapse soak

samples/nature_stack.py --soak MINUTES runs the same scene on the real-time clock for as long as asked, with the day turning fast (24 game hours in --day-minutes, 20 by default) and the weather cross-fading through fair, overcast, storm, fog, clear and rain every --weather-minutes (7 by default). It records

  • every frame’s CPU time and its time until the GPU has finished it, and the frames over twice the median and over 33 ms, with the hour and weather of the worst;

  • the process’s resident memory once a minute, judged from the end of the first weather cycle (until then every weather is new: its shaders and buffers are being made);

  • every history buffer (the anti-aliasing, the ambient occlusion, the cloud resolves and panoramas, the fog while it is on, the sky’s tables), read back once a minute and checked for NaN and infinity.

A line of JSON per minute goes to --log, and a summary at the end. The exit status is 1 when a buffer held NaN, the memory grew by more than --max-rss-growth (64 MB), or more than --max-spikes frames took over 33 ms. The full soak is two hours; keep the machine from sleeping for that long (on macOS with caffeinate), or the soak pauses with it (it notices, and leaves those frames out):

caffeinate -dimsu python samples/nature_stack.py --soak 120 --log soak.jsonl

--pstats HOST:PORT connects the soak to a PStats server with the GPU timer queries on, for per-pass GPU times where the driver supports them (macOS reports zero). A two-minute version (the day in two minutes, four weathers twice) is an opt-in test, as it holds the GPU:

ACRON_SOAK_TESTS=1 pytest test_nature_soak.py

Frame times say little on a software renderer, so the soak does not run in CI; run it by hand on the machine whose frame rate matters.