The Frame Pipeline¶

This page describes part of the Python framework that comes in the acron wheel. Its native half, the blue-noise generator, is available to C++ as well: see Blue Noise.

acron.pipeline is an opt-in frame pipeline for scenes lit by the environment modules (Sky, Weather and Forests). Once a game makes a FramePipeline, the main camera renders into offscreen targets instead of the window, and four techniques run on them:

Temporal anti-aliasing (TAA) after Karis, “High-Quality Temporal Supersampling” (2014): a sub-pixel jitter each frame, a history reprojected by motion vectors, clipped to the neighbourhood’s colours and blended, then a light sharpen. It replaces multisampling.
Motion vectors from every shader that opts in, including wind-swayed trees, instanced forests and moving nodes; the sky and any other surface get the camera’s motion from the depth.
Spatiotemporal blue noise (Wolfe et al. 2022): the per-pixel random numbers of the shadow filter, the cloud march, the light shafts and the forest’s LOD dither come from masks that are blue over the screen and over time, so TAA removes their grain.
Ground-truth ambient occlusion (GTAO, Jimenez et al. 2016, “Practical Real-Time Strategies for Accurate Indirect Occlusion”): the occlusion of the sky light, applied by the materials to their ambient term with the paper’s multi-bounce fit.

A final pass sharpens, applies the night’s rod shift and tonemaps into the window; the GUI draws on top as before. Every pass is a fragment pass, so the pipeline runs on the Windows and Linux targets (OpenGL 4.5) and on a macOS development machine (OpenGL 4.1).

Making one¶

The pipeline adds frame values and #define s that lit shaders are compiled with, so make it right after the environment and before anything compiles a lit shader – the shadows, the clouds, the probe, the forest and the game’s own materials:

from acron.sky import DayClock, Environment, EnvironmentProbe, SunShadows
from acron.pipeline import FramePipeline
from acron.trees import Forest

env = Environment(base, clock=DayClock(latitude=45.8, longitude=16.0))
pipe = FramePipeline(env, "high", cache_dir=user_cache_dir)
shadows = SunShadows(env, resolution=2048, distance=220)
probe = EnvironmentProbe(env)

terrain.set_shader(env.make_shader("lit_basic.vert", "lit_basic.frag"))
forest = Forest(env, shadows=shadows, alpha_to_coverage=False)

pipe.track(car)               # motion vectors for a node the game moves

Leave multisampling off (framebuffer-multisample 0): the scene buffer is single-sampled and TAA does the anti-aliasing, which is also why the forest is made with alpha_to_coverage=False (its leaves then use an alpha test, which TAA smooths). cache_dir keeps the generated blue noise on disk between runs; without it the masks are generated at startup (about a second for the two channels).

A shader compiled before the pipeline existed still draws: it writes no normals or motion, and both are reconstructed from the depth. One compiled under it still draws after destroy(): the pipeline leaves 1 x 1 stand-ins for its blue noise and AO, the environment writes no motion and AO off every frame, and the sky is recompiled without its defines.

The passes¶

Pass	Size	What it does
scene	window	the main camera: HDR colour (RGBA16F), depth, view-space normals and motion vectors (two RGBA16F auxiliary targets)
gtao	1/`ao_scale`	horizon search on depth and normals
ao temporal	1/`ao_scale`	3x3 depth-aware denoise and reprojected history (ping-pong)
ao out	1/`ao_scale`	the newest result copied into `ao_texture`, the one AO texture the scene reads
taa	window	Karis’s resolve into the history (ping-pong)
composite	the window’s region	sharpen, Purkinje shift, tonemap, or a debug view

The scene buffer renders after the sky, cloud, shadow and probe passes and before the others (buffer sort -10 to -6).

Targets. textures holds them by name: color, depth, normal (view space: x right, y up, z towards the eye; a = 1 where a shader wrote one), motion (rg = screen uv now minus uv a frame ago; b = 1 where a shader wrote one), the newest ao and ao_raw, and the TAA’s resolved image. A game’s own post passes read them; each is exactly its pass’s size. After reading one back to the CPU, clear its RAM image (tex.clear_ram_image()), or the engine uploads it over the next render.

Debug views. debug_view set to "ao", "motion", "normals" or "depth" shows that target in the window instead of the image.

Presets¶

quality names a preset; settings overrides any of its keys, and set_quality() switches at run time (the passes are made again; materials keep the defines they were compiled with):

pipe = FramePipeline(env, "high",
                     settings={"sharpen": 0.4, "ao_radius": 3.0})
pipe.set_quality("low")

Preset	TAA	GTAO	1080p, macOS (M4 Pro)
`off`	no	no	+0.1 ms over no pipeline
`low`	yes	half res, 1 slice x 3 steps	+1.4 ms
`medium`	yes	half res, 2 slices x 3 steps	+1.4 ms
`high`	yes	half res, 2 slices x 4 steps	+1.5 ms
`ultra`	yes	full res, 3 slices x 6 steps, bent normals, 16- frame jitter	+3.0 ms

The timings are wall time of finished frames on a light scene with the pipeline against the same scene without it, measured on a macOS development machine (OpenGL 4.1) – not on the Windows and Linux targets, whose numbers will differ. Of high’s 1.5 ms, the TAA resolve takes about 0.4 to 0.6 ms, the GTAO search and its temporal pass 0.05 to 0.2 ms each, and the rest is the scene’s extra targets, the materials’ AO lookups and the composite; ultra’s full-resolution search takes about 1.1 ms. measure() repeats such a measurement on the game’s own scene: the frame time and each pass group’s share (the frame time with it switched off taken from the frame time with it on, interleaved batches, medians). python samples/frame_pipeline_demo.py --measure --size 1920x1080 prints it for the forest sample.

The other keys: jitter (the Halton sequence’s length), feedback and feedback_moving (how much history a still and a fast pixel keep), motion_px, gamma (the variance clip’s width), filter (the reconstruction filter: 1 is Karis’s soft one, 3 about a pixel), sharpen, sky_history, ao_radius (world units), ao_falloff, ao_max_px, ao_thickness, ao_power, ao_feedback, ao_bent, blue_noise, purkinje and tonemap: "shoulder" (the default: linear up to 0.76, so a lit albedo shows as itself, then Khronos PBR Neutral’s shoulder rolls the brighter off to white instead of clipping each channel), "neutral" (the whole Khronos PBR Neutral, with its toe: punchier, but it crushes a moonlit scene and pulls dark greens towards olive), "clamp" (the clip, as without the pipeline), "aces" or "reinhard".

Temporal anti-aliasing¶

Each frame the main lens is shifted by an offset from the Halton (2, 3) sequence, less than a pixel, through its film offset; the shift is taken off again before the environment writes the frame’s matrices, so VIEWPROJ – and the clouds’ reprojection and the fog’s froxels built on it – stay unjittered. The frame value JITTER holds this frame’s and the last frame’s offset in NDC for any pass that needs them.

The resolve, per pixel:

the 3x3 neighbourhood’s colours, weighted by Karis’s 1 / (1 + max) so that a bright outlier cannot ghost, filtered back to the pixel centre with a Gaussian fit of a Blackman-Harris window, and their mean and variance in YCoCg;
the motion vector of the nearest surface in the neighbourhood (the edge follows the foreground);
the history at the reprojected position, sampled with a five-tap Catmull-Rom filter, clipped towards the mean of the variance box (Salvi);
less history where the pixel moves fast (feedback_moving) and where its motion disagrees with the motion the history was made with (a disocclusion);
sky pixels away from geometry keep only sky_history.

The clouds resolve themselves: the cloud buffer is sampled unjittered and already accumulates its own history, so TAA blending it again would smear it twice. sky_history (0 by default) keeps the sky as drawn; along a silhouette, where the neighbourhood holds geometry, the full resolve runs and the edge is anti-aliased as any other. A camera cut calls reset_history(); a window resize rebuilds the targets and starts afresh by itself.

Motion vectors¶

A shader writes its motion with two includes from acron.shaderlib:

// vertex
#include "gbuffer_vertex.glsl"
...
gbuffer_vertex(world_pos);               // or (world_pos, prev_world_pos)

// fragment
#include "gbuffer.glsl"
...
o_color = vec4(color, 1.0);
gbuffer_write(world_normal);

gbuffer_vertex projects the vertex through this frame’s and the last frame’s unjittered matrices; gbuffer_write writes the view-space normal and the difference. Without the pipeline (or with GBUFFER off) gbuffer_vertex is empty and gbuffer_write writes zeros, “none”, so a material can include them unconditionally. lit_basic and the forest’s shaders do. A shader of your own drawn into the scene targets must write all three outputs, zeros where it has no normal or motion: OpenGL leaves an output a shader does not write undefined, and NVIDIA’s driver copies the colour into it, which the AO and the TAA then read as a normal and a motion.

What moves, and how:

The camera: always right, from the two matrices.
The sky and any surface that writes no motion: the TAA and the AO reconstruct the motion from the depth and the camera; the sky (depth at the far plane) by its direction alone. Right for everything static. (The sky shader writes zeros, “none”, into the normal and motion targets, so no target is left undefined where it draws.)
A moving node: track() gives it a per-frame input (gb_prev_world, the world now to the world a frame ago), which gbuffer_vertex applies. It covers the node and everything under it that moves with it rigidly.
Trees in the wind: the tree shader evaluates the wind a second time with the previous frame’s time and wind (frame values MOTION.x and PREV_WIND), so swaying branches and fluttering leaves have exact motion. A game’s own vertex animation does the same: compute the previous position and pass it as gbuffer_vertex’s second argument.
Instanced forests: the instances stand still; their wind is per vertex as above. Impostors write their normal only (GBUFFER_NO_MOTION) and take the camera’s motion from their depth.
Skinned characters: only the node’s own motion. The engine keeps no previous bone matrices, so a limb’s motion relative to its body is missing; the neighbourhood clip keeps that from ghosting, at the price of a little blur on fast limbs.

Ambient occlusion¶

The GTAO pass searches, for each pixel, a few slices through the view direction; in each it finds the highest horizon on either side within ao_radius world units (fading over the outer ao_falloff of it, with the paper’s thickness heuristic for thin occluders such as trunks), clamps it to the hemisphere round the normal and integrates the visible arc in closed form. The slice angles and the step offsets come from the blue noise; the temporal pass filters 3x3 by depth and accumulates the result with the history, dropping it where the surface’s previous depth disagrees.

The scene reads the AO in its materials, through ao.glsl:

#include "gbuffer.glsl"
#include "ao.glsl"
...
vec3 occlusion = ao_multibounce(gbuffer_ambient_occlusion(), albedo.rgb);
vec3 ambient = env_irradiance(n) * occlusion;

ambient_occlusion(screen_uv) and ambient_occlusion(screen_uv, view_depth) (depth-aware: no halo where the AO is at half resolution) read it at a screen position; gbuffer_ambient_occlusion() reads it where this fragment was a frame ago. The AO is computed after the scene has drawn, so the scene reads the previous frame’s: looked up through each fragment’s own motion vector, a static or slowly moving scene cannot tell the difference, and a disocclusion shows unoccluded light for a frame. ao_multibounce is the paper’s fit for light bouncing between nearby surfaces: bright surfaces lose less. ao_bent_normal(screen_uv, n) turns the normal towards the open sky when ao_bent is on (the ultra preset); lit_basic looks the sky light up along it.

Only the ambient term is occluded: the key light has its shadows. In a forest, the AO darkens the inner crowns and the ground at the trunks; how much it shows depends on how much of a pixel’s light is ambient – little in full sun, most of it in the shade or under an overcast sky.

Screen-space AO sees ao_radius round a pixel, never the canopy ten metres overhead (or above the screen) that keeps most of the sky off a forest floor: on its own it took about 12 % of the sky light under the trees. acron.trees.Forest therefore paints its crowns into a top-down canopy map (cover, and the crowns’ base and top heights) and hands it to the GTAO pass (Environment.canopy_map, the CANOPY and CANOPY_Z frame slots): a point under the crowns loses Forest(canopy_occlusion=0.6) x the cover of its sky light, fading out through the crowns to nothing above them. The floor of a closed wood reads dark green under a lit sky instead of sky-blue, the crowns’ inner leaves darken, and a clearing stays open.

Blue noise¶

stbn.glsl gives any shader the pipeline’s noise:

Function	Returns
`screen_noise(pixel)`	0..1 for this frame and pixel
`screen_noise2(pixel)`	two independent values
`screen_noise_at(pixel, f)`	the same for a caller’s own frame counter
`stbn(pixel, f)`	slice `f` of the masks (pipeline only)

Without the pipeline (no STBN define) the screen_noise functions fall back to interleaved gradient noise, so a shader can use them either way. The masks are 64 x 64 texels and 32 frames, generated by the engine’s Blue Noise module; acron.pipeline.bluenoise makes the texture and keeps the disk cache. The shadow filter’s rotation, the cloud march’s start, the light shafts and the forest’s LOD cross-fade use it.

The night¶

The composite applies purkinje(color, frame_night() * purkinje) before the tonemap: as the night deepens, colour drains towards the blue-grey of rod vision, except where the picture is bright (mesopic vision: a lamp-lit road keeps its colour). purkinje in the presets is its strength (1 by default, 0 off). Without the pipeline the materials do it themselves in frame_display(); a game with its own post pass calls the same function from common.glsl. The exposure the night is shown at follows the night’s light (Look.adaptation, The Sky, Day and Night), and the shoulder tonemap leaves those dark values as they are.

PBR materials¶

Materials shaded by the engine’s PBR shader join the pipeline through acron.pbr: pass the environment to pbr.init(), after the pipeline and the modules that add defines (shadows, clouds, probe), as for any lit shader:

from acron import pbr

env = Environment(base, clock=DayClock(latitude=45.8, longitude=16.0))
pipe = FramePipeline(env, "high", cache_dir=user_cache_dir)
shadows = SunShadows(env, resolution=2048, distance=220)
probe = EnvironmentProbe(env)
pipeline = pbr.init(environment=env)

model = loader.load_model("car.glb")
model.reparent_to(render)
pipe.track(model)             # rigid motion vectors when it moves

pbr.init() then installs the PBR shader generator on the scene and adds no post pass of its own: the camera already draws into the pipeline’s targets, and the composite tonemaps, with the same tonemappers as the engine’s PbrPipeline (pbr_tonemap.glsl). The environment lights the materials (see Materials and PBR).

pbr.init(post_process=False) without the environment draws the plain PBR shader inside the pipeline, lit only by light nodes: it is anti-aliased but writes no normals or motion (both come from the depth) and reads no AO. The sample’s --pbr plain shows it.

The sample¶

python/samples/frame_pipeline_demo.py shows the pipeline on the forest sample with a moving box, and PBR spheres and a crate in front, one of them rolling: P cycles the presets, V the debug views. --pbr plain draws the PBR props without the environment’s light, --pbr off leaves them out. --compare out.png renders the same frame with the off preset and with --quality side by side, with enlarged crops; --measure prints the timings above.

Limits¶

Perspective lenses only; the main camera only (other display regions render as before).
The scene buffer is not multisampled; alpha-to-coverage does nothing in it.
Skinned motion is per node (above). Objects drawn in the background bin with depth writes off, other than the sky, are treated as sky.
The AO the scene reads lags one frame; a camera cut should reset the history.
Changing presets does not recompile materials: a preset that turns GTAO on reaches only shaders compiled while it was on.