Gaussian Splats

This page describes part of the Python framework that comes in the acron wheel. The renderer is Python over the public engine API; only its sort is native, and that part is available to C++ as well: see Gaussian Splat Sorter.

A Gaussian-splat scan – the output of a 3D Gaussian Splatting trainer, or a compressed scene from PlayCanvas’s splat-transform – is millions of small oriented, semi-transparent ellipsoids that together reproduce a captured place from any angle. Acron draws them in real time with two modules: acron.sog reads the files, and acron.splats renders them. The sort that orders them for drawing is native code, described in Gaussian Splat Sorter.

The renderer needs an OpenGL 4.1 core profile, for instanced drawing, texel fetches and floating-point render targets. Request it before the window opens; splats.PRC holds the line:

from acron.core import load_prc_file_data
from acron import splats

load_prc_file_data('splats', splats.PRC)

Reading scans

acron.sog.read_splats reads a .sog (a file, or the same files in a directory) or a 3DGS .ply and returns a SogBundle, whichever the source. The bundle keeps the scan in the packed form the GPU reads: five RGBA8 planes (position, rotation, scale, colour and opacity), their codebooks and, with with_sh=True, the higher spherical-harmonic bands that make the colour change with the view.

from acron.sog import read_splats

bundle = read_splats('scans/courtyard.sog')
print(bundle.count, 'splats,', bundle.sh_bands, 'SH bands')
lo, hi = bundle.bounds()          # for placing the camera
preview = bundle.decimate(4)      # every 4th splat

The module is plain numpy. positions(), quaternions(), scales() and colors() decode the bundle on the CPU, for tools and tests; write_sog and write_ply go the other way. A .sog stores its splats in spatial order, so the first thousand are one corner of the scene: decimate() gives an even sample.

Drawing them

Four objects make up the renderer, and a game creates each once:

SplatSource

The bundle’s planes, codebooks and SH palette as textures, uploaded once and decoded on the GPU.

WorkBuffer

One render-to-texture pass that decodes every splat for the current view: its centre, covariance and colour (with the SH bands evaluated for the camera), and its depth for the sort. It runs only on frames whose view changed. sh_bands says how many bands to evaluate: None for all the source has, 0 for none.

SplatNode

The draw: one instanced draw of 128 quads per instance in back-to-front order, culled to the frustum and to a minimum pixel size, with optional compensation for the anti-aliasing dilation (antialias).

SplatSorter

Produces the draw order on a worker thread from the work buffer’s depth plane, with the native sorter when it is there and numpy otherwise.

Each frame the game tells the work buffer and the node about the view, hands finished sorts to the node, and lets the work buffer run:

from direct.showbase.ShowBase import ShowBase
from acron import splats
from acron.sog import read_splats


class Viewer(ShowBase):
    def __init__(self, path):
        super().__init__()
        self.source = splats.SplatSource(read_splats(path))
        self.work = splats.WorkBuffer(self, self.source)
        self.node = splats.SplatNode(self.render, self.work, bin_sort=10)
        self.sorter = splats.SplatSorter(self.source.count)
        self.work.on_updated.append(
            lambda work: self.sorter.submit_plane(work.depth_plane))
        self.task_mgr.add(self.update, 'splats', sort=10)

    def update(self, task):
        w, h = self.win.get_x_size(), self.win.get_y_size()
        self.node.set_screen(w, h,
                             *splats.focal_pixels(self.camLens, w, h))
        self.work.set_camera(self.camera, self.camLens)
        order = self.sorter.poll()
        if order is not None:
            self.node.apply_order(order)
        self.work.before_frame()
        return task.cont

    def destroy(self):
        self.sorter.stop()
        super().destroy()

A real program calls set_screen and set_camera only when the camera or the window changed, as here they cost a work-buffer pass every frame. Stop the sorter before the program ends: it owns a thread.

Options that switch live

The render options change on the same objects, without uploading anything again:

  • work.set_sh_bands(n) – fewer or more view-dependent colour bands;

  • node.set_options(antialias=...) – the anti-aliasing compensation;

  • node.set_min_pixel_size(pixels) – the size below which a splat is culled;

  • node.show(False) – hide the scan.

Linear-light blending

By default the splats blend straight into the window, in the gamma-encoded colours the scan was trained with. LinearSplatBuffer blends them in linear light instead, in a 16-bit floating-point buffer, and composites the result into the window through a tonemap ('none', 'aces' or 'neutral'). A node drawn into it is made with linear=True, and is moved between the two paths with attach and detach:

linear = splats.LinearSplatBuffer(self, tonemap='aces', exposure=1.0)
node = splats.SplatNode(linear.root, work, bin_sort=10, linear=True)

linear.set_tonemap('neutral', 1.2)     # a uniform write
linear.detach(node)                    # back to direct blending
linear.set_active(False)

Call the buffer’s before_frame() each frame, as for the work buffer.

Deforming splats

A SplatModifier is the game’s hook into the work-buffer pass: a piece of GLSL that may move each splat and reshape it – a curved-world view, a fisheye, an explosion. It defines two functions, and may use the helpers splatQuatBasis and splatCovFromAxes:

bulge = splats.SplatModifier(
    name='bulge',
    glsl='''
uniform float bulge;

void modifySplatCenter(inout vec3 center) {
    center *= 1.0 + bulge * length(center);
}

void modifySplatCovariance(vec3 originalCenter, vec4 rotation,
                           vec3 scale, out vec3 covA, out vec3 covB,
                           out float visibility) {
    vec3 r0, r1, r2;
    splatQuatBasis(rotation, r0, r1, r2);
    splatCovFromAxes(r0 * scale.x, r1 * scale.y, r2 * scale.z,
                     covA, covB);
    visibility = 1.0;
}
''',
    uniforms={'bulge': 0.0},
)
work = splats.WorkBuffer(base, source, modifier=bulge)
work.set_uniforms({'bulge': 0.05})

uniforms gives every uniform the chunk declares its first value – each must have one – and set_uniforms changes them later. The modifier runs on the GPU and writes the depth the sorter reads, so the draw order follows whatever it does to a splat; splats.IDENTITY is the modifier that changes nothing, and the default.