PlenOctrees for Real-time Rendering of Neural Radiance Fields


  • Achieves 150+ FPS to render 800x800 images (3000x faster than NeRFs)
  • It pre-tabulates the NeRF into a PlenocTree
  • It removes viewing direction as an input to the neural network, and predicts a spherical harmonic representation of radiandce (??)


  • Lambertian (= diffuse?) vs non Lambertian
  • Neural Volumes

What's wrong with NeRF?

NeRF rendering is slow since it requires dense sampling of the scene, where every sample requires a neural network inference They overcome ths by pre-sampling the NeRF into a tabulated view-dependent volume (PlenOctree)

:::collapse Plenoptic functions what the heck is that? :::

Spherical Harmonics (SH) A standard basis for functions defined on the surface of the sphere. This enables to recover the view-dependent color

How to integrate SH:

  • They train a network that produces coefficients for the SH functions instead of raw RGB values
    • The predicted coefficients are stored in PlenOctree
  • They also introduce a sparsity prior during NeRF training to improve the memory efficiency

How they train NeRF faster: Stop the training of NeRf early and convert it into a PleOctree. Then it can be trained much faster as there's no neural nets.


NeRF-SH: a variant of the NeRF that directly outputs the SH coefficients (no input for view-direction)

Converting NeRF representation to PlenOctree is straightforward: simply evaluate it on a uniform grid.