A Hitchhiker’s Guide to the 3D Ecosystem

Suvaditya Mukherjee; Merve Noyan; Aritra Roy Gosthipaty; Pedro Cuenca

01 / 3D ECOSYSTEM

A Hitchhiker’s Guide to the
3D Ecosystem

From surfaces to fields, from space to time.
An article you can read, and a lamp you can take apart.

Suvaditya Mukherjee · Merve Noyan · Aritra Roy Gosthipaty · Pedro CuencaUpdated September 10, 2026Cite this article

Articulated desk lamp construction — Choose a representation, then drag to orbit. Select Pan or hold Shift to move the view; use + and − to zoom. Arrow keys rotate, Shift + arrows pan. These geometric previews introduce the detailed visualizations below.

Data in 3D is confusing

3D data is sparse and valuable for many different tasks and applications. But the way it is represented and stored can vary significantly. It is also generally non-trivial to change from one format to another. Just like the idea of pixels in images and videos, 3D data can even have multiple data formats with the same representation (for example, JPEGs and PNGs → data format, pixels → representation).

This can get really confusing really quickly, especially when you begin factoring in the idea that, while 2D data is generally simple to represent mathematically, 3D data must also consider physical effects like camera positioning, lighting, scene dynamics, and other such properties as a fundamental piece of the equation.

Let’s take a closer look at some of these formats, and understand why there’s no real consensus (yet!), why you want to choose one over the other, and when.

Why not an unified format?

A proposal to unify fourteen competing standards results in fifteen competing standards. — Standards, xkcd #927 by Randall Munroe. CC BY-NC 2.5.

Every format started out as the defining standard for 3D, until it was replaced by another one. Different industries need 3D data, but they demand different guarantees and properties from it. For example, manufacturing companies like their data to have very low tolerance for errors on dimensions and surfaces, but game developers can trade those off for speed and ease of rendering. Robotics cares about strong physics and volumetric properties, mapping (like LIDAR Scans) can trade. Encoding all of these properties into a single format that was concise enough, fast enough, but informative enough is still a challenge.

Even in the case of 4D data (where you take multiple 3D data points over time), the same problems are exacerbated by compression, while storage becomes difficult due to the large volume of data we’re dealing with.

But, maybe this is a good thing. You do not use a katana to cut fruits, nor do you use a butter knife to cut watermelons. Every task having its own format is useful because that ensures optimal performance for it, taking advantage of the tradeoffs we make for a specific format. Keeping this in mind, let’s take a look at what are some of the most popular formats and representations out there, how they work, what they look like (standardized 3D data visualization is also a major problem!), and what are some works that build upon them.

Types of 3D Formats
Type	Data Formats	Data Representation Style	Specialty	What is it used for?
Mesh	GLB, PLY, OBJ, STL	Polygons	Highly-scalable, malleable	Games, Simulation
Point Clouds	PLY, PCD	Points	Easy to store	LIDAR Scans
Voxels	OpenVDB, NPY/NPZ (numpy)	Unit Blocks	Easy to interpret	Medical Imaging
NeRFs	No standard format	Learned Neural Network	Novel view synthesis	Robotics, VFX
3D Gaussian Splats	PLY, SPLAT	3D Gaussians	Captures accurate scene lighting	VFX, Digital Twins

Mesh

If you see 3D objects on the internet, it’s likely a mesh. A mesh is a collection of polygons or triangles put together to form a shape. Generally, these polygons are defined by a standard set of vertices, edges, and faces. They are represented as coordinates in a standard 3D space (like [0, 2, 1]), followed by specifying relationships between them to form connected edges, which implicitly form faces. There are several different ways to store these details (data formats!) which optimize for different things like access speed, complexity of fetching random faces/points, and so on.

Static construction view of the articulated lamp: From vertices to a surface — Play the assembly or scrub its slider, then reveal wireframe and normals. The construction video below can be played independently. The same indexed vertices define every view of the lamp.

We can also tag on additional properties to these vertices (that end up making objects), such as:

information about surface normals (an unit vector defined in 3D space that will help us calculate any reflections or shadows we might want to render),
materials information (that can help us in modelling different physical properties like rigidness, flexibility, color, etc.),
UV coordinates (a 1:1 mapping that can help us accurately place textures or designs onto the object), and more.

One surface, several attributes

Triangle on the shade 4

Inspect the lamp’s shade. UV coordinates unwrap the curved surface into a rectangle; normals describe its orientation.

Mesh is one of the most popular choices today due to its versatility and support by modern software frameworks.

Some popular data formats that store meshes include:

OBJ: Simple and compute-friendly, stores basic geometry and materials information.
STL: Especially popular among 3D-printing tools for low-error tolerance and ensuring manifold. Does not store color, textures, or materials information.
PLY: Flexible information on vertex and face attributes, along with color information. Used to store data directly captured from a spatial camera or 3D scanner.
GLB/gLTF: Capable of storing geometry, along with arbitrary attributes, graphics, textures, and even basic animations! Also stored either as a binary or as a human-readable JSON.
FBX: Mainly used in game engines, popular for storing rigging data (information that can define relationships between vertices, allowing us to animate and run physics simulations on those 3D objects), texture, and heavy animation information.
Alembic: Used in scientific computing/simulation or in VFX for films, mainly used to encapsulate long-duration animation along with arbitrary properties for all objects.

Within Generative AI, meshes are popular as the stored meshes can easily be converted to tensors. They are also a great representation for classification, segmentation, and object detection since the discrete points make it easy to treat them as an ordered collection of samples with a hierarchical relationship.

Point Clouds

This format is well known for literally being clouds of… points. The idea is to place unit-sized points in 3D space in an arrangement that resembles the object we are modeling. Density of points within a certain region will determine properties (imagine these points as atoms in the physical world deciding hardness, rigidity, and more). These points are stored as an unordered set, often with point-based properties attached (such as normals, colors, and more) in a PLY data format.

Static construction view of the articulated lamp: A shape, one observation at a time — A depth sensor measures distance along each viewing ray, so a surface in front hides one behind it. Enable “Depth sensor only” to retain the nearest sample in each simulated sensor pixel. Sensor azimuth is the sensor’s horizontal angle around the lamp, in degrees. The cyan camera and top-view locator mark its position. Change it to reveal another side; orbiting this view moves your inspection camera independently.

Devices like LiDAR, Photogrammetry sensors, and Depth cameras use PLY as their data representation of choice due to simplicity, and the ease with which it can handle streaming data. Some key applications of point clouds are in 3D segmentation, reconstruction, and famously, autonomous driving. Fun fact: If you’ve been in a Waymo, the displays show a point-cloud visualization as part of its LiDAR sensor suite.

Some popular data formats that store point clouds include:

PLY: Seen earlier, it will store unit vertices as points, attach attributes to them, and store point clouds just like it would store meshes.
LAS/LAZ: Mainly popular in geospatial/aeronautics LiDAR, it’s optimized for storing GPS coordinates, classification attributes, LiDAR return signal intensity, among other things suited to those fields.
PCD: Literally called Point Cloud Data, meant for storing point clouds in an ordered fashion, allowing for complex relationships to be captured (like establishing hierarchies to control properties for many points at once). It’s also able to exploit the use of mmap for faster loading, support all primitive data types, and capture n-D histograms over different axes, useful for 3D perception tasks.

But, of course, there are limitations. There is no way to define connectivity between points due to their unordered nature. There are also issues arising from varying point counts (if my renderer can support only 300k points while my file has 500k, which 200k should be dropped?), and occlusion in high-density point clouds.

Voxels

Voxels are probably the easiest convention within 3D to understand, because they are analogous to pixels. What pixels are in 2D space, voxels are in 3D space. A unit-sized 3-axis cube that is then mapped in a 3D grid to create a field/space. Occupancy of voxels in a given space defines what the intended object should look like. Voxels can either be present within a given space or not, also known as binary occupancy. We can then attach properties to these voxels to make richer representations. This has many benefits, especially since a lot of previously-developed software for 2D can be directly extended to 3D with voxels (ideas such as convolution kernels, pooling, and so on). It is also much simpler to visualize and perform traditional tasks on. Voxels are generally used within AI as a format to represent medical imaging datasets, perform occupancy prediction, classification, and detection.

Static construction view of the articulated lamp: Look inside the grid — These occupied surface cells are derived from the lamp mesh. A horizontal cut reveals the grid; thin arms disappear at coarse resolutions.

Voxels seem like the best standard, but it’s hard to scale them due to their sparsity. Let’s do some quick math. If a pixel is 8-bit RGB, it requires 8 x 3 = 24 bits, which is equivalent to 3 bytes. Now, in a 1024x1024 image, there would be a requirement for 3 x $2^{10}$ x $2^{10}$ bytes. Let’s extend this to a voxel. If it was a 1024x1024x1024 grid, you would now have 1024 times more data to store. Images find it trivial to scale to larger sizes, where increasing dimensions by 2x would mean the file size increases by 4x (2x width and 2x height). But for voxel-based grids, the trade-off becomes worse at 8x instead (because now you have 2x width, 2x length, and 2x height). This can make any valuable object with decent fidelity very difficult to store. For context, to store the 1024-size grid we saw earlier with RGB color information, we would need 3 x $2^{10}$ x $2^{10}$ x $2^{10}$ = 3 GiB of storage whereas a similar pixel-based image would be 3 x $2^{10}$ x $2^{10}$ = 3 MiB of storage. Scaling dimensions will dramatically increase the sizes of files.

But there are some ways to overcome it. Storage formats such as sparse tensors, hierarchical grids, and octrees are ways to represent the object of interest. (Minecraft being a very popular example, of course).

The most popular format for storing voxels is OpenVDB, which makes use of a sparse hierarchical format meant to be used for volumetric data. It can also handle arbitrary attributes, which makes it useful for use in several fields at the same time.

The extra dimension gets expensive

Side length 1024ChannelsBits per channel

🤗 = 1 byte. Different display sizes.

2D image 3.00 MiB

3,145,728 bytes

3,145,728 balls

3D volume 3.00 GiB

3,221,225,472 bytes

3,221,225,472 balls

1,024× more bytes in the volume.

Storage math

2D image3.00 MiB1024² × 3 × 8/8

3D volume3.00 GiB1024³ × 3 × 8/8

3 MiB = 3,145,728 bytes; 3 GiB = 3,221,225,472 bytes.

Totals are exact; the boxes show a sample of individual byte balls, drawn larger for the image and smaller for the volume. Ball sizes and visible counts are illustrative, not a shared area scale. Each ball denotes one byte, never a KiB, MiB or GiB. Falling and bouncing are procedural; billions of particles are not simulated individually.

Spend memory where something exists

Subdivision depth 4

Compare the same two-dimensional occupancy mask at the same resolution. The dense grid allocates every finest cell. The quadtree stops subdividing empty regions, while occupied regions retain detail. Counts describe leaf cells, not bytes; a real hierarchy also stores internal nodes and metadata. An octree applies the same idea in 3D.

NeRF

NeRF (Neural Radiance Fields) is a technique for reconstructing novel views from a limited set of 2D images of an object taken from multiple poses. The gist of the idea is to learn an MLP that can map 3D position and viewing direction (5 inputs - x, y, z, $\theta$ , $\phi$ ) and predict the density of the object and the color of the object from the given view. It’s important to emphasize that unlike the other representations, this one is the first we’re seeing that must be learnt over an optimization process. The earlier ones can simply be stored as they are, but NeRFs require training.

To actually render an image for the object from a given view, NeRF simply runs inference on the trained model. It does the following:

Position a camera at the specific position and viewing direction as needed, in the 3D space (x, y, z, $\theta$ , $\phi$ )
Shoot a ray from the camera for each pixel
For each pixel, sample 3D points as you keep going in a straight line along the ray until you find dense-enough regions as predicted by the MLP
Combine all the volumes and colors found with alpha compositing

Static construction view of the articulated lamp: Follow a ray to a pixel — This analytic lamp field demonstrates volume rendering, not a trained neural network. Drag the camera and vary the sampling budget; inspect the ray toward the bulb’s contributions below.

The goal is to understand how light behaves and reflects off surfaces, and to use that information to represent the objects. It’s hard to package, so you will require several parts (like the rendering code, the inference code for the specific MLP trained, coordinate conventions, camera information - which is basically the intrinsics of the camera used to originally train the NeRF model) in different files to actually render views with the NeRF.

But if they’re so clunky, why use them? Because they also enable novel view synthesis. Views that may not have been captured during training can also be rendered effectively without any additional information, relying solely on extrapolation by the NeRF MLP. There have also been optimizations on top of the original version, leading to higher-quality rendering, faster speeds, and more compact results.

Unlike the other representations we’ve seen so far that keep geometry, NeRFs don’t actually have any information on geometry. It is only able to render views for the object. This makes it incompatible for use on problems that rely on physics or require relighting (using surface normal information to change how light looks off an object). They also suffer from slow rendering times and are challenging to edit.

3D Gaussian Splats

This is all the rage right now, and chances are you’ve already seen some fancy demos on the internet for it. 3D Gaussian Splats represent scenes with Gaussians that differ from each other and are non-uniform.

Static construction view of the articulated lamp: A cloud of soft ellipsoids — Surface-aligned anisotropic Gaussians are projected through the camera’s Jacobian and blended back to front. Change covariance and opacity, or outline projected footprints.

The idea is to create an initial basic 3D Point Cloud, done using Structure-from-Motion algorithms. We then place 3D Gaussians with unit mean and variance at each of those points. Each of them encodes a certain position, covariance, orientation, and opacity/color in the scene. We then run optimization steps which compare different views from various camera poses and reduce the difference between the original frame and the Gaussians. Converting these 3D Gaussians to a 2D representation is simple, because we flatten/“splat” the Gaussians against the camera view-direction and blend them.

This representation is especially useful for cheap and quick reconstruction, and gives a very high-quality rendering with real-time rasterization. It is also able to capture lighting nuances better than any other 3D representation, but is only able to capture them during optimization and cannot change/re-light the scene.

Popular formats for storing Gaussian Splats include the following:

PLY: Flexible, can store the Gaussian position, the mean and covariance, and other properties. Not optimized well for storing Gsplat-specific properties, doesn’t follow naming conventions.
.splat: Binary data format, stores point information, scales, covariances, and color information. Still nascent, only has limited support.

To dive deep into this topic, consider reading this blog.

What about 4D?

4D is 3D evolving over time. That’s it. Time is our fourth dimension. But it brings its own challenges. How do you store geometry efficiently? Is motion learnt or baked in? How do cameras work over time? How much does time affect size? These are all open problems which have similar tradeoffs as seen in 3D representations. Let’s take a quick look on how 3D representations have been adapted for storing 4D information.

Animated/Deformed Mesh

This representation is analogous to how modern MP4 compression works with pixel frames, where we calculate movement and differences between keyframes and store changes instead of whole keyframes. The goal here is to learn some object/skeleton dynamics (a shape and how the object’s joints and parts move with respect to each other), and use that information to calculate how that object would move given changes.

Static construction view of the articulated lamp: Move the joints, keep the topology — Every pose uses the same triangle indices. Time drives a scripted joint motion; the elbow offset is an independent control. Reveal the wireframe to follow connectivity.

This is very powerful, because it also allows us to bake in unique animations cheaply without increasing file size dramatically. But this can only be done with scenes where the topology/objects stay fixed. If objects undergo material changes wherein they undergo deformations, then we’d have to store per-deformation changes which can get expensive if done frequently. This format also comes with its own struggle of performing the arduous process of rigging (assigning weights and relationships between arbitrary faces/parts of objects in the scene), along with limitations around processing fluids, clothing, or rapid topological changes.

Within Generative AI, this format is heavily used for Motion Capture, for Pose Estimation, and for making Human Avatars. MoCap studios prefer using this due to ease of importing this into game engines and similar software with strong heuristics to minimize rigging.

4D NeRF

4D NeRFs are an interesting case-study, wherein the MLP now learns time-based densities and colors. Unlike deformed meshes, though, we must learn all our information into a single MLP. It’s also more challenging, because the model must be made robust against distinguishing between motion induced by the camera itself, motion of the object, or changes in appearance. This naturally creates a need for a bigger data scale for an already-data-hungry representation.

Static construction view of the articulated lamp: Time is a separate input — Keep the camera fixed and scrub time, or pause time and move the camera. The field changes with the lamp’s articulated pose. No learned dynamic NeRF is used.

If done correctly, it unlocks novel-view synthesis over time and volumetric objects which can be useful for games, animation, media, and entertainment.

4D Gaussian Splats

4DGS can be achieved in many different ways, which include storing separate Gaussians per frame, only recording deformations for existing Gaussians over time, or choosing certain key Gaussian configurations and then recording deformations locally over time against each such configuration.

Static construction view of the articulated lamp: Three ways to store motion — Compare eight discrete frame configurations, continuous joint deformation, and linear interpolation between four key configurations. Primitive identity stays fixed in this teaching example.

This representation is especially useful because it can encode lighting information over time for a fixed scene, record trajectories, and also allow for storing long sequences if they are stable. But it also doesn’t fit the bill for cases where objects come and go into the scene, or if the motion of objects is too fast to be captured well by the representation.

OpenUSD, a Special Mention

OpenUSD was developed originally by Pixar as a scene-description and composition system, which encapsulates other representations into a single scene. It lays down strong guidelines and standards for how assets should be referenced and how they should interact with each other. As of now, this is a format which is helping converge all the data representations we discussed under one umbrella. It is also useful for defining relationships between heterogeneous objects in scenes, something that has been difficult to do until now. It’s also important as a dataset format, because it allows for consistent scene structures/hierarchies, annotations, and is very compliant with formats that synthetic data generation tools like. The ability to bring together different objects is also useful for robotics and simulation practitioners too.

Looking forward

That’s it! That was a largely all-encompassing view of the confusing yet interesting world of 3D representations. To quickly recap, we looked at:

Some of the most popular 3D representations out there
How to work with them
What their data formats look like, and
A sneak-peek at how 4D representations are being shaped by their predecessors.

While we discussed a lot about each of these representations, we did not talk about how you can convert between them (spoiler: it’s not trivial!). There’s also much ground to cover on how we can compress these representations further, whether we can help bring real-time delivery to browser/phones, usable and physics-aware topology, strong provenance, and lastly around dynamic worlds generated on-the-fly (a.k.a. World Models 👀). It’s also important to peek into what the state-of-the-art is within 3D AI, and how we can leverage fine-tuning to create our own customized model generators (altering reality, quite literally!). You can take a look at some of the best 3D models out there over in this collection.

Thank you for following along! See you in the next one 👋

Comments

Please share all comments, questions, and feedback on the Space’s discussions page.

Cite this article

Suvaditya Mukherjee, Merve Noyan, Aritra Roy Gosthipaty, and Pedro Cuenca. (2026). A Hitchhiker’s Guide to the 3D Ecosystem. Hugging Face Spaces.

@misc{mukherjee2026hitchhiker,
  author = {Mukherjee, Suvaditya and Noyan, Merve and Gosthipaty, Aritra Roy and Cuenca, Pedro},
  title = {A Hitchhiker's Guide to the 3D Ecosystem},
  year = {2026},
  url = {https://huggingface.co/spaces/suvadityamuk/3d-representations-guide},
  note = {Interactive article}
}

Download BibTeX

Source and updates

Chapters, figure defaults, shared geometry and animations live in the Space repository. Improvements are tracked as reviewed changes, so the article and its history stay together.

Motion

Article by Suvaditya Mukherjee, Merve Noyan, Aritra Roy Gosthipaty, Pedro Cuenca. Article layout adapted from Thomas Frere’s research-article template (CC BY 4.0); navigation and styling modified. Inspired by AdithyaSK’s RL environments guide.

Interactive views use PlayCanvas, Canvas 2D and WebGL. Animations use Manim. Original prose retains its existing rights; the template license applies to template-derived material only.