What If Images Didn't Store Pixels At All?
A deep dive into implicit neural representations and the shift from data to functions
Introduction
Every photo, video, or 3D object we use digitally is ultimately stored as raw data. A photograph is usually saved as millions of tiny pixels, while a 3D model is stored as thousands or even millions of geometric points and surfaces. This approach has worked for decades, but it comes with a growing problem: as quality increases, so does the amount of data we need to store, process, and transmit.
Think about zooming into an image online. Eventually, the image becomes blurry or pixelated because the computer only has a fixed number of pixels saved. The same issue exists in 3D graphics. Highly detailed 3D worlds in games, simulations, or virtual reality require enormous amounts of memory because every detail has to be explicitly stored somewhere.
Implicit Neural Representations (INRs) approach this problem differently. Instead of storing every individual pixel or every point in a 3D object, an INR learns the pattern behind the data itself. Rather than memorizing a picture, the model learns a mathematical function that can recreate the picture from any coordinate you ask for.
A simple way to think about it is the difference between storing a drawing versus storing the instructions to recreate the drawing perfectly at any size. Traditional methods store the final result directly. INRs store the underlying rules that generate the result.
This becomes powerful because the representation is no longer locked to a fixed resolution. An INR can generate smooth details at different scales, fill in missing information naturally, and represent highly detailed structures more compactly than traditional methods. These properties are why INRs have become increasingly important in areas such as image enhancement, AI-generated graphics, and realistic 3D scene reconstruction.
What Are INRs?
Implicit Neural Representations (INR) aim to learn a continuous function that maps an input coordinate x to a value a: a(x)
while satisfying:
At a high level, this means the network learns how to generate the signal directly from coordinates. For an image, the input may be a pixel coordinate (x,y) and the output may be an RGB value.
For a 3D object, the input may be (x,y,z), while the output describes geometry or density.
The function is typically modeled using a multilayer perceptron (MLP), where each layer is defined as:
and the final output is:
Traditional activation functions such as ReLU work well for many tasks, but standard MLPs often struggle to represent highly detailed signals because they naturally prioritize smooth, low-frequency patterns before sharp details. Techniques such as positional encoding help address this limitation by transforming the inputs into richer representations that allow the network to better capture high-frequency details and fine spatial variations.
Core Properties of INRs
Continuity: INRs represent data as smooth continuous functions instead of discrete samples.
Differentiability: INR functions are fully differentiable, making them easy to optimize using gradient descent.
Resolution Independence: INRs can generate outputs at any resolution because they are not tied to a fixed grid.
Compactness: Information is stored within neural network weights rather than massive arrays of pixels or voxels.
Frequency Adaptivity: With techniques like positional encoding, INRs can capture both smooth structures and fine details.
Smoothness: INR outputs are naturally smooth due to their continuous functional form.
Scalability: INRs scale based on function complexity rather than raw output dimensionality.
How are INRs Used
Image Super-Resolution
One everyday example of INRs can be seen in image upscaling. Normally, when a low-resolution image is enlarged, it becomes blurry because the computer is simply stretching existing pixels. INR-based systems instead learn the structure and patterns within the image itself.
For example, imagine taking an old blurry photograph and enlarging it onto a large monitor. Traditional methods might create blocky edges and smudged details, while an INR can generate smoother lines, sharper textures, and more natural-looking details because it treats the image as a continuous function rather than a fixed grid of pixels.
This makes INRs useful for restoring old photos, enhancing video quality, and improving image resolution in medical or satellite imaging.
Neural Rendering and 3D Scenes
INRs are also transforming how computers represent 3D environments.
Traditionally, creating realistic 3D scenes requires storing huge amounts of geometry, lighting, and texture information. INR-based approaches instead learn a continuous representation of the scene itself.
Imagine taking a few pictures of a room from different angles and then allowing an AI system to generate entirely new viewpoints as if a virtual camera were moving through the room. Instead of explicitly modeling every wall, chair, and light source, the INR learns the structure of the space and can recreate it continuously from different perspectives.
This technology is already influencing virtual reality, gaming, film production, robotics, and digital twins of real-world environments.
Because the representation is continuous, scenes can appear smoother, more realistic, and more flexible than traditional 3D pipelines.
Mixture of Experts with INR
While INRs are powerful, they still face limitations when representing highly complex scenes or very large datasets. Training can become expensive, and a single network may struggle to efficiently model every region of a signal equally well.
One promising direction is the use of Mixture of Experts (MoE) architectures for INRs.
Instead of relying on one neural network to represent an entire signal, MoE INRs divide the problem into specialized regions handled by multiple expert networks. A manager network determines which expert is responsible for different parts of the input space.
Traditional INRs approximate a signal using one continuous function. MoE INRs instead approximate signals using multiple specialized functions. This allows different experts to focus on different structures, textures, or spatial regions.
The advantages of MoE INRs include improved scalability, better parallelization, more efficient parameter usage, and stronger reconstruction quality.
Researchers have demonstrated strong performance across applications such as image reconstruction, audio synthesis, and 3D surface reconstruction.
Mixture of Experts architectures represent an important step toward making implicit neural representations more flexible, efficient, and practical for real-world systems.