Explaining the Implicit Neural Canvas: Connecting Pixels to Neurons by Tracing their Contributions

Abstract

The many variations of Implicit Neural Representations (INRs), where a neural network is trained as a continuous representation of a signal, have tremendous practical utility for downstream tasks including novel view synthesis, video compression, and image superresolution. Unfortunately, the inner workings of these networks are seriously under-studied. Our work, eXplaining the Implicit Neural Canvas (XINC), is a unified framework for explaining properties of INRs by examining the strength of each neuron’s contribution to each output pixel. We call the aggregate of these contribution maps the Implicit Neural Canvas and we use this concept to demonstrate that the INRs which we study learn to “see” the frames they represent in surprising ways. For example, INRs tend to have highly distributed representations. While lacking high-level object semantics, they have a significant bias for color and edges, and are almost entirely space-agnostic. We arrive at our conclusions by examining how objects are represented across time in video INRs, using clustering to visualize similar neurons across layers and architectures, and show that this is dominated by motion. These insights demonstrate the general usefulness of our analysis framework. Our project page is available here.

1 Introduction

Figure 1: How do INRs “see” the images they represent? We propose XINC, which we use to show what parts of a learned visual signal are important to each neuron of an INR. Here, we take a sample from the neural canvas, with contribution maps for 5 neurons sampled from the last layer of a NeRV trained on a Cityscapes [11] video. Some neurons attend to colors and textures, while others focus on low-level features like edges.

Leveraging neural networks to represent data, where the network computes feature maps and embeddings for various visual inputs, is central to computer vision. Recently, implicit neural representations (INRs) have emerged as an exciting, radically different approach where a multi-layer perceptron (MLP) or convolutional neural network (CNN) is overfit on an image as a generative network, becoming a continuous approximation of the discrete visual data [40, 35, 44, 45, 36]. The most well-known type of INRs are Neural Radiance Fields, where a network learns to represent a scene, allowing for continuous scene representation for novel view synthesis, editing, etc. [31, 38, 52, 50, 16]. However, INRs are useful for many other tasks as well, in particular, visual data compression [29, 7, 8, 9, 54, 18, 27, 13, 14, 46, 5, 53].

In spite of their immense utility, the inner workings of INRs remain relatively under-explored. While some studies shed light on MLP-based INRs [51], NeRF [52], and the impact of hypernetworks [10, 51], the hidden details of non-coordinate methods such as NeRV [7] are even more opaque. This is not completely unique to INRs – explainability and interpretability are still massive challenges for other areas in vision as well [17, 2, 1, 34]. Nevertheless, there is a significant gap for understanding INRs, considering that popular analysis techniques such as NetDissect [3] and GradCAM [42] do not transfer to INRs in straightforward ways. Even analysis work for generative networks tends to focus on control and manipulation [21], and the coordinate-to-signal paradigm of INR is distinct enough from the generative noise-to-signal paradigm that most analysis work cannot be trivially extended [33].

While understanding INRs would obviously be beneficial for improving their representation, analyzing with the intent to explain is valuable on its own. As INRs trend towards commercial viability for tasks like compression, explaining their behaviors becomes critical for real-world enterprises to consider them as alternatives to standard codecs. Traditional compression methods are not perfect, but their failure modes are known and easily explainable. Deep INR methods, by contrast, have failure modes that are somewhat unknown and quite opaque. Thus, beyond hel** researchers to improve the models’ performance, explaining INRs will help with their actual adoption in the real world.

Our work proposes eXplaining the Implicit Neural Canvas (XINC) as a novel framework for understanding INRs. Specifically, we propose a technique for dissecting INRs to create contribution maps for each “neuron” (group of weights), which connect neurons and individual pixels. Together, these contribution maps comprise the “implicit neural canvas” for a given visual signal. XINC can thus be used to help us understand the relationships between INRs and the images and videos they represent.

We analyze MLP-based and CNN-based INRs. While some other analysis works help explain MLP-based INRs [51], our analysis of convolution-based INRs is first of its kind. To demonstrate the utility of XINC we use it to derive novel insights by characterizing INRs in 5 key ways.

Neuron Contributions Correlate with Colors and Edges We show representative examples of contribution maps for MLPs and CNNs, and show how they attend to objects in terms of low-level features like edges and colors, while lacking coordinate proximity-based correlations.

INRs Must Represent to Omit We use sums of contribution maps to show that the representations learned by INRs, while intensity-biased, are not dictated by raw pixel magnitude alone. INRs must learn representations even for missing objects, where the final output colors are near zero.

Representation is Distributed We examine how many pixels each neuron represents, and to what extent, as well as conversely how many neurons are involved in the representation of each pixel. We demonstrate that these representations are quite distributed in nature.

Motion Drives Contribution Change Over Time We show how NeRV’s handle categories of objects and objects in motion, demonstrating that contributions to a given object remain relatively constant over time. This is true even for objects in motion, and fluctuations in our neural canvas across time for videos prove that motion drives changes in which pixels each neuron contributes to.

We Can Group Similar Neurons We cluster neurons from various INRs to show how we can group neurons with similar representation properties using contribution maps.

2 Related Work

Implicit Neural Representation is a way to represent and compactly encode a variety of high resolution signals. The key idea is to map a set of coordinates to a specific signal, by employing a neural network as the approximating function [44, 31, 7, 49, 41]. SIREN [44] utilizes periodic activation functions in MLPs to fit and map complicated signals, including images, 3D shapes and videos using very small networks. COIN [13], the first image-specific INR method, uses implicit neural representation for image compression tasks and COIN++ [14] extends this work to encode multiple images. In the challenging realm of videos, NeRV [7] is the first method to scale video compression using image-wise implicit representation. It uses convolution layers in addition to MLPs and outputs all RGB values of a frame given the positional embedding of frame index $t$ as input. HNeRV [9] treats the frame itself as the index embedding by introducing a small encoder, thus improving performance over index-based methods. NIRVANA [29] harnesses spatio-temporal redundancies in videos by treating them as groups of frames and fitting distinct networks to each group, enabling it to encode videos of varying lengths and resolutions. Meta-learning approaches try to address long per-item fitting times by learning a hypernetwork for predicting network weights [10, 23]. [28] further expand on the utility of INR by proposing a method for encoding weights as vectors for a wide variety of downstream tasks.

Figure 2: (left) We dissect MLP-based INRs by aggregating their activations (weights multiplied by previous layer outputs) for each pixel at each neuron. (right) We extend this core idea of pixel-to-neuron map** for the CNN-based INR, NeRV, by computing intermediate feature maps that are not yet summed on the input dimension. For layers prior to the head, we also account for the PixelShuffle and apply a filter to account for the subsequent layer’s kernels, which propagate that neuron’s contribution to neighboring pixels. We provide more details in the appendix.

Interpreting and Visualizing Deep Neural Networks is a longstanding field of study in machine learning. A large number of techniques aim to understand the internal representations of convolutional neural networks and deep networks in general. [20] analyzes effects of various object characteristics on detection performance. Grad-CAM [42] aims to provide visual explanations for the decisions of CNNs. Using the gradients flowing into the final convolutional layer from a target concept, Grad-CAM and related subsequent works [12, 6, 15] generate a coarse localization map that highlights crucial image regions contributing to the prediction of the concept, and [22] proposes a method for automatically captioning these regions to describe image features. Network Dissection [3] considers each unit in a network as a concept detector and compares their activity with human-interpretable pattern-matching tasks such as the detection of object classes. GAN Dissection [4] extends these ideas to visualize and understand GANs at the unit, object, and scene-level. Other works characterize neuron behavior with natural language [19], by combining NetDissect-style concepts [32, 48, 25], or in terms of how they behave in conjunction with other closely-related neurons [37]. [26] creates a comprehensive test methodology that systematically perturbs model inputs and monitors the effect on model predictions, hel** identify potential instabilities. These works are insightful for their target domains, but do not directly apply to INR. So, our work proposes to analyze the relationships between various parameters in MLP- and CNN-based INR networks and the spatio-temporal locations of the reconstructed output.

3 Implicit Neural Canvases

In this section, we explain how we compute the contribution maps that comprise the implicit neural canvas. These contribution maps connect neurons to pixels in terms of their activations ($\text{weights}\times\text{inputs}$) that contribute to each pixel, for INRs. We first explain how we do this computation for MLP-based INRs in Section 3.1. We then explain how we extend this for CNN-based INRs in Section 3.2.

3.1 Dissecting Multi-Layer Perceptrons

In this study, as a representative method for MLP-based INRs, we use a simple Fourier Feature Network (FFN) [47] with a Random Fourier Matrix [39] as the positional encoding, followed by linear layers. We represent the MLP as $\theta$ and the $n$ linear layers as $\{l_{1},l_{2},\dots,l_{n}\}$. At each MLP layer, the input pixels are operated upon independently, without any spatial rearrangement or sharing of information between the pixels. Thus, we can directly obtain a map** from the neurons in a layer to the $h\times w$ spatial locations (pixels) in the output image.

For each neuron, $\eta$, we compute a map** of how its output contributes to all neurons in the next layer (or the output channels, for the last layer). Consider a layer, $l$, with a weight matrix $W$ of shape $m\times n$. We use $i$ and $j$ to denote the weight connecting $i$-th neuron in the current layer to $j$-th neuron in the next layer, which are also the indices of the current neuron of interest, $\eta_{i}$, and some neuron in the next layer, $\eta_{j}$, respectively. When we map a neuron, $\eta_{i}$, to the MLP INR’s output, we are thus forming a map** from the weights $\{w_{i1},w_{i2},\dots,w_{in}\}$ to the output. Now consider the image the INR represents, $I$, of shape $h\times w\times 3$. Let us consider that the INR takes pixel location ${x_{k},y_{k}}$ as input, and neuron $\eta_{i}$ produces an output scalar, $o$. We compute its contribution to pixel ${x_{k},y_{k}}$ as $\sum_{j=0}^{n}o\cdot w_{ij}$. When we compute this for all pixels, we obtain a map of shape $h\times w$ for $\eta_{i}$, which we refer to as the contribution map for $\eta_{i}$.

3.2 Dissecting Convolutional Neural Networks

For CNNs, we consider kernels as neurons, and we use the NeRV architecture [7], which has two types of layers. Except for the last, all layers consist of a learned convolutional layer, followed by a PixelShuffle [43] and a nonlinear activation function. The last layer consists of a convolutional layer and an optional nonlinear activation function. For this network, the construction of the contribution map** is less straightforward compared to MLP, since convolutions operate over groups of pixels at each stage, and PixelShuffles rearrange contributions. We first explain how to address the head layer, and then how to extend this for earlier layers.

3.2.1 Head Layer

We begin by analyzing the contribution from kernels in the convolutional head layer to the output RGB image pixels. Consider the head layer, $l_{h}$, with input $v_{\text{in}}$ of shape $h\times w\times ch_{\text{in}}$. We thus have a set of $ch_{\text{in}}*ch_{\text{out}}$ convolutional kernels, meaning $ch_{\text{in}}$ kernels per each R, G, B channel of the output, and thus, $ch_{\text{in}}*3$ kernels that can potentially contribute to each output pixel. We obtain the individual contribution of each kernel by passing each kernel in $l_{h}$ over $v_{\text{in}}$ and storing the outputs separately, yielding a set of features maps, $h\times w\times(ch_{\text{in}}*ch_{\text{out}})$. To get the true output contribution, and thus the final contribution maps for all $ch_{\text{in}}*3$ neurons, we pass these feature maps through the $l_{h}$ activation (e.g. tanh).

3.2.2 PixelShuffle

All NeRV layers except the head layer perform some upsampling using PixelShuffle [43]. PixelShuffle upsamples feature maps by moving elements from the channel dimension into the spatial dimension. Let the number of input channels of the convolution preceding the PixelShuffle in layer $i$ be $ch_{i,\text{in}}$ and the number of output channels be $ch_{i,\text{out}}$. For upsampling factor $r$, the number of output channels from the PixelShuffle, which is the same as the number of input channels to the $(i+1)^{th}$ layer, is computed as

$$ch_{i+1,\text{in}}=\frac{ch_{i,\text{out}}}{r^{2}}$$ (1)

We require a contribution map of shape $h\times w\times(ch_{i,\text{in}}*ch_{i,\text{out}})$. To obtain this, we first follow the same approach to the head layer and obtain a map** of shape $\frac{h}{r}\times\frac{w}{r}\times ch_{i,\text{in}}\times ch_{i,\text{out}}$. Considering the penultimate NeRV layer for the sake of simplicity, we can apply PixelShuffle to this downsampled contribution map and obtain the map of size $h\times w\times ch_{i,\text{in}}\times ch_{(i+1),\text{in}}$. However, the contributions of distinct output kernels $ch_{i,\text{out}}$ are now no longer separated since they are moved into the spatial dimension.

To resolve this, we repeat the operation $r^{2}$ times, yielding a new map of size $r^{2}\times h\times w\times ch_{i,\text{in}}\times ch_{(i+1),\text{in}}$. This is equivalent to a map** of $h\times w\times ch_{i,\text{in}}\times ch_{i,\text{out}}$ by moving the $r^{2}$ dimension to the end and substituting according to Equation 1. While this map has the dimensions we require for further processing by downstream layers, we must take care that each non-overlap** $r\times r$ block contains samples from the same filter in order to preserve each filter’s contribution. If nearby samples were from different filters, as in the traditional PixelShuffle, then the contribution of different filters would be mixed and aggregated in future layers. We can conveniently accomplish this with a simple re-arranging of the repeated channels followed by masking such that each $r\times r$ block contains only one non-zero element (to prevent overcounting the contribution of each kernel). The final result is then passed through the activation layer before proximity correction.

Table 1: Dataset Statistics. For these densely annotated datasets we provide the typical number of annotated instances per video, video length, and the portion of frames which are annotated.

Dataset	Videos	Instances	Frames	% Labeled	Domain
Cityscapes-VPS [24]	$50$ (val)	$5-57$	$30$	$1/6^{th}$	Streets
VIPSeg [30]	$3536$	$1-78$	$4-81$	All	Open

3.2.3 Proximity Correction

Layers before the head layer do not contribute directly to pixels. Instead, their outputs are the inputs to some later set of convolution kernels. While we account for the upsampling with how we handle PixelShuffle, we still must address the fact that these kernels act on $k\times k$ neighborhoods at each filter map. So, we use an aggregation filter to account for the fact that a given point in some intermediate feature map contributes to a $k\times k$ neighborhood of points in the next layer’s feature maps. In the case of the penultimate layer, this is a simple $k\times k$ box filter. For the sake of clarity, we reserve further details for our code.

4 Understanding Representations

4.1 Dataset and Training Details

Figure 3: The implicit neural canvas. We show representative example contribution maps for various layers of FFN [47] and NeRV [7]. Notice how early FFN layers manifest strong fourier patterns, and how the last layers of both FFN and NeRV tend to resemble the image, with NeRV layers being reminiscent of classical image processing filters.

Figure 4: Grou** contributions. We compute the variance of the difference from expected contribution for different grou**s of contributions - Instances of objects and background, RGB color-based clusters, Gabor filter-based clusters and regular gridcells. These results suggest that INRs ignore space while preferring instances, color, and edge features.

To analyze the relationship between neurons and pixels in encoded signals, we use videos from the Cityscapes-VPS [24] and VIPSeg [30] video panoptic segmentation datasets. To select a few videos from these datasets, we look for examples containing objects of diverse shapes, sizes and categories, to afford diversity in analysis. Table 1 provides the number of instances per frame in each dataset, the percentage of annotated frames and other information. Having instance and object category level information enables us to answer questions pertaining to whether INRs understand semantics and instances and also helps analyze the distribution of representations for objects of different types. Further, it would help us determine whether the groups of parameters attending to the same objects remains consistent across frames, whether the model omits representing small objects, and investigate other important properties. We train our FFNs and NeRVs on downsampled images, to keep the model and contribution map sizes small. Specifically, we resize and crop all videos to $128\times 256$ resolution. We choose model sizes and training times such that bits-per-pixel and reconstruction quality are roughly equivalent across models and video frames. We provide more details in the appendix.

4.2 Contributions Correlate with Colors and Edges

Figure 3 shows some example contribution maps for neurons at different layers for both MLP-based and CNN-based INRs. FFN and NeRV head layer neurons seem to learn some form of whole scene representation, and the CNN penultimate layer has similar characteristics. The contribution maps of the FFN layers suggest a progression from representing Fourier features in early layers, to representing the image in question, with Fourier artifacts, in the last layer. For both the NeRV layers, there seem to be a variety of features captured by various neurons, including edges, textures, colors, and depth. These maps show that the behavior of a neuron is not simply to represent a pixel or set of pixels, instead, it learns some attributes of the scene.

Figure 5: Contribution vs. intensity. We compare contribution and intensity in alternating rows. In the top row, we sum all contribution maps for the indicated layer. In the next row, we show the difference between this, and the raw image intensity (sum of all color channels) to show when contribution does and does not correlate with intensity. The next two rows repeat this for a frame from another video.

Figure 6: Pixels per neuron. We compute the pixels activated for each neuron, and plot them for each layer, with each layer sorted independently by order of increasing pixels, to show the trends in how many pixels each neuron is responsible for, for frames from two different videos.

We explore these behaviors quantitatively by aggregating neuron contributions to pixels, first in terms of instances, and then also by clustering pixels by color value (RGB Clusters), clustering on Gabor filter features (Gabor Clusters, details in appendix), and clustering pixels spatially (taking equal size rectangular gridcells). We hypothesize that for a type of pixel clustering to be meaningful for neurons, each neuron should tend to have high activation in some clusters, and little to no activation in others. Quantitatively, the variance of contributions across clusters would be high. However, using raw variance of contributions would give massive bias to clusterings that form large and small regions, where some contributions would be trivially small. So instead, we first compute the expected contribution for each cluster, which is the contribution over the whole image, normalized by size of the given cluster.

$$\small\text{cont}_{\text{expected}}=\left(\frac{\text{area}_{\text{cluster}}}{\text{area}_{\text{image}}}\right)*\text{cont}_{\text{image}}$$ (2)

We then obtain the actual contributions to each cluster, $\text{cont}_{\text{actual}}$ by summing the contributions of a neuron to the pixel values in the cluster, and contribution deltas are taken as $\delta_{\text{cont}}=\text{cont}_{\text{expected}}-\text{cont}_{\text{actual}}$. We then normalize each $\delta_{\text{cont}}$ by dividing by $\text{cont}_{\text{expected}}$ such that they are percentage differences between actual and expected neuron contributions to each cluster. Finally, we compute the variance (standard deviation) of the normalized percentage differences. The above is repeated for all neurons in a layer.

We plot the result of performing this computation for Instances, RGB, Gabor, and gridcell clusters, across all neurons, for all three layers of the FFN and the last two layers of NeRV, in Figure 4. This computation reveals quantitatively what Figure 3 hints at – there is relatively little meaningful correlation between contributions and space, as seen in the low variances of gridcell representations. Instead there is in general a surprisingly high variance for neurons when we group their contributions using instance masks, which suggests that they learn some type of object semantics. Given that the variances for RGB and Gabor clusters are also quite high, we suggest that these are low-level semantics, a combination of color and edge features. There is also an interesting trend with the NeRV Block 3, where it seems to have some meaningful correlation with gridcells. However, we note that the top 25% of variances for Instances, RGB, and Gabor, are almost all higher than even the highest gridcell variance, and therefore we believe the intuition holds.

4.3 Contribution is not Intensity

Figure 7: Neurons per pixel. We show how many neurons represent significant portions of each pixel, as a percentage of the total neurons in the indicated layer, at two different thresholds for “activation.”

Figure 8: Neuron contributions to things and stuff. We aggregate the contributions of neurons to the instances (things) and background (stuff) in a frame from 2 videos. For FFN, we sample 3 neurons from each layer and for NeRV, we sample 2 neurons each from the last 2 layers. Instances belonging to the same category are depicted by distinct markers of the same color. For FFNs, we show contributions only for the given frames, whereas for NeRVs, we show these contributions over time for 6 frames from the source video.

Furthermore, we observe an interesting property that not all pixels require equal representation. Figure 5 shows the sum of absolute contributions over all neurons for distinct layers of the FFN and NeRV. We compare these aggregated, layer-level contribution maps to the total intensity (summing over color channels) of the image they correspond to. The heatmaps reveal a correspondence between the areas of large contributions and the areas of high intensity in the input image. This suggests that the neurons of an INR largely attend to higher intensities in the image.

While there is naturally some intensity bias, as a result of the pixelwise loss functions used by these networks, this does not fully explain the behavior of the representation. To discover other concepts that the INR potentially pays attention to, we subtract the intensity image from the contribution heat map. In Figure 5, this reveals that in addition to intensity, the INR layers also pay attention to other low-level information such as edges and textures. We note that the NeRV layers and the early FFN layers have a tendency to over-represent the people in the second image, relative to intensity, while the FFN layer 3 contributions are largely consistent with intensity. Additionally, not all edges are treated equally, with NeRV having large contributions for some edges, and small contributions for others.

4.4 Representation is Distributed

When training an INR, one might naturally wonder, since there are more pixels than neurons, how many pixels are represented by each neuron, and if this representation tends to be distributed. To analyze the manner in which neuron-to-pixel contributions are distributed, we compute the ratio of a neuron’s contribution to a pixel to the total contribution across neurons at a pixel. We threshold on this ratio to determine whether a pixel receives a significant portion of its contribution from a specific neuron or not. The threshold used is a simple reciprocal of the number of neurons in a layer. We then compute the total number of pixels that a specific neuron contributes to meaningfully and sort this across neurons, repeating this analysis individually for different layers of the FFN and NeRV.

We show these contribution counts in Figure 6. Since different layers have different numbers of neurons, we resample the neurons of each layer to a common length using linear interpolation to preserve their trends. Notice how the outermost layers of the FFN and NeRV (layer 3 and head layer respectively) show the highest trends with a decreasing trend towards the inner layers of both networks. Note that nearly $50\%$ of the neurons in FFN Layer 2 have almost no contributions, producing a flat line in the first half. This suggests that INRs follow similar trends to discriminative networks, where earlier layers contribute towards structural features while later layers have more stochastic features.

To continue our exploration of how the learned INRs distribute the image representations, we compute at each spatial image location, the density of neurons that contribute to it above a threshold, $\tau$. To set $\tau$ for a certain layer, consider the sorted raw contributions of all neurons across locations of the image. We find that a large portion of the contributions are small and the number of larger contributions is fewer in comparison (see Appendix). We obtain the raw contribution value that lies at the $x$-th percentile of this curve, i.e., the contribution of the neuron to the left of which lies $x\%$ of the area under the curve. The value corresponding to this percentile is chosen as the threshold for all neuron contribution maps. We show this for two different thresholds in Figure 7, which shows the regions of the image with a higher density of dedicated neurons.

High intensity and highly textured areas not only have high contributions as seen earlier in Figure 5, but they also have a large portion of neurons dedicated to them. Increasing the threshold reveals the image areas with the highest neuron density, and this reveals an overwhelming bias for edges with NeRV. However, all these trends are greatly subdued for the FFN at both its head and penultimate layers, suggesting that while the magnitude of its contributions correlates almost perfectly with the image intensity, its actual representation is more evenly distributed by comparison.

4.5 Objects, Categories, and Motion

To analyze whether INRs have meaningful representations in terms of objects, we aggregate neuron contributions over pixels in each instance’s segmentation map. Consider the two sample videos shown in Figure 8. Each has a different set of objects with some categories having multiple instances. Instances may move, disappear and reappear over time. For the FFN, we sample three neurons each from the three layers and depict each neuron’s normalized contributions to every instance (as a percentage of contributions to all instances) in the first frame of the video. Notice how contributions to instances of the same category are not grouped together. This is related our findings in Figure 4, and motivates our claim that these networks learn low-level object features, but lack class-level semantics.

For the NeRV, we sample two neurons each from the head layer Block 3, and track the normalized per-instance contributions over multiple frames of the video. We see that a neuron’s contributions to an instance remains relatively static over time. When there is fluctuation, we notice that this is often correlated with instances that move, such as persons. Another thing to note is how contributions to high intensity instances such as sky are higher, which is in agreement with our earlier observations.

Figure 9: Neurons and motion. We show the correlation between motion and changes in neuron contribution over time by computing optical flow between two frames and the difference in contribution maps between those same frames. We plot both of these. Fluctuation is driven by motion, and it seems the areas revealed by motion matter equally to the objects that are actually moving.

Further delving into how neurons in NeRV respond to motion, we visualize the optical flow map and the fluctuation of total contribution in the last 2 layers of a NeRV. We are careful to select videos with no camera motion, which could lead to spurious or whole-frame flow. Figure 9 depicts this for two different time steps in each video (the flow and contribution fluctuation are computed between a specific frame and the immediately preceding frame). We see that the changes in contributions are focused more in areas around the regions of flow. It appears that when a moving object reveals new background, this causes fluctuations in the spatially proximal pixels, even though they technically contain no motion themselves. Thus, we see that in spite of its lack of high-level object semantics, the changes in how a NeRV represents a video across time are dominated by the motion of the entities in the video.

4.6 Neurons can be Clustered by Contributions

Figure 10: UMAP for INRs. We plot the neurons from the head layers of NeRVs with different seeds, and show the contribution maps corresponding to the neurons indicated.

We propose a unique approach for computing a vector representation of INR neurons. Inspired by the relatively high variance for Gabor clusters in Figure 4, we take the pixels in each frame of each video, and cluster them according to Gabor filter features with $k$ clusters. We then aggregate the contribution map values according to the clusters, yielding a $k$ dimensional vector representation of every neuron. We can then project these representations with UMAP to allow for plotting them in 2D space, which we do in Figure 10. As this Figure shows, there are many different “types” of neurons when we separate them according to edge contributions. This representation mechanism we develop with XINC gives another powerful method for visualizing and examining the internals of INR.

5 Conclusion

In this paper, we propose XINC, a novel framework for analyzing INRs for image and video representation. We use XINC to find interesting trends with CNNs and MLPs, such as how while they lack high-level object semantics, they do have biases that mimic object representation, particularly with edges and colors. We also show how fluctuations in contribution for NeRV are driven by motion. We show how XINC can help us identify interesting groups of related neurons. We provide XINC, which is flexible for MLPs and CNNs, as a resource to help drive future research for INRs.

Supplementary Material

The main purpose of this appendix is to expand our results from the main paper by showing analysis for frames from other videos, to show that our analysis and findings hold across a variety of different videos. For additional implementation details, including how we construct the contribution maps for various layers, please refer to our code.

6 Gabor features

Gabor filters are used to analyze different textures by detecting patterns at different orientations and scales. To extract Gabor features in our experiments, we employ Gabor filters at four distinct orientations and three different scales to account for a diverse range of textures. For further details on how we aggregate Gabor features for grou** contributions into Gabor clusters, please refer to our code.

7 Contribution Maps

Using Gabor filter features, we cluster neurons at each layer of MLP-based and CNN-based INRs and plot them using UMAP. For a few of these clusters, Figures 11, 12, 13, 14 and 15 show the contribution maps of four neurons sampled from each cluster. We see that early FFN layers learn Fourier patterns, and the last layer of NeRV tends to resemble the image. Further, note how neurons belonging to the same cluster tend to learn similar contribution maps as compared to neurons belonging to different clusters.

Figure 11: Neuron Clusters. Contribution maps of sample clustered neurons for the first (red-bordered) frame for video 1.

Figure 12: Neuron Clusters. Contribution maps of sample clustered neurons for the first (red-bordered) frame for video 2.

Figure 13: Neuron Clusters. Contribution maps of sample clustered neurons for the first (red-bordered) frame for video 3.

Figure 14: Neuron Clusters. Contribution maps of sample clustered neurons for the first (red-bordered) frame for video 4.

Figure 15: Neuron Clusters. Contribution maps of sample clustered neurons for the first (red-bordered) frame for video 5.

Figure 16: Layerwise UMAP for INRs trained with different seeds. We show results of Gabor filter based clustering neurons in each layer of MLP- and CNN-based models with different seeds. As seen in Figure 10, each cluster has neurons belonging to different seeds, indicating that models of different seeds learn a set of neuron “types”.

In Figure 16, we provide a supplement to Figure 10 by clustering neurons from models with five different seeds, for each layer of both MLP-based and CNN-based INRs on frames from five videos.

8 Grou** Contributions

In Figure 17, we provide a supplement to Figure 4 by showing results for frames from 5 additional videos. As expected, the major trends hold. We see far higher neuron contribution difference variances when using the instances, RGB clusters, and Gabor clusters, compared to space clusters (gridcells). We see that in general Gabor and RGB clusters have equal explanatory power for neuron contributions compared to instances, reinforcing our hypothesis that these networks have low-level, rather than high-level, object semantics. Interestingly, the trends are consistent across all layers, except for the trends for space are weaker with the NeRV than the FFN, perhaps due to the spatial bias from NeRV’s convolutional kernels.

Figure 17: Grou** contributions. Similar to Figure 4, we observe higher variances in neuron contribution differences when using instances, RGB clusters, and Gabor clusters in contrast to space clusters (gridcells). These observations lend support to our hypothesis that INRs prefer low-level object semantics while demonstrating a tendency to disregard space.

9 Representation is Distributed

We extend the results from Section 4.4 for frames from some additional videos.

Figure 18: Pixels per neuron. We supplement Figure 6 by plotting the pixels activated per neuron for frames from 5 additional videos. These results reveal similar trends as seen in Figure 6.

Figure 18 reveals the same trends as Figure 6, for frames from 5 additional videos. This is consistent whether the frame is street or open domain, and whether it is dim or more brightly lit.

Figure 19: Neurons per pixel. We show the percentage of neurons in each layer, that represent significant portions of each pixel, at two different thresholds. This figure reveals properties consistent with Figure 7 for 5 additional video frames.

Figure 20: Distribution of Neuron Contributions. We plot the distribution over all neuron contributions in each layer. This shows that a large fraction of neuron contributions tend to be relatively smaller in magnitude. The contribution values corresponding to the $10^{th}$ and $50^{th}$ percentile of each distribution are used in selecting thresholds for Figures 7 and 19.

The same trends from the video in Figure 7 are consistent for 5 additional video frames in Figure 19. Pixels for the FFN layers tend to be represented by similar amounts of neurons, whereas for the NeRV, these numbers vary widely. Some neurons are represented by many pixels, others by very few (relatively). The contrast is sharper in general for last layers (NeRV head layer, FFN layer 3).

Figure 20 shows how a large portion of the raw contributions are relatively smaller in magnitude for each layer. The thresholds for Figure 19 are selected using the contribution value at the $10^{th}$ and $50^{th}$ percentile of these curves.