TRIP: Trainable Region-of-Interest Prediction for Hardware-Efficient Neuromorphic Processing on Event-based Vision ^††thanks: This work was partially funded by research and innovation projects REBECCA (KDT JU under grant agreement No. 101097224), NeuroKIT2E (KDT JU under grant agreement No. 101112268), and NimbleAI (Horizon EU under grant agreement 101070679).
^† Corresponding author. Email: [email protected]
^∗ Code: https://github.com/ERNIS-LAB/TRIP

Hard attention strategies for restricting computations by directing image processing towards relevant regions of input space have long been explored in computer vision. Early models analyze low-level image features to predict regions of high saliency based on variations in pixel intensity [15]. Later works increasingly emphasized the task of salient region prediction as an action selection policy [5], iteratively improving predictions over time. Reinforcement learning (RL) algorithms have been adopted in hard attention to learn the optimal policy for placing a sensor with limited bandwidth on a given input region [7, 6, 16]. While RL-based hard attention is effective, the training complexity poses major challenges for hardware-aware optimization.

Deep Recurrent Attention Writer (DRAW) uses recurrent units within a variational autoencoder to iteratively predict salient regions of input images [17]. Importantly, DRAW uses a differentiable mechanism for generating an ROI with Gaussian kernels, enabling end-to-end backpropagation training without using RL. Neuromorphic DRAW applied the differentiable crop** of DRAW to event-based classification tasks to improve accuracy by filtering out irrelevant events [18]. Our TRIP framework leverages DRAW’s Gaussian kernels to facilitate differentiable hard attention while introducing hardware-efficient algorithm designs for event-based vision processing on neuromorphic processors.

Event-based CNN, trained by specialized activation regularization methods, has high activation sparsity within each network layer [19]. SENECA is a multi-core embedded digital neuromorphic processor specialized in processing event-based CNNs [10]. It performs event-driven computation that exploits the sparsity in sensory inputs and network activations. Additionally, it executes data-flow processing across cores, increasing the parallelism of network processing and diminishing the memory cost for neural activations. Event-driven depth-first convolution is a unique scheduling method SENECA supports for event-based CNNs [2]. The method prioritizes the network’s layer dimension by consuming neural activation events right after their generation. Therefore, it maximizes the neuromorphic processor’s benefits on parallelism and latency. Our TRIP framework with event-based CNN maximizes the hardware efficiency of hard attention on SENECA by exploiting the hardware advantages.

We propose the Trainable Region-of-Interest Prediction (TRIP) framework for efficient event-based classification. It uses hard attention within an event-driven neuromorphic processing pipeline. The framework efficiently classifies event streams using an actively generated ROI that is predicted from the input events. An ROI’s receptive field covers a small region of the event-based camera’s field of view. As shown in Figure 2, our TRIP framework consists of three subsequent components: ROI prediction, ROI generation, and classification. The ROI prediction component consists of an event-based CNN that determines the location and receptive field of the ROI. It predicts the ROI parameters using a downsampled low-resolution input, reducing the processing overhead of ROI prediction. The ROI generation component generates the cropped ROI using the predicted parameters. It uses an $N\times N$ grid of differentiable truncated Gaussian kernels (tGK) to produce a fixed $N\times N$ output from a varying-size receptive field. This ensures consistently low processing cost of classification. Moreover, we introduce dynamic average pooling (DAP) to replace tGK for efficient inference on the embedded neuromorphic processor. The classification component consists of an event-based CNN that performs classification on the ROI. The entire framework is differentiable, allowing it to be trained end-to-end. For efficient computing on SENECA, we increase the activation sparsity of the event-based networks during training.

The ROI prediction component produces the ROI parameters based on the downsampled input events from max-pooling. The ROI prediction network outputs three scalar values. These values are decoded to determine the ROI location and receptive field as follows,

$$g_{x}=\frac{A}{2}\cdot(tanh(\hat{g_{x}})+1)$$

(1)

$$g_{y}=\frac{B}{2}\cdot(tanh(\hat{g_{y}})+1)$$

(2)

$$\delta=S\cdot(sigmoid(\hat{\delta})+1)$$

(3)

where $\hat{g_{x}},\hat{g_{y}},\hat{\delta}$ are the raw scalar outputs from the ROI prediction network, ($g_{x},g_{y}$) is the center location of the overall receptive field formed by the $N\times N$ grid of tGK, $\delta$ is the distance between two adjacent tGK, $A$ and $B$ are the image width and height, and $S$ is a distance scaling factor. The $g_{x}$ and $g_{y}$ parameters always initialize in the center of the image at the start of training and can move across the entire input space. The variable $\delta$ allows control over the size of the receptive field of the ROI.

We employed tGK for ROI generation, an efficient variation of the method introduced in DRAW [17]. The ROI generation component generates the $N\times N\times 2$ fixed-resolution input ROI event streams for classification, in which $N$ is the width and height of the ROI and $2$ is the polarity channel. The component uses $N\times N$ differentiable tGK to compute the ROI during training. The 2D mean positions of the tGK are computed according to the predicted center location of the overall receptive field as follows,

$$\mu_{x}^{i}=g_{x}+(i-\frac{N}{2}-0.5)\cdot\delta,\ i\in[0,N-1]$$

(4)

where $\mu_{x}^{i}$ is the mean x-axis position of the tGK on the $i^{th}$ column. The mean y-axis position of the tGK can be computed using the same equation with $g_{y}$. Eventually, each tGK has a two-dimensional mean position $(\mu_{x}^{i},\mu_{y}^{j})$, in which $i$ and $j$ are the column and row index. Here, we assume $N$ is an even number.

By knowing the mean positions of the tGK, we compute the weight of the tGK corresponding to each pixel location of the input events as follows,

$$F_{x}^{i}[n]=\begin{cases}\exp({\frac{(n-\mu_{x}^{i})^{2}}{2\sigma}})&\text{for }n\in[\mu_{x}^{i}-\frac{\theta}{2},\ \mu_{x}^{i}+\frac{\theta}{2}]\\ 0&\text{otherwise}\end{cases}$$

(5)

where $F_{x}^{i}[n]$ is the x-dimension weight component of tGK on the $i^{th}$ column corresponding to pixel locations at column $n$, $\sigma$ is the variance which is a pre-defined parameter, and $\theta$ is the size of the tGK with non-zero weights. The y-dimension weight component is computed in a similar manner.

Each ROI input event value to the classification network is computed by the corresponding tGK as follows,

$$v_{(x_{i},y_{j})}=\mathbf{F_{x}^{i}}\cdot\mathbf{I}\cdot\mathbf{{F_{y}^{j}}}$$

(6)

where $v_{(x_{i},y_{j})}$ is the value of the event at location $(x_{i},y_{j})$ of the $N\times N\times 2$ input to the classification network, $\mathbf{F_{x}^{i}}$ and $\mathbf{{F_{y}^{j}}}$ are the weights for the corresponding tGK, and $\mathbf{I}$ are the binned raw input events from one polarity. The two polarity channels of the classification inputs are computed using the same equation.

Compared to Gaussian kernels introduced by DRAW, our tGK significantly reduces the computation required for ROI generation while maintaining differentiable. Specifically, our adoption of tGK reduces the computational complexity of ROI generation from $O(AB)$ to $O(\theta^{2})$ by skip** the pixel locations with insignificant weights. Since $\theta$ is at least ten times smaller than $A$ and $B$ in practice, the tGK can be orders of magnitude more efficient than Gaussian kernels.

The tGK can be accelerated by customized application-specific integrated circuit (ASIC) designs. However, its efficiency is hard to achieve on the embedded CPU within our targeted neuromorphic processor. Though the number of computations is small, the overheads of locating the non-zero elements and performing weighted operations for tGK are substantial. Firstly, assigning each input event to multiple overlap** kernels requires a complex implementation to avoid iterating all kernels, introducing significant overhead to the instruction memory. Secondly, the distance between the kernel center and the event location must be computed for each assigned event, bringing additional overhead on computation. Hence, tGK on an embedded core with limited instruction memory and compute capability is not feasible.

To mitigate the problem, we introduce Dynamic Average Pooling (DAP) as a hardware-efficient alternative for ROI generation during inference on the embedded neuromorphic processor. The DAP replaces the Gaussian kernels with simple non-overlap** average poolings. The kernel size of the average pooling changes dynamically based on the size of the overall receptive field the ROI corresponds to. We compute the range of the overall receptive field using the ROI parameters from the ROI prediction component as follows,

$$x_{max}=g_{x}+(\frac{N}{2}-0.5)\cdot\delta+\frac{\theta}{2}$$

(7)

$$x_{min}=g_{x}-(\frac{N}{2}+0.5)\cdot\delta-\frac{\theta}{2}$$

(8)

and the dynamic kernel size of each average pooling in the DAP is computed as follows,

$$k_{DAP}=(x_{max}-x_{min})/N$$

(9)

where $x_{max}$ and $x_{min}$ define the range of the receptive field on the x-axis of the raw input space. The range on the y-axis can be computed using the same equations with $g_{y}$ and the receptive field is square.

The embedded implementation of DAP is simple. A closed-form equation exists to compute the sole corresponding kernel of each input event, and the ROI generation is distance invariant. However, the ROI parameters in DAP are not differentiable. Therefore, we first train the networks with tGK, and then fine-tune the classification network with a fixed ROI prediction network and DAP.

The event-driven neuromorphic processor exploits activation sparsities in neural networks by only processing non-zero activations. Therefore, input to each layer for synaptic operation is supposed to be as sparse as possible. To maximize the efficiency of TRIP on the event-driven neuromorphic processor, we adopt event-based CNNs for ROI prediction and classification. Unlike regular CNN, our event-based convolutional layer for the neuromorphic processor performs BatchNorm and MaxPool before the ReLU activation, outputting sparse events straight to the subsequent layer for synaptic integration. Moreover, we used ReLU function in the vanilla RNN for sparse recurrent processing. Furthermore, we perform hardware-aware optimizations on the event-based CNNs. The optimizations comprise sparsity-aware and quantization-aware training, reducing projected computation cost and memory requirement on the hardware.

To increase the activation sparsity of event-based CNNs in TRIP, we adopt the $L1$ regularization loss [19] on the activation values of the layers that have ReLU as the activation function. The loss encourages the network to reduce the activation values so as to have fewer non-zero activations and increase the sparsity. Additionally, we use quantization-aware training to reduce weight precision to 4 bits [2]. There is a shared power-of-two scaling factor $s$ for all the weights of the same layer. During on-chip computation, a weight value is obtained by multiplying the saved 4-bit integer with $2^{s}$. The quantized parameters reduce the on-chip memory required for network deployments and the computation cost of synaptic integration.

Gesture recognition is an ideal task for evaluating approaches with hard attention, as a compact region of the input can provide sufficient information for classification. The DvsGesture dataset [11] enables us to compare our method with other state-of-the-art solutions on the task of gesture recognition with an event-based camera.

The Marshalling Signals dataset [12] is more recent, less explored, and more difficult than DvsGesture. The dataset presents gestures at multiple distances from the event-based camera. Therefore, it allows us to further test the ROI prediction, particularly its ability to adjust to varying sizes.

To study the effects of the reduced input resolutions and the hard attention overheads in a controlled setup, we synthetically generated a dataset based on the N-MNIST dataset [23]. The generated dataset enables us to test the performance of TRIP under different input resolutions and structured event noises.

To accurately assess the hardware efficiency of TRIP, we implemented our hardware-optimized TRIP algorithm on the SENECA neuromorphic processor [10]. To compare with state-of-the-art neuromorphic solutions on event-based vision, we used the DvsGesture dataset [11] to benchmark the performance of our solution in terms of accuracy, latency, energy consumption, and hardware’s effective area.