Investigating Event-Based Cameras for
Video Frame Interpolation in Sports

Abstract

Slow-motion replays provide a thrilling perspective on pivotal moments within sports games, offering a fresh and captivating visual experience. However, capturing slow-motion footage typically demands high-tech, expensive cameras and infrastructures. Deep learning Video Frame Interpolation (VFI) techniques have emerged as a promising avenue, capable of generating high-speed footage from regular camera feeds. Moreover, the utilization of event-based cameras has recently gathered attention as they provide valuable motion information between frames, further enhancing the VFI performances. In this work, we present a first investigation of event-based VFI models for generating sports slow-motion videos. Particularly, we design and implement a bi-camera recording setup, including an RGB and an event-based camera to capture sports videos, to temporally align and spatially register both cameras. Our experimental validation demonstrates that TimeLens, an off-the-shelf event-based VFI model, can effectively generate slow-motion footage for sports videos. This first investigation underscores the practical utility of event-based cameras in producing sports slow-motion content and lays the groundwork for future research endeavors in this domain.

Keywords. Event-based camera, video frame interpolation, video understanding, sports analysis, slow motion.

I Introduction

Slow-motion footage has become an indispensable tool in sports broadcasting, serving to highlight or replay pivotal moments with enhanced emotions and accurate details. Beyond its aesthetic appeal, slow-motion technology offers a multifaceted utility in sports analysis. By slowing down the action, it provides analysts and coaches with a finer temporal resolution to scrutinize player movements, dissecting techniques and strategies with precision. Moreover, slow-motion footage empowers referees to make more informed decisions regarding fouls, as it allows for a comprehensive review of contentious incidents at lower speeds. This dual function of slow-motion technology, therefore, both enriches the viewing experience for fans and enhances the decisions of coaches and officials.

Figure 1: Bi-camera recording setup for event-based video frame interpolation in sports. We propose a two camera recording setup, including an RGB camera and an event-based camera to capture sports videos, and temporally align and spatially register both cameras. We record video footage of racquet sports characterized by high-speed movements of the ball and racquets, utilizing our specialized setup, and demonstrate the effectiveness of off-the-shelf event-based video frame interpolation techniques in producing slow-motion footage.

However, the widespread adoption of slow-motion technology in sports broadcasting has been held back by the prohibitive costs associated with high-speed camera setups, which are typically only feasible for major competitions. As a result, lower leagues and smaller-scale sporting events often lack access to this technology, further increasing the accessibility gap in sports media production. Additionally, the infrastructure required to support high-speed camera systems entails considerable investment, presenting a significant barrier for many organizations.

Fortunately, recent advancements in deep learning started to revolutionize the landscape of slow-motion production. New techniques enable Video Frame Interpolation (VFI), aiming to generate intermediate frames produced by standard cameras, circumventing the need for specialized, high-cost equipment. Moreover, the emergence of event-based cameras represents a groundbreaking development in this field. Unlike conventional cameras that capture entire frames at fixed intervals, event-based cameras detect and record individual pixel-level changes in luminance asynchronously, only transmitting data when a visual event occurs. This innovative approach reduces data redundancy and provides crucial information on motion dynamics between frames at a much higher frame rate. By capturing motion in a continuous fashion, event-based cameras offer unparalleled precision in frame interpolation, enabling the generation of seamless slow-motion sequences with enhanced realism.

In this study, we investigate the use of an off-the-shelf event-based video frame interpolation method, namely TimeLens [1], for the creation of slow-motion footage in the sports domain. In particular, we design and implement a bi-camera recording setup, including an RGB camera and an event-based camera to capture sports videos. We then temporally align and spatially register both cameras to project the information provided by the event-based camera to the RGB frames. As illustrated in Figure 1, our experimental approach entails recording sports video footage, with a particular focus on racquet sports, characterized by high-speed movements of the ball and racquets, utilizing our specialized setup. Through our analysis, we demonstrate the effectiveness of TimeLens in generating slow-motion sequences, providing valuable insights into the dynamics of sports events. This study serves as an initial step towards exploring the potential of event-based camera systems in sports video production, laying the groundwork for future research aimed at refining and expanding this technology for broader applications in the sports industry.

II Related Work

Our work relates to the research fields of video frame interpolation, event-based cameras, and sports video understanding.

II-A Video frame interpolation

Video Frame Interpolation (VFI) involves generating intermediate frames between existing video frames, effectively increasing the frame rate and allowing the creation of slow-motion content [2]. Over the years, various methods have been proposed, leveraging convolutional neural networks (CNNs) [3], phase-based techniques [4], optical flow algorithms [5], or Generative Adversarial Networks (GANs) [6]. Moreover, industrial tools such as XtraMotion from EVS Broadcast Equipment [7] facilitate slow-motion video production in professional settings. The field can count on the availability of numerous datasets and benchmarks to develop methods, including popular resources such as Vimeo90K [8], UCF101 [9], X4K1000FPS [10], Middlebury [11], and the SNU-FILM [12] datasets. Recently, SportsSlowMo [13] introduced the first dataset dedicated to human-centric sports video clips for VFI using only the RGB modality. Our study also aims to investigate sports-centric videos, incorporating event-based camera feeds as an additional modality for VFI.

II-B Event-based camera

Event-based cameras, also known as neuromorphic sensors, operate differently from traditional cameras. Rather than capturing entire frames at fixed intervals, event-based cameras only record changes in brightness, known as “events”, at pixel level. These cameras provide several advantages over conventional cameras, such as a high temporal resolution of up to $10{,}000$ frames per second, low power consumption, wide dynamic range, sparse data output for efficient data transmission and storage, and low latency [14]. Recently, event-based cameras have emerged as integral components in video frame interpolation pipelines thanks to their ability to capture information between RGB frames. This feature enables the modeling of more intricate scenarios, including non-linear motion often present in sports content. Notable methods like Timelens [1], Timelens++ [15], or SuperFast [16] have demonstrated impressive performance in generating slow-motion footage from both modalities. However, their application has predominantly focused on close-up shots within controlled environments. In this study, we investigate the ability of event-based VFI methods to handle sports videos captured in natural, uncontrolled settings.

II-C Sports Video understanding

Over the past decade, there has been an increased focus on sports video understanding research [17, 18] thanks to the availability of large-scale datasets [19, 20, 21, 22]. The SoccerNet datasets and challenges [23, 24] have contributed to multiple video understanding tasks in sports such as, action spotting [25], replay grounding [26], camera calibration and player re-identification [27], multiple player tracking [28], multi-view video recognition [29], and dense video captioning [30]. Recently, deep learning enabled real-time data and analysis on player performance [31, 32, 33], including, segmentation and tracking of players and the ball [34, 35], game tactics [36], action spotting [25, 26, 37, 38, 39, 40, 41, 42, 43], and the creation of video highlights or summaries  [44, 45]. Overall, these developments elevate coaching strategies and enhance the overall viewer experience [46, 47]. In this work, we show the capabilities of event-based VFI methods to generate high-speed footage for downstream sports video analysis.

III Methodology

In this section, we provide a rigorous definition of the event-based video frame interpolation task, describe our sports data recording setup, and explain the temporal alignment and spatial registration between our bi-camera setup.

III-A Event-based video frame interpolation

Let $\mathbf{I}(t_{i})$ represent the video image recorded at time $t_{i}$ by a regular RGB camera, where $\mathbf{I}(t_{i})\in\mathbb{R}^{H\times W\times C}$ denotes an image frame with height $H$, width $W$, and $C$ channels. Then, let $\mathbf{E}(t_{i})$ be the event data recorded by the event-based camera between time $t_{i}$ and $t_{i+1}$, consisting of a set of events $\{\mathbf{e}_{j}\}$, where each event $\mathbf{e}_{j}=(x_{j},y_{j},t^{\prime}_{j},p_{j})$ corresponds to a change in luminance at pixel location $(x_{j},y_{j})$ and at time $t^{\prime}_{j}$ with polarity $p_{j}\in\{-1,1\}$.

The event-based video frame interpolation problem can be formulated as follows. Given a sequence of event data $\{\mathbf{E}(t_{1}),\mathbf{E}(t_{2}),...,\mathbf{E}(t_{n})\}$, where $t_{1}<t_{2}<...<t_{n}$, and the corresponding image frames $\{\mathbf{I}(t_{1}),\mathbf{I}(t_{2}),...,\mathbf{I}(t_{n})\}$, the objective is to estimate some intermediate frame $\hat{\mathbf{I}}(t_{i+\Delta t})$ at any arbitrary time $t_{i+\Delta t}\in\interval[open]{t_{i}}{t_{i+1}}$. Hence, event-based video frame interpolation aims to generate intermediate frames between consecutive frames captured by a regular RGB camera, using the information provided by an event-based camera.

In this work, we focus on a particular case of event-based video frame interpolation where only the bounding image frames $\mathbf{I}(t_{i})$ and $\mathbf{I}(t_{i+1})$, along with the registered event data, $\mathbf{E}(t_{i})$ are used to compute intermediate frames $\hat{\mathbf{I}}(t_{i+\Delta t})$.

III-B Bi-camera sports data recording setup

We design and implement a bi-camera (RGB and event-based) recording setup for sports data collection by aligning two camera supports and mounting them on tripods, as illustrated in Figure 2. To ensure accurate alignment, we position the event-based camera such that its optical center is aligned with the center of the RGB camera lens.

Figure 2: Camera setup. Picture of our bi-camera setup used for sports data collection. We align both the RGB camera and event-based camera on a tripod.

For the RGB camera, we use an iPhone $12$, capable of capturing videos at $120$ frames per second (FPS) with a resolution of $1920\times 1080$ pixels. For the event-based camera, we employ the Prophesee EVK4, which offers a resolution of $720\times 1280$ pixels, a time resolution equivalent to over $10{,}000$ FPS, and a dynamic range exceeding $120$ decibels (dB). Moreover, the event-based camera exhibits a low light cutoff of $0.08$ lux, ensuring reliable performance even in dimly lit conditions. We employed the Metavision software suite for both recording and post-processing of the event-based camera footage directly on-site.

We recorded footage of two semi-professional tennis players engaging in a game on an indoor court illuminated by LED panels to avoid flickering caused by traditional neon lighting. The players were instructed to play a regular game while we moved the cameras between each game session. In total, we recorded approximately $30$ minutes of usable footage captured from the RGB and event-based cameras.

III-C Temporal alignment and spatial registration

Before proceeding with event-based video frame interpolation, it is crucial to temporally align and spatially register both cameras. Hereafter, we describe the components and objectives, and then detail our alignment and registration processes step-by-step.

Let $(x_{j},y_{j})$ denote a pixel location at time $t_{j}$ in the event-based camera, and $(x_{i},y_{i})$ represent the corresponding pixel location at time $t_{i}$ in the RGB camera. We define the projection function $\mathbf{P}:\mathbb{R}^{H_{\text{event}}\times W_{\text{event}}\times T_{\text{event}}}\rightarrow\mathbb{R}^{H_{\text{image}}\times W_{\text{image}}\times T_{\text{image}}}$ as

$$(x_{i},y_{i},t_{i})=\mathbf{P}(x_{j},y_{j},t_{j})\,,$$ (1)

where $H_{\text{event}}$, $W_{\text{event}}$, $T_{\text{event}}$, $H_{\text{image}}$, $W_{\text{image}}$, and $T_{\text{image}}$ denote the height, width, and time of the event-based camera image, and the RGB camera image, respectively. We compute the temporal and spatial projection from the event-based camera to the RGB camera in three steps:

1. Temporal synchronization: We aim to find the event sequence $\mathbf{E}(t_{j})$ that aligns temporally with $\mathbf{I}(t_{i})$. Since the frame rate of $\mathbf{I}$ is 120 FPS, we first calculate a sequence of accumulated events, denoted by $\mathbf{A()}$, at the same frame rate, and proceed to a visual inspection to align both sequences. This produces a first estimate of $t_{i}$ and $t_{j}$.

We then reduce the temporal shift $\Delta t_{j\rightarrow i}=t_{j}-t_{i}$ as follows. As the greatest common divisor between the sampling rates of $\mathbf{I}()$ and $\mathbf{E}()$ is 40, we split the $\mathbf{I}()$ original sequence into three interleaved sub-sequences $\mathbf{Seq}$ at 40 FPS. Likewise, we generate 250 sequences of accumulated events over a period of 25 ms, starting at $t_{j}+k\times 100\mu s$ with $k\in\{0,\,...,\,249\}$. Finally, we compute the mean structural similarity index measure (SSIM, see [48]) between the grayscale differences of successive frames of Seq and $\mathbf{A}(t_{j}+k\times 100\mu s)$ over 10 frames for all values of $k$, and select the sequence and $k$ value that maximize the SSIM. An illustration of the evolution of the SSIM with respect to $k$ is shown in Figure 3. In that example, we chose Seq 3 and $k$ around 220. This calculation is repeated for each video snapshot, except that we use a previous estimate to skip further visual inspection steps.

Figure 3: SSIM values for different time shifts calculated for three interleaved sequences at a frame rate of 40 FPS.

2. Spatial shift estimation: Once synchronized temporally, we identify common features between the two images, such as the borders of sports equipment or the athletes. Let $(x_{j}^{(1)},y_{j}^{(1)})$ denote a first feature location in the event-based camera, and $(x_{i}^{(1)},y_{i}^{(1)})$ represent the corresponding feature location in the RGB camera. We calculate the horizontal and vertical shift as $\Delta_{x}=x_{j}^{(1)}-x_{i}^{(1)}$ and $\Delta_{y}=y_{j}^{(1)}-y_{i}^{(1)}$, respectively.

Original 120 FPS

40$\rightarrow$ 120 FPS

20$\rightarrow$ 120 FPS

10$\rightarrow$ 120 FPS

Figure 4: Qualitative results obtained by Timelens with different input frame rates. The first line shows the original $120$ FPS video, the second the $40$ FPS with two intermediate frame interpolated, the third line a $20$ FPS subsampled with $5$ out of $6$ frames interpolated, and finally a $10$ FPS input video with $11$ frames interpolated (only $6$ shown). It can be seen that the tennis ball is well-placed on all interpolated frames. However, for the racquet, the fast movement is only well interpolated with an initial frame rate of $20$ and $40$ while $10$ FPS does not provide satisfactory results. Original frames are shown in red, while interpolated ones are shown in green.

3. Scale estimation: Similarly, let $(x_{j}^{(2)},x_{j}^{(2)})$ and $(x_{i}^{(2)},y_{i}^{(2)})$ denote another set of feature locations, we compute the scale ratio between the two images based on the distance between the matched features. Let $\Delta_{\text{event}}$ and $\Delta_{\text{image}}$ denote the Euclidean distances between the matched features in the event-based and RGB cameras, respectively. The scale ratio $r$ is given by

$$r=\frac{\Delta_{\text{image}}}{\Delta_{\text{event}}}=\frac{\sqrt{(x_{i}^{(2)}-x_{i}^{(1)})^{2}+(y_{i}^{(2)}-y_{i}^{(1)})^{2}}}{\sqrt{(x_{j}^{(2)}-x_{j}^{(1)})^{2}+(y_{j}^{(2)}-y_{j}^{(1)})^{2}}}\,.$$

The overall projection function then becomes

$$\mathbf{P}=\begin{cases}x_{i}=r\times(x_{j}-\Delta_{x})\\ y_{j}=r\times(y_{j}-\Delta_{y})\\ t_{i}=t_{j}-\Delta_{j\rightarrow i}\,,\end{cases}$$ (2)

which enables the projection of pixel coordinates from the event-based camera to the corresponding locations on the RGB camera, facilitating the generation of intermediate frames for event-based video frame interpolation.

IV Experiments

IV-A Experimental setup

In this section, we provide a detailed overview of the experimental setup used to generate intermediate frames on our captured data footage to produce slow-motion content. We focus on two main aspects: the initial temporal resolution and upscaling factor, and the summary of the literature method Timelens [1] that we used in our experiments.

Temporal resolution and upscaling. To comprehensively evaluate the performance of our method, we explore various combinations of initial temporal resolution and upscaling factors. Given access to a $120$ FPS camera, we conduct experiments with different subsampling rates, including $10$, $20$, $40$, and the original $120$ FPS. Additionally, we test different upscaling factors applied by the event-based VFI method, ranging from $\times 3$, $\times 6$, $\times 10$, $\times 12$, and $\times 25$. By examining a wide range of temporal resolutions and upscaling factors, we aim to assess the robustness and scalability of the event-based VFI method across different input configurations.

Timelens. Timelens [1] leverages the complementary strengths of synthesis-based and war**-based interpolation techniques. The method consists of four modules: war**-based interpolation, war** refinement, interpolation by synthesis, and attention-based averaging. Each module serves a specific purpose, such as estimating optical flow, refining interpolated frames, and blending synthesis-based and war**-based results. The method utilizes a voxel grid representation for encoding event sequences and a backbone architecture based on an hourglass network with skip connections. Interpolation by synthesis directly regresses new frames given input RGB key frames and event sequences, allowing it to handle changes in lighting. However, it may distort image edges and textures in the presence of noisy or insufficient event data. War**-based interpolation estimates optical flow from events to warp boundary key frames, making it suitable for handling blur and non-linear motion. The war** refinement module computes refined interpolated frames by estimating residual optical flow between war**-based and synthesis-based results. Finally, the attention averaging module blends synthesis-based and war**-based results to achieve a final interpolation result. We use the open-source code of Timelens with default parameters for generating our sequences.

IV-B Qualitative results

Qualitative results of the interpolated frames are provided in Figure 4 for all four input frame rates brought up to $120$ FPS. It can be seen that the interpolation process effectively tracks the trajectory of the tennis ball across all interpolated frames, showcasing Timelens’ ability in maintaining object consistency. However, the racquet’s swift motion demonstrates uneven performance at the initial frame rate. Notably, the $20$ FPS and $40$ FPS inputs exhibit more faithful interpolation of the racquet’s trajectory compared to the $10$ FPS input, where evident artifacts emerge. We supplement these observations with additional original frame rates and scale factors, provided in a side-by-side video format for enhanced visualization, in supplementary material . Despite the promising outcomes, residual artifacts persist, suggesting avenues for improvement in future iterations of the methodology.

IV-C Future work

In future work, we plan to explore several avenues for enhancing the method and extend the scope of our research:

Optical Alignment Improvement. We noticed that misaligned events generate artifacts in the interpolated stream. Therefore, we intend to implement a beam splitter setup to achieve better alignment of the optical centers between the RGB camera and the event-based camera.

Performance enhancement. We also aim to investigate techniques to further improve the performance of Timelens, particularly when applied to sports-centric videos captured by event-based cameras. This may involve incorporating domain-specific features to enhance the accuracy and realism of the generated slow-motion footage.

Exploration of downstream analysis techniques. Beyond video frame interpolation, we plan to explore downstream analysis techniques enabled by the generated slow-motion footage. One potential avenue is player movement analysis, where we can leverage the detailed temporal information provided by the slow-motion videos to gain insights into player movement and performance.

V Conclusion

In conclusion, this project contributes to the advancement of sports video understanding through the analysis of the potential of event-based video frame interpolation. Our primary contributions lie in the recording of sports data, including the alignment of the RGB and event-based camera temporally and spatially. Leveraging these recorded sport sequences, we conducted tests of Timelens, an already established event-based video frame interpolation method. Through qualitative analysis, we have provided insights into the performance of Timelens in generating slow-motion footage on our dataset, showcasing its applicability and effectiveness in the context of sports videos captured in dynamic and uncontrolled environments. Moving forward, further research in this domain could explore better physical setups, refinements to existing methods with the development of specialized algorithms tailored to the unique characteristics of sports footage, ultimately enhancing the quality and utility of sports video analysis tools.