-
Enhancing CTC-based speech recognition with diverse modeling units
Authors:
Shiyi Han,
Zhihong Lei,
Mingbin Xu,
Xingyu Na,
Zhen Huang
Abstract:
In recent years, the evolution of end-to-end (E2E) automatic speech recognition (ASR) models has been remarkable, largely due to advances in deep learning architectures like transformer. On top of E2E systems, researchers have achieved substantial accuracy improvement by rescoring E2E model's N-best hypotheses with a phoneme-based model. This raises an interesting question about where the improvem…
▽ More
In recent years, the evolution of end-to-end (E2E) automatic speech recognition (ASR) models has been remarkable, largely due to advances in deep learning architectures like transformer. On top of E2E systems, researchers have achieved substantial accuracy improvement by rescoring E2E model's N-best hypotheses with a phoneme-based model. This raises an interesting question about where the improvements come from other than the system combination effect. We examine the underlying mechanisms driving these gains and propose an efficient joint training approach, where E2E models are trained jointly with diverse modeling units. This methodology does not only align the strengths of both phoneme and grapheme-based models but also reveals that using these diverse modeling units in a synergistic way can significantly enhance model accuracy. Our findings offer new insights into the optimal integration of heterogeneous modeling units in the development of more robust and accurate ASR systems.
△ Less
Submitted 11 June, 2024; v1 submitted 5 June, 2024;
originally announced June 2024.
-
Real-Time 4K Super-Resolution of Compressed AVIF Images. AIS 2024 Challenge Survey
Authors:
Marcos V. Conde,
Zhijun Lei,
Wen Li,
Cosmin Stejerean,
Ioannis Katsavounidis,
Radu Timofte,
Kihwan Yoon,
Ganzorig Gankhuyag,
Jiangtao Lv,
Long Sun,
**shan Pan,
Jiangxin Dong,
**hui Tang,
Zhiyuan Li,
Hao Wei,
Chenyang Ge,
Dongyang Zhang,
Tianle Liu,
Huaian Chen,
Yi **,
Menghan Zhou,
Yiqiang Yan,
Si Gao,
Biao Wu,
Shaoli Liu
, et al. (50 additional authors not shown)
Abstract:
This paper introduces a novel benchmark as part of the AIS 2024 Real-Time Image Super-Resolution (RTSR) Challenge, which aims to upscale compressed images from 540p to 4K resolution (4x factor) in real-time on commercial GPUs. For this, we use a diverse test set containing a variety of 4K images ranging from digital art to gaming and photography. The images are compressed using the modern AVIF cod…
▽ More
This paper introduces a novel benchmark as part of the AIS 2024 Real-Time Image Super-Resolution (RTSR) Challenge, which aims to upscale compressed images from 540p to 4K resolution (4x factor) in real-time on commercial GPUs. For this, we use a diverse test set containing a variety of 4K images ranging from digital art to gaming and photography. The images are compressed using the modern AVIF codec, instead of JPEG. All the proposed methods improve PSNR fidelity over Lanczos interpolation, and process images under 10ms. Out of the 160 participants, 25 teams submitted their code and models. The solutions present novel designs tailored for memory-efficiency and runtime on edge devices. This survey describes the best solutions for real-time SR of compressed high-resolution images.
△ Less
Submitted 25 April, 2024;
originally announced April 2024.
-
FusionMamba: Efficient Image Fusion with State Space Model
Authors:
Siran Peng,
Xiangyu Zhu,
Haoyu Deng,
Zhen Lei,
Liang-Jian Deng
Abstract:
Image fusion aims to generate a high-resolution multi/hyper-spectral image by combining a high-resolution image with limited spectral information and a low-resolution image with abundant spectral data. Current deep learning (DL)-based methods for image fusion primarily rely on CNNs or Transformers to extract features and merge different types of data. While CNNs are efficient, their receptive fiel…
▽ More
Image fusion aims to generate a high-resolution multi/hyper-spectral image by combining a high-resolution image with limited spectral information and a low-resolution image with abundant spectral data. Current deep learning (DL)-based methods for image fusion primarily rely on CNNs or Transformers to extract features and merge different types of data. While CNNs are efficient, their receptive fields are limited, restricting their capacity to capture global context. Conversely, Transformers excel at learning global information but are hindered by their quadratic complexity. Fortunately, recent advancements in the State Space Model (SSM), particularly Mamba, offer a promising solution to this issue by enabling global awareness with linear complexity. However, there have been few attempts to explore the potential of the SSM in information fusion, which is a crucial ability in domains like image fusion. Therefore, we propose FusionMamba, an innovative method for efficient image fusion. Our contributions mainly focus on two aspects. Firstly, recognizing that images from different sources possess distinct properties, we incorporate Mamba blocks into two U-shaped networks, presenting a novel architecture that extracts spatial and spectral features in an efficient, independent, and hierarchical manner. Secondly, to effectively combine spatial and spectral information, we extend the Mamba block to accommodate dual inputs. This expansion leads to the creation of a new module called the FusionMamba block, which outperforms existing fusion techniques such as concatenation and cross-attention. We conduct a series of experiments on five datasets related to three image fusion tasks. The quantitative and qualitative evaluation results demonstrate that our method achieves SOTA performance, underscoring the superiority of FusionMamba. The code is available at https://github.com/PSRben/FusionMamba.
△ Less
Submitted 10 May, 2024; v1 submitted 11 April, 2024;
originally announced April 2024.
-
Encoding Time and Energy Model for SVT-AV1 based on Video Complexity
Authors:
Lena Eichermüller,
Gaurang Chaudhari,
Ioannis Katsavounidis,
Zhijun Lei,
Hassene Tmar,
Christian Herglotz,
André Kaup
Abstract:
The share of online video traffic in global carbon dioxide emissions is growing steadily. To comply with the demand for video media, dedicated compression techniques are continuously optimized, but at the expense of increasingly higher computational demands and thus rising energy consumption at the video encoder side. In order to find the best trade-off between compression and energy consumption,…
▽ More
The share of online video traffic in global carbon dioxide emissions is growing steadily. To comply with the demand for video media, dedicated compression techniques are continuously optimized, but at the expense of increasingly higher computational demands and thus rising energy consumption at the video encoder side. In order to find the best trade-off between compression and energy consumption, modeling encoding energy for a wide range of encoding parameters is crucial. We propose an encoding time and energy model for SVT-AV1 based on empirical relations between the encoding time and video parameters as well as encoder configurations. Furthermore, we model the influence of video content by established content descriptors such as spatial and temporal information. We then use the predicted encoding time to estimate the required energy demand and achieve a prediction error of 19.6 % for encoding time and 20.9 % for encoding energy.
△ Less
Submitted 30 January, 2024; v1 submitted 29 January, 2024;
originally announced January 2024.
-
Acoustic Model Fusion for End-to-end Speech Recognition
Authors:
Zhihong Lei,
Mingbin Xu,
Shiyi Han,
Leo Liu,
Zhen Huang,
Tim Ng,
Yuanyuan Zhang,
Ernest Pusateri,
Mirko Hannemann,
Yaqiao Deng,
Man-Hung Siu
Abstract:
Recent advances in deep learning and automatic speech recognition (ASR) have enabled the end-to-end (E2E) ASR system and boosted the accuracy to a new level. The E2E systems implicitly model all conventional ASR components, such as the acoustic model (AM) and the language model (LM), in a single network trained on audio-text pairs. Despite this simpler system architecture, fusing a separate LM, tr…
▽ More
Recent advances in deep learning and automatic speech recognition (ASR) have enabled the end-to-end (E2E) ASR system and boosted the accuracy to a new level. The E2E systems implicitly model all conventional ASR components, such as the acoustic model (AM) and the language model (LM), in a single network trained on audio-text pairs. Despite this simpler system architecture, fusing a separate LM, trained exclusively on text corpora, into the E2E system has proven to be beneficial. However, the application of LM fusion presents certain drawbacks, such as its inability to address the domain mismatch issue inherent to the internal AM. Drawing inspiration from the concept of LM fusion, we propose the integration of an external AM into the E2E system to better address the domain mismatch. By implementing this novel approach, we have achieved a significant reduction in the word error rate, with an impressive drop of up to 14.3% across varied test sets. We also discovered that this AM fusion approach is particularly beneficial in enhancing named entity recognition.
△ Less
Submitted 10 October, 2023;
originally announced October 2023.
-
Weakly Supervised YOLO Network for Surgical Instrument Localization in Endoscopic Videos
Authors:
Rongfeng Wei,
**lin Wu,
Xuexue Bai,
Ming Feng,
Zhen Lei,
Hongbin Liu,
Zhen Chen
Abstract:
In minimally invasive surgery, surgical instrument localization is a crucial task for endoscopic videos, which enables various applications for improving surgical outcomes. However, annotating the instrument localization in endoscopic videos is tedious and labor-intensive. In contrast, obtaining the category information is easy and efficient in real-world applications. To fully utilize the categor…
▽ More
In minimally invasive surgery, surgical instrument localization is a crucial task for endoscopic videos, which enables various applications for improving surgical outcomes. However, annotating the instrument localization in endoscopic videos is tedious and labor-intensive. In contrast, obtaining the category information is easy and efficient in real-world applications. To fully utilize the category information and address the localization problem, we propose a weakly supervised localization framework named WS-YOLO for surgical instruments. By leveraging the instrument category information as the weak supervision, our WS-YOLO framework adopts an unsupervised multi-round training strategy for the localization capability training. We validate our WS-YOLO framework on the Endoscopic Vision Challenge 2023 dataset, which achieves remarkable performance in the weakly supervised surgical instrument localization. The source code is available at https://github.com/Breezewrf/WS-YOLO.
△ Less
Submitted 20 June, 2024; v1 submitted 23 September, 2023;
originally announced September 2023.
-
Encoder Complexity Control in SVT-AV1 by Speed-Adaptive Preset Switching
Authors:
Lena Eichermüller,
Gaurang Chaudhari,
Ioannis Katsavounidis,
Zhijun Lei,
Hassene Tmar,
André Kaup,
Christian Herglotz
Abstract:
Current developments in video encoding technology lead to continuously improving compression performance but at the expense of increasingly higher computational demands. Regarding the online video traffic increases during the last years and the concomitant need for video encoding, encoder complexity control mechanisms are required to restrict the processing time to a sufficient extent in order to…
▽ More
Current developments in video encoding technology lead to continuously improving compression performance but at the expense of increasingly higher computational demands. Regarding the online video traffic increases during the last years and the concomitant need for video encoding, encoder complexity control mechanisms are required to restrict the processing time to a sufficient extent in order to find a reasonable trade-off between performance and complexity. We present a complexity control mechanism in SVT-AV1 by using speed-adaptive preset switching to comply with the remaining time budget. This method enables encoding with a user-defined time constraint within the complete preset range with an average precision of 8.9 \% without introducing any additional latencies.
△ Less
Submitted 11 July, 2023;
originally announced July 2023.
-
SAR-NeRF: Neural Radiance Fields for Synthetic Aperture Radar Multi-View Representation
Authors:
Zhengxin Lei,
Feng Xu,
Jiangtao Wei,
Feng Cai,
Feng Wang,
Ya-Qiu **
Abstract:
SAR images are highly sensitive to observation configurations, and they exhibit significant variations across different viewing angles, making it challenging to represent and learn their anisotropic features. As a result, deep learning methods often generalize poorly across different view angles. Inspired by the concept of neural radiance fields (NeRF), this study combines SAR imaging mechanisms w…
▽ More
SAR images are highly sensitive to observation configurations, and they exhibit significant variations across different viewing angles, making it challenging to represent and learn their anisotropic features. As a result, deep learning methods often generalize poorly across different view angles. Inspired by the concept of neural radiance fields (NeRF), this study combines SAR imaging mechanisms with neural networks to propose a novel NeRF model for SAR image generation. Following the map** and projection pinciples, a set of SAR images is modeled implicitly as a function of attenuation coefficients and scattering intensities in the 3D imaging space through a differentiable rendering equation. SAR-NeRF is then constructed to learn the distribution of attenuation coefficients and scattering intensities of voxels, where the vectorized form of 3D voxel SAR rendering equation and the sampling relationship between the 3D space voxels and the 2D view ray grids are analytically derived. Through quantitative experiments on various datasets, we thoroughly assess the multi-view representation and generalization capabilities of SAR-NeRF. Additionally, it is found that SAR-NeRF augumented dataset can significantly improve SAR target classification performance under few-shot learning setup, where a 10-type classification accuracy of 91.6\% can be achieved by using only 12 images per class.
△ Less
Submitted 11 July, 2023;
originally announced July 2023.
-
On Robust Observer Design for System Motion on SE(3) Using Onboard Visual Sensors
Authors:
Tong Zhang,
Ying Tan,
Xiang Chen,
Zike Lei
Abstract:
Onboard visual sensing has been widely used in the unmanned ground vehicle (UGV) and/or unmanned aerial vehicle (UAV), which can be modeled as dynamic systems on SE(3). The onboard sensing outputs of the dynamic system can usually be applied to derive the relative position between the feature marks and the system, but bearing with explicit geometrical constraint. Such a visual geometrical constrai…
▽ More
Onboard visual sensing has been widely used in the unmanned ground vehicle (UGV) and/or unmanned aerial vehicle (UAV), which can be modeled as dynamic systems on SE(3). The onboard sensing outputs of the dynamic system can usually be applied to derive the relative position between the feature marks and the system, but bearing with explicit geometrical constraint. Such a visual geometrical constraint makes the design of the visual observer on SE(3) very challenging, as it will cause a time-varying or switching visible set due to the varying number of feature marks in this set along different trajectories. Moreover, the possibility of having mis-identified feature marks and modeling uncertainties might result in a divergent estimation error. This paper proposes a new robust observer design method that can accommodate these uncertainties from onboard visual sensing. The key design idea for this observer is to estimate the visible set and identify the mis-identified features from the measurements. Based on the identified uncertainties, a switching strategy is proposed to ensure bounded estimation error for any given trajectory over a fixed time interval. Simulation results are provided to demonstrate the effectiveness of the proposed robust observer.
△ Less
Submitted 21 March, 2023; v1 submitted 29 November, 2022;
originally announced November 2022.
-
Optimization of Directional Landmark Deployment for Visual Observer on SE(3)
Authors:
Zike Lei,
Xi Chen,
Ying Tan,
Xiang Chen,
Li Chai
Abstract:
An optimization method is proposed in this paper for novel deployment of given number of directional landmarks (location and pose) within a given region in the 3-D task space. This new deployment technique is built on the geometric models of both landmarks and the monocular camera. In particular, a new concept of Multiple Coverage Probability (MCP) is defined to characterize the probability of at…
▽ More
An optimization method is proposed in this paper for novel deployment of given number of directional landmarks (location and pose) within a given region in the 3-D task space. This new deployment technique is built on the geometric models of both landmarks and the monocular camera. In particular, a new concept of Multiple Coverage Probability (MCP) is defined to characterize the probability of at least n landmarks being covered simultaneously by a camera at a fixed position. The optimization is conducted with respect to the position and pose of the given number of landmarks to maximize MCP through globally exploration of the given 3-D space. By adopting the elimination genetic algorithm, the global optimal solutions can be obtained, which are then applied to improve the convergent performance of the visual observer on SE(3) as a demonstration example. Both simulation and experimental results are presented to validate the effectiveness of the proposed landmark deployment optimization method.
△ Less
Submitted 28 March, 2022;
originally announced March 2022.
-
User Based Design and Evaluation Pipeline for Indoor Airships
Authors:
Zhaoliang Zheng,
Jiahao Li,
Parth Agrawal,
Zhao Lei,
Aaron John-Sabu,
Ankur Mehta
Abstract:
Designing a controllable airship for non-expert users or preemptively evaluating the performance of desired airships has always been a very challenging problem. This paper explores the blimp design parameter space from the aspect of the user by considering various distributions of thrust, combinations of propulsive mechanisms, and balloon shapes. We provide open-source modular hardware and reconfi…
▽ More
Designing a controllable airship for non-expert users or preemptively evaluating the performance of desired airships has always been a very challenging problem. This paper explores the blimp design parameter space from the aspect of the user by considering various distributions of thrust, combinations of propulsive mechanisms, and balloon shapes. We provide open-source modular hardware and reconfigurable software design tools that allow inexperienced users to design a custom airship in a short time. Based on these design parameters, this paper develops a more engineering-focused evaluation system that can characterize the performance of different indoor blimps. An analytical comparison and some case studies that consider various points in the design parameter space have been conducted to prove the feasibility and validity of our design and evaluation system.
△ Less
Submitted 23 November, 2021; v1 submitted 19 October, 2021;
originally announced October 2021.
-
Learning Friction Model for Magnet-actuated Tethered Capsule Robot
Authors:
Yi Wang,
Yuyang Tu,
Yuchen He,
Xutian Deng,
Ziwei Lei,
Jianwei Zhang,
Miao Li
Abstract:
The potential diagnostic applications of magnet-actuated capsules have been greatly increased in recent years. For most of these potential applications, accurate position control of the capsule have been highly demanding. However, the friction between the robot and the environment as well as the drag force from the tether play a significant role during the motion control of the capsule. Moreover,…
▽ More
The potential diagnostic applications of magnet-actuated capsules have been greatly increased in recent years. For most of these potential applications, accurate position control of the capsule have been highly demanding. However, the friction between the robot and the environment as well as the drag force from the tether play a significant role during the motion control of the capsule. Moreover, these forces especially the friction force are typically hard to model beforehand. In this paper, we first designed a magnet-actuated tethered capsule robot, where the driving magnet is mounted on the end of a robotic arm. Then, we proposed a learning-based approach to model the friction force between the capsule and the environment, with the goal of increasing the control accuracy of the whole system. Finally, several real robot experiments are demonstrated to showcase the effectiveness of our proposed approach.
△ Less
Submitted 1 October, 2021; v1 submitted 15 September, 2021;
originally announced September 2021.
-
Fast Camera Image Denoising on Mobile GPUs with Deep Learning, Mobile AI 2021 Challenge: Report
Authors:
Andrey Ignatov,
Kim Byeoung-su,
Radu Timofte,
Angeline Pouget,
Fenglong Song,
Cheng Li,
Shuai Xiao,
Zhongqian Fu,
Matteo Maggioni,
Yibin Huang,
Shen Cheng,
Xin Lu,
Yifeng Zhou,
Liangyu Chen,
Donghao Liu,
Xiangyu Zhang,
Haoqiang Fan,
Jian Sun,
Shuaicheng Liu,
Minsu Kwon,
Myungje Lee,
Jaeyoon Yoo,
Changbeom Kang,
Shinjo Wang,
Bin Huang
, et al. (7 additional authors not shown)
Abstract:
Image denoising is one of the most critical problems in mobile photo processing. While many solutions have been proposed for this task, they are usually working with synthetic data and are too computationally expensive to run on mobile devices. To address this problem, we introduce the first Mobile AI challenge, where the target is to develop an end-to-end deep learning-based image denoising solut…
▽ More
Image denoising is one of the most critical problems in mobile photo processing. While many solutions have been proposed for this task, they are usually working with synthetic data and are too computationally expensive to run on mobile devices. To address this problem, we introduce the first Mobile AI challenge, where the target is to develop an end-to-end deep learning-based image denoising solution that can demonstrate high efficiency on smartphone GPUs. For this, the participants were provided with a novel large-scale dataset consisting of noisy-clean image pairs captured in the wild. The runtime of all models was evaluated on the Samsung Exynos 2100 chipset with a powerful Mali GPU capable of accelerating floating-point and quantized neural networks. The proposed solutions are fully compatible with any mobile GPU and are capable of processing 480p resolution images under 40-80 ms while achieving high fidelity results. A detailed description of all models developed in the challenge is provided in this paper.
△ Less
Submitted 17 May, 2021;
originally announced May 2021.
-
Deep Controllable Backlight Dimming
Authors:
Lvyin Duan,
Demetris Marnerides,
Alan Chalmers,
Zhichun Lei,
Kurt Debattista
Abstract:
Dual-panel displays require local dimming algorithms in order to reproduce content with high fidelity and high dynamic range. In this work, a novel deep learning based local dimming method is proposed for rendering HDR images on dual-panel HDR displays. The method uses a Convolutional Neural Network to predict backlight values, using as input the HDR image that is to be displayed. The model is des…
▽ More
Dual-panel displays require local dimming algorithms in order to reproduce content with high fidelity and high dynamic range. In this work, a novel deep learning based local dimming method is proposed for rendering HDR images on dual-panel HDR displays. The method uses a Convolutional Neural Network to predict backlight values, using as input the HDR image that is to be displayed. The model is designed and trained via a controllable power parameter that allows a user to trade off between power and quality. The proposed method is evaluated against six other methods on a test set of 105 HDR images, using a variety of quantitative quality metrics. Results demonstrate improved display quality and better power consumption when using the proposed method compared to the best alternatives.
△ Less
Submitted 19 August, 2020;
originally announced August 2020.
-
Multi-Site Infant Brain Segmentation Algorithms: The iSeg-2019 Challenge
Authors:
Yue Sun,
Kun Gao,
Zhengwang Wu,
Zhihao Lei,
Ying Wei,
Jun Ma,
** Yang,
Xue Feng,
Li Zhao,
Trung Le Phan,
Jitae Shin,
Tao Zhong,
Yu Zhang,
Lequan Yu,
Caizi Li,
Ramesh Basnet,
M. Omair Ahmad,
M. N. S. Swamy,
Wenao Ma,
Qi Dou,
Toan Duc Bui,
Camilo Bermudez Noguera,
Bennett Landman,
Ian H. Gotlib,
Kathryn L. Humphreys
, et al. (8 additional authors not shown)
Abstract:
To better understand early brain growth patterns in health and disorder, it is critical to accurately segment infant brain magnetic resonance (MR) images into white matter (WM), gray matter (GM), and cerebrospinal fluid (CSF). Deep learning-based methods have achieved state-of-the-art performance; however, one of major limitations is that the learning-based methods may suffer from the multi-site i…
▽ More
To better understand early brain growth patterns in health and disorder, it is critical to accurately segment infant brain magnetic resonance (MR) images into white matter (WM), gray matter (GM), and cerebrospinal fluid (CSF). Deep learning-based methods have achieved state-of-the-art performance; however, one of major limitations is that the learning-based methods may suffer from the multi-site issue, that is, the models trained on a dataset from one site may not be applicable to the datasets acquired from other sites with different imaging protocols/scanners. To promote methodological development in the community, iSeg-2019 challenge (http://iseg2019.web.unc.edu) provides a set of 6-month infant subjects from multiple sites with different protocols/scanners for the participating methods. Training/validation subjects are from UNC (MAP) and testing subjects are from UNC/UMN (BCP), Stanford University, and Emory University. By the time of writing, there are 30 automatic segmentation methods participating in iSeg-2019. We review the 8 top-ranked teams by detailing their pipelines/implementations, presenting experimental results and evaluating performance in terms of the whole brain, regions of interest, and gyral landmark curves. We also discuss their limitations and possible future directions for the multi-site issue. We hope that the multi-site dataset in iSeg-2019 and this review article will attract more researchers on the multi-site issue.
△ Less
Submitted 11 July, 2020; v1 submitted 4 July, 2020;
originally announced July 2020.
-
Radial Coverage Strength for Optimization of Multi-Camera Deployment
Authors:
Zike Lei,
Xi Chen,
Xiang Chen,
Li Chai
Abstract:
In this paper, a new concept, radial coverage strength, is first proposed to characterize the visual sensing performance when the orientation of the target pose is considered. In particular, the elevation angle of the optical pose of the visual sensor is taken to decompose the visual coverage strength into effective and ineffective components, motivated by the imaging intuition. An optimization pr…
▽ More
In this paper, a new concept, radial coverage strength, is first proposed to characterize the visual sensing performance when the orientation of the target pose is considered. In particular, the elevation angle of the optical pose of the visual sensor is taken to decompose the visual coverage strength into effective and ineffective components, motivated by the imaging intuition. An optimization problem is then formulated for a multi-camera network to maximize the coverage of the object area based on the strength information fusion along the effective coverage strength direction through the deployment of the angle between radial coverage vector of the camera optical pose. Both simulation and experiments are conducted to validate the proposed approach and comparison with existing methods is also provided.
△ Less
Submitted 1 April, 2020;
originally announced April 2020.
-
Eye in the Sky: Drone-Based Object Tracking and 3D Localization
Authors:
Haotian Zhang,
Gaoang Wang,
Zhichao Lei,
Jenq-Neng Hwang
Abstract:
Drones, or general UAVs, equipped with a single camera have been widely deployed to a broad range of applications, such as aerial photography, fast goods delivery and most importantly, surveillance. Despite the great progress achieved in computer vision algorithms, these algorithms are not usually optimized for dealing with images or video sequences acquired by drones, due to various challenges su…
▽ More
Drones, or general UAVs, equipped with a single camera have been widely deployed to a broad range of applications, such as aerial photography, fast goods delivery and most importantly, surveillance. Despite the great progress achieved in computer vision algorithms, these algorithms are not usually optimized for dealing with images or video sequences acquired by drones, due to various challenges such as occlusion, fast camera motion and pose variation. In this paper, a drone-based multi-object tracking and 3D localization scheme is proposed based on the deep learning based object detection. We first combine a multi-object tracking method called TrackletNet Tracker (TNT) which utilizes temporal and appearance information to track detected objects located on the ground for UAV applications. Then, we are also able to localize the tracked ground objects based on the group plane estimated from the Multi-View Stereo technique. The system deployed on the drone can not only detect and track the objects in a scene, but can also localize their 3D coordinates in meters with respect to the drone camera. The experiments have proved our tracker can reliably handle most of the detected objects captured by drones and achieve favorable 3D localization performance when compared with the state-of-the-art methods.
△ Less
Submitted 18 October, 2019;
originally announced October 2019.
