-
ALSTER: A Local Spatio-Temporal Expert for Online 3D Semantic Reconstruction
Authors:
Silvan Weder,
Francis Engelmann,
Johannes L. Schönberger,
Akihito Seki,
Marc Pollefeys,
Martin R. Oswald
Abstract:
We propose an online 3D semantic segmentation method that incrementally reconstructs a 3D semantic map from a stream of RGB-D frames. Unlike offline methods, ours is directly applicable to scenarios with real-time constraints, such as robotics or mixed reality. To overcome the inherent challenges of online methods, we make two main contributions. First, to effectively extract information from the…
▽ More
We propose an online 3D semantic segmentation method that incrementally reconstructs a 3D semantic map from a stream of RGB-D frames. Unlike offline methods, ours is directly applicable to scenarios with real-time constraints, such as robotics or mixed reality. To overcome the inherent challenges of online methods, we make two main contributions. First, to effectively extract information from the input RGB-D video stream, we jointly estimate geometry and semantic labels per frame in 3D. A key focus of our approach is to reason about semantic entities both in the 2D input and the local 3D domain to leverage differences in spatial context and network architectures. Our method predicts 2D features using an off-the-shelf segmentation network. The extracted 2D features are refined by a lightweight 3D network to enable reasoning about the local 3D structure. Second, to efficiently deal with an infinite stream of input RGB-D frames, a subsequent network serves as a temporal expert predicting the incremental scene updates by leveraging 2D, 3D, and past information in a learned manner. These updates are then integrated into a global scene representation. Using these main contributions, our method can enable scenarios with real-time constraints and can scale to arbitrary scene sizes by processing and updating the scene only in a local region defined by the new measurement. Our experiments demonstrate improved results compared to existing online methods that purely operate in local regions and show that complementary sources of information can boost the performance. We provide a thorough ablation study on the benefits of different architectural as well as algorithmic design decisions. Our method yields competitive results on the popular ScanNet benchmark and SceneNN dataset.
△ Less
Submitted 3 December, 2023; v1 submitted 29 November, 2023;
originally announced November 2023.
-
Quad-networks: unsupervised learning to rank for interest point detection
Authors:
Nikolay Savinov,
Akihito Seki,
Lubor Ladicky,
Torsten Sattler,
Marc Pollefeys
Abstract:
Several machine learning tasks require to represent the data using only a sparse set of interest points. An ideal detector is able to find the corresponding interest points even if the data undergo a transformation typical for a given domain. Since the task is of high practical interest in computer vision, many hand-crafted solutions were proposed. In this paper, we ask a fundamental question: can…
▽ More
Several machine learning tasks require to represent the data using only a sparse set of interest points. An ideal detector is able to find the corresponding interest points even if the data undergo a transformation typical for a given domain. Since the task is of high practical interest in computer vision, many hand-crafted solutions were proposed. In this paper, we ask a fundamental question: can we learn such detectors from scratch? Since it is often unclear what points are "interesting", human labelling cannot be used to find a truly unbiased solution. Therefore, the task requires an unsupervised formulation. We are the first to propose such a formulation: training a neural network to rank points in a transformation-invariant manner. Interest points are then extracted from the top/bottom quantiles of this ranking. We validate our approach on two tasks: standard RGB image interest point detection and challenging cross-modal interest point detection between RGB and depth images. We quantitatively show that our unsupervised method performs better or on-par with baselines.
△ Less
Submitted 10 April, 2017; v1 submitted 22 November, 2016;
originally announced November 2016.
-
CANGAROO-III search for TeV Gamma-rays from two clusters of galaxies
Authors:
R. Kiuchi,
M. Mori,
G. V. Bicknell,
R. W. Clay,
P. G. Edwards,
R. Enomoto,
S. Gunji,
S. Hara,
T. Hara,
T. Hattori,
S. Hayashi,
Y. Higashi,
Y. Hirai,
K. Inoue,
C. Itoh,
S. Kabuki,
F. Ka**o,
H. Katagiri,
A. Kawachi,
T. Kifune,
H. Kubo,
J. Kushida,
Y. Matsubara,
T. Mizukami,
Y. Mizumoto
, et al. (25 additional authors not shown)
Abstract:
Because accretion and merger shocks in clusters of galaxies may accelerate particles to high energies, clusters are candidate sites for the origin of ultra-high-energy (UHE) cosmic-rays. A prediction was presented for gamma-ray emission from a cluster of galaxies at a detectable level with the current generation of imaging atmospheric Cherenkov telescopes. The gamma-ray emission was produced via…
▽ More
Because accretion and merger shocks in clusters of galaxies may accelerate particles to high energies, clusters are candidate sites for the origin of ultra-high-energy (UHE) cosmic-rays. A prediction was presented for gamma-ray emission from a cluster of galaxies at a detectable level with the current generation of imaging atmospheric Cherenkov telescopes. The gamma-ray emission was produced via inverse Compton upscattering of cosmic microwave background (CMB) photons by electron-positron pairs generated by collisions of UHE cosmic rays in the cluster. We observed two clusters of galaxies, Abell 3667 and Abell 4038, searching for very-high-energy gamma-ray emission with the CANGAROO-III atmospheric Cherenkov telescope system in 2006. The analysis showed no significant excess around these clusters, yielding upper limits on the gamma-ray emission. From a comparison of the upper limit for the north-west radio relic region of Abell 3667 with a model prediction, we derive a lower limit for the magnetic field of the region of ~0.1 micro G. This shows the potential of gamma-ray observations in studies of the cluster environment. We also discuss the flux upper limit from cluster center regions using a model of gamma-ray emission from neutral pions produced in hadronic collisions of cosmic-ray protons with the intra-cluster medium (ICM). The derived upper limit of the cosmic-ray energy density within this framework is an order of magnitude higher than that of our Galaxy.
△ Less
Submitted 23 August, 2009;
originally announced August 2009.
-
CANGAROO-III Observation of TeV Gamma Rays from the vicinity of PSR B1 706-44
Authors:
R. Enomoto,
J. Kushida,
T. Nakamori,
T. Kifune,
G. V. Bicknell,
R. W. Clay,
P. G. Edwards,
S. Gunji,
S. Hara,
T. Hara,
T. Hattori,
S. Hayashi,
Y. Higashi,
Y. Hirai,
K. Inoue,
H. Ishioka,
S. Kabuki,
F. Ka**o,
H. Katagiri,
A. Kawachi,
R. Kiuchi,
H. Kubo,
T. Kunisawa,
T. Matoba,
Y. Matsubara
, et al. (28 additional authors not shown)
Abstract:
Observation by the CANGAROO-III stereoscopic system of the Imaging Cherenkov Telescope has detected extended emission of TeV gamma rays in the vicinity of the pulsar PSR B1706$-$44. The strength of the signal observed as gamma-ray-like events varies when we apply different ways of emulating background events. The reason for such uncertainties is argued in relevance to gamma-rays embedded in the…
▽ More
Observation by the CANGAROO-III stereoscopic system of the Imaging Cherenkov Telescope has detected extended emission of TeV gamma rays in the vicinity of the pulsar PSR B1706$-$44. The strength of the signal observed as gamma-ray-like events varies when we apply different ways of emulating background events. The reason for such uncertainties is argued in relevance to gamma-rays embedded in the "off-source data", that is, unknown sources and diffuse emission in the Galactic plane, namely, the existence of a complex structure of TeV gamma-ray emission around PSR B1706$-$44.
△ Less
Submitted 12 August, 2009;
originally announced August 2009.
