-
Unifying Scene Representation and Hand-Eye Calibration with 3D Foundation Models
Authors:
Weiming Zhi,
Haozhan Tang,
Tianyi Zhang,
Matthew Johnson-Roberson
Abstract:
Representing the environment is a central challenge in robotics, and is essential for effective decision-making. Traditionally, before capturing images with a manipulator-mounted camera, users need to calibrate the camera using a specific external marker, such as a checkerboard or AprilTag. However, recent advances in computer vision have led to the development of \emph{3D foundation models}. Thes…
▽ More
Representing the environment is a central challenge in robotics, and is essential for effective decision-making. Traditionally, before capturing images with a manipulator-mounted camera, users need to calibrate the camera using a specific external marker, such as a checkerboard or AprilTag. However, recent advances in computer vision have led to the development of \emph{3D foundation models}. These are large, pre-trained neural networks that can establish fast and accurate multi-view correspondences with very few images, even in the absence of rich visual features. This paper advocates for the integration of 3D foundation models into scene representation approaches for robotic systems equipped with manipulator-mounted RGB cameras. Specifically, we propose the Joint Calibration and Representation (JCR) method. JCR uses RGB images, captured by a manipulator-mounted camera, to simultaneously construct an environmental representation and calibrate the camera relative to the robot's end-effector, in the absence of specific calibration markers. The resulting 3D environment representation is aligned with the robot's coordinate frame and maintains physically accurate scales. We demonstrate that JCR can build effective scene representations using a low-cost RGB camera attached to a manipulator, without prior calibration.
△ Less
Submitted 17 April, 2024;
originally announced April 2024.
-
Diagrammatic Instructions to Specify Spatial Objectives and Constraints with Applications to Mobile Base Placement
Authors:
Qilin Sun,
Weiming Zhi,
Tianyi Zhang,
Matthew Johnson-Roberson
Abstract:
This paper introduces Spatial Diagrammatic Instructions (SDIs), an approach for human operators to specify objectives and constraints that are related to spatial regions in the working environment. Human operators are enabled to sketch out regions directly on camera images that correspond to the objectives and constraints. These sketches are projected to 3D spatial coordinates, and continuous Spat…
▽ More
This paper introduces Spatial Diagrammatic Instructions (SDIs), an approach for human operators to specify objectives and constraints that are related to spatial regions in the working environment. Human operators are enabled to sketch out regions directly on camera images that correspond to the objectives and constraints. These sketches are projected to 3D spatial coordinates, and continuous Spatial Instruction Maps (SIMs) are learned upon them. These maps can then be integrated into optimization problems for tasks of robots. In particular, we demonstrate how Spatial Diagrammatic Instructions can be applied to solve the Base Placement Problem of mobile manipulators, which concerns the best place to put the manipulator to facilitate a certain task. Human operators can specify, via sketch, spatial regions of interest for a manipulation task and permissible regions for the mobile manipulator to be at. Then, an optimization problem that maximizes the manipulator's reachability, or coverage, over the designated regions of interest while remaining in the permissible regions is solved. We provide extensive empirical evaluations, and show that our formulation of Spatial Instruction Maps provides accurate representations of user-specified diagrammatic instructions. Furthermore, we demonstrate that our diagrammatic approach to the Mobile Base Placement Problem enables higher quality solutions and faster run-time.
△ Less
Submitted 25 March, 2024; v1 submitted 19 March, 2024;
originally announced March 2024.
-
DarkGS: Learning Neural Illumination and 3D Gaussians Relighting for Robotic Exploration in the Dark
Authors:
Tianyi Zhang,
Kaining Huang,
Weiming Zhi,
Matthew Johnson-Roberson
Abstract:
Humans have the remarkable ability to construct consistent mental models of an environment, even under limited or varying levels of illumination. We wish to endow robots with this same capability. In this paper, we tackle the challenge of constructing a photorealistic scene representation under poorly illuminated conditions and with a moving light source. We approach the task of modeling illuminat…
▽ More
Humans have the remarkable ability to construct consistent mental models of an environment, even under limited or varying levels of illumination. We wish to endow robots with this same capability. In this paper, we tackle the challenge of constructing a photorealistic scene representation under poorly illuminated conditions and with a moving light source. We approach the task of modeling illumination as a learning problem, and utilize the developed illumination model to aid in scene reconstruction. We introduce an innovative framework that uses a data-driven approach, Neural Light Simulators (NeLiS), to model and calibrate the camera-light system. Furthermore, we present DarkGS, a method that applies NeLiS to create a relightable 3D Gaussian scene model capable of real-time, photorealistic rendering from novel viewpoints. We show the applicability and robustness of our proposed simulator and system in a variety of real-world environments.
△ Less
Submitted 16 March, 2024;
originally announced March 2024.
-
V-PRISM: Probabilistic Map** of Unknown Tabletop Scenes
Authors:
Herbert Wright,
Weiming Zhi,
Matthew Johnson-Roberson,
Tucker Hermans
Abstract:
The ability to construct concise scene representations from sensor input is central to the field of robotics. This paper addresses the problem of robustly creating a 3D representation of a tabletop scene from a segmented RGB-D image. These representations are then critical for a range of downstream manipulation tasks. Many previous attempts to tackle this problem do not capture accurate uncertaint…
▽ More
The ability to construct concise scene representations from sensor input is central to the field of robotics. This paper addresses the problem of robustly creating a 3D representation of a tabletop scene from a segmented RGB-D image. These representations are then critical for a range of downstream manipulation tasks. Many previous attempts to tackle this problem do not capture accurate uncertainty, which is required to subsequently produce safe motion plans. In this paper, we cast the representation of 3D tabletop scenes as a multi-class classification problem. To tackle this, we introduce V-PRISM, a framework and method for robustly creating probabilistic 3D segmentation maps of tabletop scenes. Our maps contain both occupancy estimates, segmentation information, and principled uncertainty measures. We evaluate the robustness of our method in (1) procedurally generated scenes using open-source object datasets, and (2) real-world tabletop data collected from a depth camera. Our experiments show that our approach outperforms alternative continuous reconstruction approaches that do not explicitly reason about objects in a multi-class formulation.
△ Less
Submitted 13 March, 2024; v1 submitted 12 March, 2024;
originally announced March 2024.
-
Front-end electronics development of large-area SiPM arrays for high-precision single-photon time measurement
Authors:
Wei Zhi,
Ruike Cao,
Jiannan Tang,
Mingxin Wang,
Yongqi Tan,
Weihao Wu,
Donglian Xu
Abstract:
TRopIcal DEep-sea Neutrino Telescope (TRIDENT) plans to incorporate silicon photomultipliers (SiPMs) with superior time resolution in addition to photomultiplier tubes (PMTs) into its detection units, namely hybrid Digital Optical Modules (hDOMs), to improve its angular resolution. However, the time resolution significantly degrades for large-area SiPMs due to the large detector capacitance, posin…
▽ More
TRopIcal DEep-sea Neutrino Telescope (TRIDENT) plans to incorporate silicon photomultipliers (SiPMs) with superior time resolution in addition to photomultiplier tubes (PMTs) into its detection units, namely hybrid Digital Optical Modules (hDOMs), to improve its angular resolution. However, the time resolution significantly degrades for large-area SiPMs due to the large detector capacitance, posing significant challenges for the readout electronics of SiPMs in hDOM. We analyzed the influences of series and parallel connections when constructing a large-area SiPM array and designed a series-parallel connection SiPM array with differential output. We also designed a high-speed pre-amplifier based on transformers (MABA-007159) and radio frequency amplifiers (BGA2803), and an analog multi-channel summing circuit based on operational amplifiers (LMH6629). We measured the single photon time resolution (SPTR) of a $4\times4$ SiPM (Hamamatsu S13360-3050PE) array ($12\times12~\mathrm{mm}^2$) of approximately 300 ps FWHM. This front-end readout design enables the large-area SiPM array to achieve high-precision single photon time measurement in one readout channel.
△ Less
Submitted 7 June, 2024; v1 submitted 5 March, 2024;
originally announced March 2024.
-
Imaging de Haas-van Alphen quantum oscillations and milli-Tesla pseudomagnetic fields
Authors:
Haibiao Zhou,
Nadav Auerbach,
Matan Uzan,
Yaozhang Zhou,
Nasrin Banu,
Weifeng Zhi,
Martin E. Huber,
Kenji Watanabe,
Takashi Taniguchi,
Yuri Myasoedov,
Binghai Yan,
Eli Zeldov
Abstract:
A unique attribute of atomically thin quantum materials is the in-situ tunability of their electronic band structure by externally controllable parameters like electrostatic do**, electric field, strain, electron interactions, and displacement or twisting of atomic layers. This unparalleled control of the electronic bands has led to the discovery of a plethora of exotic emergent phenomena. But d…
▽ More
A unique attribute of atomically thin quantum materials is the in-situ tunability of their electronic band structure by externally controllable parameters like electrostatic do**, electric field, strain, electron interactions, and displacement or twisting of atomic layers. This unparalleled control of the electronic bands has led to the discovery of a plethora of exotic emergent phenomena. But despite its key role, there is currently no versatile method for map** the local band structure in advanced 2D materials devices in which the active layer is commonly embedded in various insulating layers and metallic gates. Utilizing a scanning superconducting quantum interference device, we image the de Haas-van Alphen quantum oscillations in a model system, the Bernal-stacked trilayer graphene with dual gates, which displays multiple highly-tunable bands. By resolving thermodynamic quantum oscillations spanning over 100 Landau levels in low magnetic fields, we reconstruct the band structure and its controllable evolution with the displacement field with unprecedented precision and spatial resolution of 150 nm. Moreover, by develo** Landau level interferometry, we reveal shear-strain-induced pseudomagnetic fields and map their spatial dependence. In contrast to artificially-induced large strain, which leads to pseudomagnetic fields of hundreds of Tesla, we detect naturally occurring pseudomagnetic fields as low as 1 mT corresponding to graphene twisting by just 1 millidegree over one μm distance, two orders of magnitude lower than the typical angle disorder in high-quality twisted bilayer graphene devices. This ability to resolve the local band structure and strain on the nanoscale opens the door to the characterization and utilization of tunable band engineering in practical van der Waals devices.
△ Less
Submitted 3 November, 2023;
originally announced November 2023.
-
Learning Orbitally Stable Systems for Diagrammatically Teaching
Authors:
Weiming Zhi,
Tianyi Zhang,
Matthew Johnson-Roberson
Abstract:
Diagrammatic Teaching is a paradigm for robots to acquire novel skills, whereby the user provides 2D sketches over images of the scene to shape the robot's motion. In this work, we tackle the problem of teaching a robot to approach a surface and then follow cyclic motion on it, where the cycle of the motion can be arbitrarily specified by a single user-provided sketch over an image from the robot'…
▽ More
Diagrammatic Teaching is a paradigm for robots to acquire novel skills, whereby the user provides 2D sketches over images of the scene to shape the robot's motion. In this work, we tackle the problem of teaching a robot to approach a surface and then follow cyclic motion on it, where the cycle of the motion can be arbitrarily specified by a single user-provided sketch over an image from the robot's camera. Accordingly, we contribute the Stable Diffeomorphic Diagrammatic Teaching (SDDT) framework. SDDT models the robot's motion as an Orbitally Asymptotically Stable (O.A.S.) dynamical system that learns to stablize based on a single diagrammatic sketch provided by the user. This is achieved by applying a \emph{diffeomorphism}, i.e. a differentiable and invertible function, to morph a known O.A.S. system. The parameterised diffeomorphism is then optimised with respect to the Hausdorff distance between the limit cycle of our modelled system and the sketch, to produce the desired robot motion. We provide novel theoretical insight into the behaviour of the optimised system and also empirically evaluate SDDT, both in simulation and on a quadruped with a mounted 6-DOF manipulator. Results show that we can diagrammatically teach complex cyclic motion patterns with a high degree of accuracy.
△ Less
Submitted 29 March, 2024; v1 submitted 19 September, 2023;
originally announced September 2023.
-
Pedestrian Trajectory Prediction Using Dynamics-based Deep Learning
Authors:
Honghui Wang,
Weiming Zhi,
Gustavo Batista,
Rohitash Chandra
Abstract:
Pedestrian trajectory prediction plays an important role in autonomous driving systems and robotics. Recent work utilizing prominent deep learning models for pedestrian motion prediction makes limited a priori assumptions about human movements, resulting in a lack of explainability and explicit constraints enforced on predicted trajectories. We present a dynamics-based deep learning framework with…
▽ More
Pedestrian trajectory prediction plays an important role in autonomous driving systems and robotics. Recent work utilizing prominent deep learning models for pedestrian motion prediction makes limited a priori assumptions about human movements, resulting in a lack of explainability and explicit constraints enforced on predicted trajectories. We present a dynamics-based deep learning framework with a novel asymptotically stable dynamical system integrated into a Transformer-based model. We use an asymptotically stable dynamical system to model human goal-targeted motion by enforcing the human walking trajectory, which converges to a predicted goal position, and to provide the Transformer model with prior knowledge and explainability. Our framework features the Transformer model that works with a goal estimator and dynamical system to learn features from pedestrian motion history. The results show that our framework outperforms prominent models using five benchmark human motion datasets.
△ Less
Submitted 10 March, 2024; v1 submitted 16 September, 2023;
originally announced September 2023.
-
Instructing Robots by Sketching: Learning from Demonstration via Probabilistic Diagrammatic Teaching
Authors:
Weiming Zhi,
Tianyi Zhang,
Matthew Johnson-Roberson
Abstract:
Learning for Demonstration (LfD) enables robots to acquire new skills by imitating expert demonstrations, allowing users to communicate their instructions in an intuitive manner. Recent progress in LfD often relies on kinesthetic teaching or teleoperation as the medium for users to specify the demonstrations. Kinesthetic teaching requires physical handling of the robot, while teleoperation demands…
▽ More
Learning for Demonstration (LfD) enables robots to acquire new skills by imitating expert demonstrations, allowing users to communicate their instructions in an intuitive manner. Recent progress in LfD often relies on kinesthetic teaching or teleoperation as the medium for users to specify the demonstrations. Kinesthetic teaching requires physical handling of the robot, while teleoperation demands proficiency with additional hardware. This paper introduces an alternative paradigm for LfD called Diagrammatic Teaching. Diagrammatic Teaching aims to teach robots novel skills by prompting the user to sketch out demonstration trajectories on 2D images of the scene, these are then synthesised as a generative model of motion trajectories in 3D task space. Additionally, we present the Ray-tracing Probabilistic Trajectory Learning (RPTL) framework for Diagrammatic Teaching. RPTL extracts time-varying probability densities from the 2D sketches, applies ray-tracing to find corresponding regions in 3D Cartesian space, and fits a probabilistic model of motion trajectories to these regions. New motion trajectories, which mimic those sketched by the user, can then be generated from the probabilistic model. We empirically validate our framework both in simulation and on real robots, which include a fixed-base manipulator and a quadruped-mounted manipulator.
△ Less
Submitted 31 March, 2024; v1 submitted 7 September, 2023;
originally announced September 2023.
-
Simulation study on the optical processes at deep-sea neutrino telescope sites
Authors:
Fan Hu,
Zhenyu Wei,
Wei Tian,
Zi** Ye,
Fuyudi Zhang,
Zhengyang Sun,
Wei Zhi,
Qichao Chang,
Qiao Xue,
Zhuo Li,
Donglian Xu
Abstract:
The performance of a large-scale water Cherenkov neutrino telescope relies heavily on the transparency of the surrounding water, quantified by its level of light absorption and scattering. A pathfinder experiment was carried out to measure the optical properties of deep seawater in South China Sea with light-emitting diodes (LEDs) as light sources, photon multiplier tubes (PMTs) and cameras as pho…
▽ More
The performance of a large-scale water Cherenkov neutrino telescope relies heavily on the transparency of the surrounding water, quantified by its level of light absorption and scattering. A pathfinder experiment was carried out to measure the optical properties of deep seawater in South China Sea with light-emitting diodes (LEDs) as light sources, photon multiplier tubes (PMTs) and cameras as photon sensors. Here, we present an optical simulation program employing the Geant4 toolkit to understand the absorption and scattering processes in the deep seawater, which helps to extract the underlying optical properties from the experimental data. The simulation results are compared with the experimental data and show good agreements. We also verify the analysis methods that utilize various observables of the PMTs and the cameras with this simulation program, which can be easily adapted by other neutrino telescope pathfinder experiments and future large-scale detectors.
△ Less
Submitted 9 February, 2023;
originally announced February 2023.
-
A multi-cubic-kilometre neutrino telescope in the western Pacific Ocean
Authors:
Z. P. Ye,
F. Hu,
W. Tian,
Q. C. Chang,
Y. L. Chang,
Z. S. Cheng,
J. Gao,
T. Ge,
G. H. Gong,
J. Guo,
X. X. Guo,
X. G. He,
J. T. Huang,
K. Jiang,
P. K. Jiang,
Y. P. **g,
H. L. Li,
J. L. Li,
L. Li,
W. L. Li,
Z. Li,
N. Y. Liao,
Q. Lin,
F. Liu,
J. L. Liu
, et al. (33 additional authors not shown)
Abstract:
Next-generation neutrino telescopes with significantly improved sensitivity are required to pinpoint the sources of the diffuse astrophysical neutrino flux detected by IceCube and uncover the century-old puzzle of cosmic ray origins. A detector near the equator will provide a unique viewpoint of the neutrino sky, complementing IceCube and other neutrino telescopes in the Northern Hemisphere. Here…
▽ More
Next-generation neutrino telescopes with significantly improved sensitivity are required to pinpoint the sources of the diffuse astrophysical neutrino flux detected by IceCube and uncover the century-old puzzle of cosmic ray origins. A detector near the equator will provide a unique viewpoint of the neutrino sky, complementing IceCube and other neutrino telescopes in the Northern Hemisphere. Here we present results from an expedition to the north-eastern region of the South China Sea, in the western Pacific Ocean. A favorable neutrino telescope site was found on an abyssal plain at a depth of $\sim$ 3.5km. At depths below 3km, the sea current speed, water absorption and scattering lengths for Cherenkov light, were measured to be $v_{\mathrm{c}}<$10cm/s, $λ_{\mathrm{abs} }\simeq$ 27m and $λ_{\mathrm{sca} }\simeq$ 63m, respectively. Accounting for these measurements, we present the design and expected performance of a next-generation neutrino telescope, TRopIcal DEep-sea Neutrino Telescope (TRIDENT). With its advanced photon-detection technology and large dimensions, TRIDENT expects to observe the IceCube steady source candidate NGC 1068 with 5$σ$ significance within 1 year of operation. This level of sensitivity will open a new arena for diagnosing the origin of cosmic rays and probing fundamental physics over astronomical baselines.
△ Less
Submitted 13 May, 2024; v1 submitted 10 July, 2022;
originally announced July 2022.
-
L4KDE: Learning for KinoDynamic Tree Expansion
Authors:
Tin Lai,
Weiming Zhi,
Tucker Hermans,
Fabio Ramos
Abstract:
We present the Learning for KinoDynamic Tree Expansion (L4KDE) method for kinodynamic planning. Tree-based planning approaches, such as rapidly exploring random tree (RRT), are the dominant approach to finding globally optimal plans in continuous state-space motion planning. Central to these approaches is tree-expansion, the procedure in which new nodes are added into an ever-expanding tree. We st…
▽ More
We present the Learning for KinoDynamic Tree Expansion (L4KDE) method for kinodynamic planning. Tree-based planning approaches, such as rapidly exploring random tree (RRT), are the dominant approach to finding globally optimal plans in continuous state-space motion planning. Central to these approaches is tree-expansion, the procedure in which new nodes are added into an ever-expanding tree. We study the kinodynamic variants of tree-based planning, where we have known system dynamics and kinematic constraints. In the interest of quickly selecting nodes to connect newly sampled coordinates, existing methods typically cannot optimise to find nodes that have low cost to transition to sampled coordinates. Instead, they use metrics like Euclidean distance between coordinates as a heuristic for selecting candidate nodes to connect to the search tree. We propose L4KDE to address this issue. L4KDE uses a neural network to predict transition costs between queried states, which can be efficiently computed in batch, providing much higher quality estimates of transition cost compared to commonly used heuristics while maintaining almost-surely asymptotic optimality guarantee. We empirically demonstrate the significant performance improvement provided by L4KDE on a variety of challenging system dynamics, with the ability to generalise across different instances of the same model class, and in conjunction with a suite of modern tree-based motion planners.
△ Less
Submitted 17 September, 2023; v1 submitted 2 March, 2022;
originally announced March 2022.
-
Parallelised Diffeomorphic Sampling-based Motion Planning
Authors:
Tin Lai,
Weiming Zhi,
Tucker Hermans,
Fabio Ramos
Abstract:
We propose Parallelised Diffeomorphic Sampling-based Motion Planning (PDMP). PDMP is a novel parallelised framework that uses bijective and differentiable map**s, or diffeomorphisms, to transform sampling distributions of sampling-based motion planners, in a manner akin to normalising flows. Unlike normalising flow models which use invertible neural network structures to represent these diffeomo…
▽ More
We propose Parallelised Diffeomorphic Sampling-based Motion Planning (PDMP). PDMP is a novel parallelised framework that uses bijective and differentiable map**s, or diffeomorphisms, to transform sampling distributions of sampling-based motion planners, in a manner akin to normalising flows. Unlike normalising flow models which use invertible neural network structures to represent these diffeomorphisms, we develop them from gradient information of desired costs, and encode desirable behaviour, such as obstacle avoidance. These transformed sampling distributions can then be used for sampling-based motion planning. A particular example is when we wish to imbue the sampling distribution with knowledge of the environment geometry, such that drawn samples are less prone to be in collisions. To this end, we propose to learn a continuous occupancy representation from environment occupancy data, such that gradients of the representation defines a valid diffeomorphism and is amenable to fast parallel evaluation. We use this to "morph" the sampling distribution to draw far fewer collision-prone samples. PDMP is able to leverage gradient information of costs, to inject specifications, in a manner similar to optimisation-based motion planning methods, but relies on drawing from a sampling distribution, retaining the tendency to find more global solutions, thereby bridging the gap between trajectory optimisation and sampling-based planning methods.
△ Less
Submitted 22 September, 2021; v1 submitted 26 August, 2021;
originally announced August 2021.
-
Probabilistic Trajectory Prediction with Structural Constraints
Authors:
Weiming Zhi,
Lionel Ott,
Fabio Ramos
Abstract:
This work addresses the problem of predicting the motion trajectories of dynamic objects in the environment. Recent advances in predicting motion patterns often rely on machine learning techniques to extrapolate motion patterns from observed trajectories, with no mechanism to directly incorporate known rules. We propose a novel framework, which combines probabilistic learning and constrained traje…
▽ More
This work addresses the problem of predicting the motion trajectories of dynamic objects in the environment. Recent advances in predicting motion patterns often rely on machine learning techniques to extrapolate motion patterns from observed trajectories, with no mechanism to directly incorporate known rules. We propose a novel framework, which combines probabilistic learning and constrained trajectory optimisation. The learning component of our framework provides a distribution over future motion trajectories conditioned on observed past coordinates. This distribution is then used as a prior to a constrained optimisation problem which enforces chance constraints on the trajectory distribution. This results in constraint-compliant trajectory distributions which closely resemble the prior. In particular, we focus our investigation on collision constraints, such that extrapolated future trajectory distributions conform to the environment structure. We empirically demonstrate on real-world and simulated datasets the ability of our framework to learn complex probabilistic motion trajectories for motion data, while directly enforcing constraints to improve generalisability, producing more robust and higher quality trajectory distributions.
△ Less
Submitted 8 July, 2021;
originally announced July 2021.
-
Learning ODEs via Diffeomorphisms for Fast and Robust Integration
Authors:
Weiming Zhi,
Tin Lai,
Lionel Ott,
Edwin V. Bonilla,
Fabio Ramos
Abstract:
Advances in differentiable numerical integrators have enabled the use of gradient descent techniques to learn ordinary differential equations (ODEs). In the context of machine learning, differentiable solvers are central for Neural ODEs (NODEs), a class of deep learning models with continuous depth, rather than discrete layers. However, these integrators can be unsatisfactorily slow and inaccurate…
▽ More
Advances in differentiable numerical integrators have enabled the use of gradient descent techniques to learn ordinary differential equations (ODEs). In the context of machine learning, differentiable solvers are central for Neural ODEs (NODEs), a class of deep learning models with continuous depth, rather than discrete layers. However, these integrators can be unsatisfactorily slow and inaccurate when learning systems of ODEs from long sequences, or when solutions of the system vary at widely different timescales in each dimension. In this paper we propose an alternative approach to learning ODEs from data: we represent the underlying ODE as a vector field that is related to another base vector field by a differentiable bijection, modelled by an invertible neural network. By restricting the base ODE to be amenable to integration, we can drastically speed up and improve the robustness of integration. We demonstrate the efficacy of our method in training and evaluating continuous neural networks models, as well as in learning benchmark ODE systems. We observe improvements of up to two orders of magnitude when integrating learned ODEs with GPUs computation.
△ Less
Submitted 4 July, 2021;
originally announced July 2021.
-
Anticipatory Navigation in Crowds by Probabilistic Prediction of Pedestrian Future Movements
Authors:
Weiming Zhi,
Tin Lai,
Lionel Ott,
Fabio Ramos
Abstract:
Critical for the coexistence of humans and robots in dynamic environments is the capability for agents to understand each other's actions, and anticipate their movements. This paper presents Stochastic Process Anticipatory Navigation (SPAN), a framework that enables nonholonomic robots to navigate in environments with crowds, while anticipating and accounting for the motion patterns of pedestrians…
▽ More
Critical for the coexistence of humans and robots in dynamic environments is the capability for agents to understand each other's actions, and anticipate their movements. This paper presents Stochastic Process Anticipatory Navigation (SPAN), a framework that enables nonholonomic robots to navigate in environments with crowds, while anticipating and accounting for the motion patterns of pedestrians. To this end, we learn a predictive model to predict continuous-time stochastic processes to model future movement of pedestrians. Anticipated pedestrian positions are used to conduct chance constrained collision-checking, and are incorporated into a time-to-collision control problem. An occupancy map is also integrated to allow for probabilistic collision-checking with static obstacles. We demonstrate the capability of SPAN in crowded simulation environments, as well as with a real-world pedestrian dataset.
△ Less
Submitted 12 November, 2020;
originally announced November 2020.
-
OCTNet: Trajectory Generation in New Environments from Past Experiences
Authors:
Weiming Zhi,
Tin Lai,
Lionel Ott,
Gilad Francis,
Fabio Ramos
Abstract:
Being able to safely operate for extended periods of time in dynamic environments is a critical capability for autonomous systems. This generally involves the prediction and understanding of motion patterns of dynamic entities, such as vehicles and people, in the surroundings. Many motion prediction methods in the literature can learn a function, map** position and time to potential trajectories…
▽ More
Being able to safely operate for extended periods of time in dynamic environments is a critical capability for autonomous systems. This generally involves the prediction and understanding of motion patterns of dynamic entities, such as vehicles and people, in the surroundings. Many motion prediction methods in the literature can learn a function, map** position and time to potential trajectories taken by people or other dynamic entities. However, these predictions depend only on previously observed trajectories, and do not explicitly take into consideration the environment. Trends of motion obtained in one environment are typically specific to that environment, and are not used to better predict motion in other environments. In this paper, we address the problem of generating likely motion dynamics conditioned on the environment, represented as an occupancy map. We introduce the Occupancy Conditional Trajectory Network (OCTNet) framework, capable of generalising the previously observed motion in known environments, to generate trajectories in new environments where no observations of motion has not been observed. OCTNet encodes trajectories as a fixed-sized vector of parameters and utilises neural networks to learn conditional distributions over parameters. We empirically demonstrate our method's ability to generate complex multi-modal trajectory patterns in different environments.
△ Less
Submitted 25 September, 2019;
originally announced September 2019.
-
Occ-Traj120: Occupancy Maps with Associated Trajectories
Authors:
Tin Lai,
Weiming Zhi,
Fabio Ramos
Abstract:
Trajectory modelling had been the principal research area for understanding and anticipating human behaviour. Predicting the dynamic path by observing the agent and its surrounding environment are essential for applications such as autonomous driving and indoor navigation suggestions. However, despite the numerous researches that had been presented, most available dataset does not contains any inf…
▽ More
Trajectory modelling had been the principal research area for understanding and anticipating human behaviour. Predicting the dynamic path by observing the agent and its surrounding environment are essential for applications such as autonomous driving and indoor navigation suggestions. However, despite the numerous researches that had been presented, most available dataset does not contains any information on environmental factors---such as the occupancy representation of the map---which arguably plays a significant role on how an agent chooses its trajectory.
We present a trajectory dataset with the corresponding occupancy representations of different local-maps. The dataset contains more than 120 locally-structured maps with occupancy representation and more than 110K trajectories in total. Each map has few hundred corresponding simulated trajectories that navigate from a spatial location of a room to another point. The dataset is freely available online.
△ Less
Submitted 2 March, 2020; v1 submitted 5 September, 2019;
originally announced September 2019.
-
Kernel Trajectory Maps for Multi-Modal Probabilistic Motion Prediction
Authors:
Weiming Zhi,
Lionel Ott,
Fabio Ramos
Abstract:
Understanding the dynamics of an environment, such as the movement of humans and vehicles, is crucial for agents to achieve long-term autonomy in urban environments. This requires the development of methods to capture the multi-modal and probabilistic nature of motion patterns. We present Kernel Trajectory Maps (KTM) to capture the trajectories of movement in an environment. KTMs leverage the expr…
▽ More
Understanding the dynamics of an environment, such as the movement of humans and vehicles, is crucial for agents to achieve long-term autonomy in urban environments. This requires the development of methods to capture the multi-modal and probabilistic nature of motion patterns. We present Kernel Trajectory Maps (KTM) to capture the trajectories of movement in an environment. KTMs leverage the expressiveness of kernels from non-parametric modelling by projecting input trajectories onto a set of representative trajectories, to condition on a sequence of observed waypoint coordinates, and predict a multi-modal distribution over possible future trajectories. The output is a mixture of continuous stochastic processes, where each realisation is a continuous functional trajectory, which can be queried at arbitrarily fine time steps.
△ Less
Submitted 7 October, 2019; v1 submitted 11 July, 2019;
originally announced July 2019.
-
The test of the electronics system for the BESIII ETOF upgrade
Authors:
Wang Xiaozhuang,
Dai Hongliang,
Wu Zhi,
Heng Yuekun,
Zhang Jie,
Cao **,
Ji Xiaolu,
Li Cheng,
Sun Weijia,
Wang Siyu,
Wang Yun
Abstract:
It is proposed to upgrade the endcap time-of-flight (ETOF) of the Bei**g Spectrometer III (BESIII) with multi-gap resistive plate chamber (MRPC), aiming at overall time resolution about 80 ps. After the entire electronics system is ready, some experiments, such as heat radiating, irradiation hardness and large-current beam tests,are carried out to certify the electronics' reliability and stabilit…
▽ More
It is proposed to upgrade the endcap time-of-flight (ETOF) of the Bei**g Spectrometer III (BESIII) with multi-gap resistive plate chamber (MRPC), aiming at overall time resolution about 80 ps. After the entire electronics system is ready, some experiments, such as heat radiating, irradiation hardness and large-current beam tests,are carried out to certify the electronics' reliability and stability. The on-detector test of the electronics is also performed with the beam at BEPCII E3 line, the test results indicate that the electronics system fulfills its design requirements.
△ Less
Submitted 7 March, 2016;
originally announced March 2016.
