-
ViewFusion: Towards Multi-View Consistency via Interpolated Denoising
Authors:
Xianghui Yang,
Yan Zuo,
Sameera Ramasinghe,
Loris Bazzani,
Gil Avraham,
Anton van den Hengel
Abstract:
Novel-view synthesis through diffusion models has demonstrated remarkable potential for generating diverse and high-quality images. Yet, the independent process of image generation in these prevailing methods leads to challenges in maintaining multiple-view consistency. To address this, we introduce ViewFusion, a novel, training-free algorithm that can be seamlessly integrated into existing pre-tr…
▽ More
Novel-view synthesis through diffusion models has demonstrated remarkable potential for generating diverse and high-quality images. Yet, the independent process of image generation in these prevailing methods leads to challenges in maintaining multiple-view consistency. To address this, we introduce ViewFusion, a novel, training-free algorithm that can be seamlessly integrated into existing pre-trained diffusion models. Our approach adopts an auto-regressive method that implicitly leverages previously generated views as context for the next view generation, ensuring robust multi-view consistency during the novel-view generation process. Through a diffusion process that fuses known-view information via interpolated denoising, our framework successfully extends single-view conditioned models to work in multiple-view conditional settings without any additional fine-tuning. Extensive experimental results demonstrate the effectiveness of ViewFusion in generating consistent and detailed novel views.
△ Less
Submitted 28 February, 2024;
originally announced February 2024.
-
Divide and Conquer: Rethinking the Training Paradigm of Neural Radiance Fields
Authors:
Rongkai Ma,
Leo Lebrat,
Rodrigo Santa Cruz,
Gil Avraham,
Yan Zuo,
Clinton Fookes,
Olivier Salvado
Abstract:
Neural radiance fields (NeRFs) have exhibited potential in synthesizing high-fidelity views of 3D scenes but the standard training paradigm of NeRF presupposes an equal importance for each image in the training set. This assumption poses a significant challenge for rendering specific views presenting intricate geometries, thereby resulting in suboptimal performance. In this paper, we take a closer…
▽ More
Neural radiance fields (NeRFs) have exhibited potential in synthesizing high-fidelity views of 3D scenes but the standard training paradigm of NeRF presupposes an equal importance for each image in the training set. This assumption poses a significant challenge for rendering specific views presenting intricate geometries, thereby resulting in suboptimal performance. In this paper, we take a closer look at the implications of the current training paradigm and redesign this for more superior rendering quality by NeRFs. Dividing input views into multiple groups based on their visual similarities and training individual models on each of these groups enables each model to specialize on specific regions without sacrificing speed or efficiency. Subsequently, the knowledge of these specialized models is aggregated into a single entity via a teacher-student distillation paradigm, enabling spatial efficiency for online render-ing. Empirically, we evaluate our novel training framework on two publicly available datasets, namely NeRF synthetic and Tanks&Temples. Our evaluation demonstrates that our DaC training pipeline enhances the rendering quality of a state-of-the-art baseline model while exhibiting convergence to a superior minimum.
△ Less
Submitted 29 January, 2024;
originally announced January 2024.
-
BLiRF: Bandlimited Radiance Fields for Dynamic Scene Modeling
Authors:
Sameera Ramasinghe,
Violetta Shevchenko,
Gil Avraham,
Anton Van Den Hengel
Abstract:
Reasoning the 3D structure of a non-rigid dynamic scene from a single moving camera is an under-constrained problem. Inspired by the remarkable progress of neural radiance fields (NeRFs) in photo-realistic novel view synthesis of static scenes, extensions have been proposed for dynamic settings. These methods heavily rely on neural priors in order to regularize the problem. In this work, we take a…
▽ More
Reasoning the 3D structure of a non-rigid dynamic scene from a single moving camera is an under-constrained problem. Inspired by the remarkable progress of neural radiance fields (NeRFs) in photo-realistic novel view synthesis of static scenes, extensions have been proposed for dynamic settings. These methods heavily rely on neural priors in order to regularize the problem. In this work, we take a step back and reinvestigate how current implementations may entail deleterious effects, including limited expressiveness, entanglement of light and density fields, and sub-optimal motion localization. As a remedy, we advocate for a bridge between classic non-rigid-structure-from-motion (\nrsfm) and NeRF, enabling the well-studied priors of the former to constrain the latter. To this end, we propose a framework that factorizes time and space by formulating a scene as a composition of bandlimited, high-dimensional signals. We demonstrate compelling results across complex dynamic scenes that involve changes in lighting, texture and long-range dynamics.
△ Less
Submitted 24 March, 2023; v1 submitted 27 February, 2023;
originally announced February 2023.
-
Nerfels: Renderable Neural Codes for Improved Camera Pose Estimation
Authors:
Gil Avraham,
Julian Straub,
Tianwei Shen,
Tsun-Yi Yang,
Hugo Germain,
Chris Sweeney,
Vasileios Balntas,
David Novotny,
Daniel DeTone,
Richard Newcombe
Abstract:
This paper presents a framework that combines traditional keypoint-based camera pose optimization with an invertible neural rendering mechanism. Our proposed 3D scene representation, Nerfels, is locally dense yet globally sparse. As opposed to existing invertible neural rendering systems which overfit a model to the entire scene, we adopt a feature-driven approach for representing scene-agnostic,…
▽ More
This paper presents a framework that combines traditional keypoint-based camera pose optimization with an invertible neural rendering mechanism. Our proposed 3D scene representation, Nerfels, is locally dense yet globally sparse. As opposed to existing invertible neural rendering systems which overfit a model to the entire scene, we adopt a feature-driven approach for representing scene-agnostic, local 3D patches with renderable codes. By modelling a scene only where local features are detected, our framework effectively generalizes to unseen local regions in the scene via an optimizable code conditioning mechanism in the neural renderer, all while maintaining the low memory footprint of a sparse 3D map representation. Our model can be incorporated to existing state-of-the-art hand-crafted and learned local feature pose estimators, yielding improved performance when evaluating on ScanNet for wide camera baseline scenarios.
△ Less
Submitted 4 June, 2022;
originally announced June 2022.
-
Learning Instance and Task-Aware Dynamic Kernels for Few Shot Learning
Authors:
Rongkai Ma,
Pengfei Fang,
Gil Avraham,
Yan Zuo,
Tianyu Zhu,
Tom Drummond,
Mehrtash Harandi
Abstract:
Learning and generalizing to novel concepts with few samples (Few-Shot Learning) is still an essential challenge to real-world applications. A principle way of achieving few-shot learning is to realize a model that can rapidly adapt to the context of a given task. Dynamic networks have been shown capable of learning content-adaptive parameters efficiently, making them suitable for few-shot learnin…
▽ More
Learning and generalizing to novel concepts with few samples (Few-Shot Learning) is still an essential challenge to real-world applications. A principle way of achieving few-shot learning is to realize a model that can rapidly adapt to the context of a given task. Dynamic networks have been shown capable of learning content-adaptive parameters efficiently, making them suitable for few-shot learning. In this paper, we propose to learn the dynamic kernels of a convolution network as a function of the task at hand, enabling faster generalization. To this end, we obtain our dynamic kernels based on the entire task and each sample and develop a mechanism further conditioning on each individual channel and position independently. This results in dynamic kernels that simultaneously attend to the global information whilst also considering minuscule details available. We empirically show that our model improves performance on few-shot classification and detection tasks, achieving a tangible improvement over several baseline models. This includes state-of-the-art results on 4 few-shot classification benchmarks: mini-ImageNet, tiered-ImageNet, CUB and FC100 and competitive results on a few-shot detection dataset: MS COCO-PASCAL-VOC.
△ Less
Submitted 12 July, 2022; v1 submitted 6 December, 2021;
originally announced December 2021.
-
Moving outside the lab: The viability of conducting sensorimotor learning studies online
Authors:
Jonathan S. Tsay,
Alan Lee,
Richard B. Ivry,
Guy Avraham
Abstract:
Collecting data online via crowdsourcing platforms has proven to be a very efficient way to recruit a large and diverse sample. Studies of motor learning, however, have been largely confined to the lab due to the need for special equipment to record movement kinematics and, as such, are typically only accessible to specific participants (e.g., college students). As a first foray to make motor lear…
▽ More
Collecting data online via crowdsourcing platforms has proven to be a very efficient way to recruit a large and diverse sample. Studies of motor learning, however, have been largely confined to the lab due to the need for special equipment to record movement kinematics and, as such, are typically only accessible to specific participants (e.g., college students). As a first foray to make motor learning studies accessible to a larger and more diverse audience, we developed an online, web-based platform (OnPoint) to collect kinematic data, serving as a template for researchers to create their own online sensorimotor control and learning experiments. As a proof-of-concept, we asked if fundamental motor learning phenomena discovered in the lab could be replicated online. In a series of three experiments, we observed a close correspondence between the results obtained online with those previously reported from research conducted in the laboratory. This web-based platform paired with online crowdsourcing can serve as a powerful new method for the study of motor control and learning.
△ Less
Submitted 28 July, 2021;
originally announced July 2021.
-
Localising In Complex Scenes Using Balanced Adversarial Adaptation
Authors:
Gil Avraham,
Yan Zuo,
Tom Drummond
Abstract:
Domain adaptation and generative modelling have collectively mitigated the expensive nature of data collection and labelling by leveraging the rich abundance of accurate, labelled data in simulation environments. In this work, we study the performance gap that exists between representations optimised for localisation on simulation environments and the application of such representations in a real-…
▽ More
Domain adaptation and generative modelling have collectively mitigated the expensive nature of data collection and labelling by leveraging the rich abundance of accurate, labelled data in simulation environments. In this work, we study the performance gap that exists between representations optimised for localisation on simulation environments and the application of such representations in a real-world setting. Our method exploits the shared geometric similarities between simulation and real-world environments whilst maintaining invariance towards visual discrepancies. This is achieved by optimising a representation extractor to project both simulated and real representations into a shared representation space. Our method uses a symmetrical adversarial approach which encourages the representation extractor to conceal the domain that features are extracted from and simultaneously preserves robust attributes between source and target domains that are beneficial for localisation. We evaluate our method by adapting representations optimised for indoor Habitat simulated environments (Matterport3D and Replica) to a real-world indoor environment (Active Vision Dataset), showing that it compares favourably against fully-supervised approaches.
△ Less
Submitted 8 November, 2020;
originally announced November 2020.
-
EMPNet: Neural Localisation and Map** Using Embedded Memory Points
Authors:
Gil Avraham,
Yan Zuo,
Thanuja Dharmasiri,
Tom Drummond
Abstract:
Continuously estimating an agent's state space and a representation of its surroundings has proven vital towards full autonomy. A shared common ground among systems which successfully achieve this feat is the integration of previously encountered observations into the current state being estimated. This necessitates the use of a memory module for incorporating previously visited states whilst simu…
▽ More
Continuously estimating an agent's state space and a representation of its surroundings has proven vital towards full autonomy. A shared common ground among systems which successfully achieve this feat is the integration of previously encountered observations into the current state being estimated. This necessitates the use of a memory module for incorporating previously visited states whilst simultaneously offering an internal representation of the observed environment. In this work we develop a memory module which contains rigidly aligned point-embeddings that represent a coherent scene structure acquired from an RGB-D sequence of observations. The point-embeddings are extracted using modern convolutional neural network architectures, and alignment is performed by computing a dense correspondence matrix between a new observation and the current embeddings residing in the memory module. The whole framework is end-to-end trainable, resulting in a recurrent joint optimisation of the point-embeddings contained in the memory. This process amplifies the shared information across states, providing increased robustness and accuracy. We show significant improvement of our method across a set of experiments performed on the synthetic VIZDoom environment and a real world Active Vision Dataset.
△ Less
Submitted 1 August, 2019; v1 submitted 30 July, 2019;
originally announced July 2019.
-
Traversing Latent Space using Decision Ferns
Authors:
Yan Zuo,
Gil Avraham,
Tom Drummond
Abstract:
The practice of transforming raw data to a feature space so that inference can be performed in that space has been popular for many years. Recently, rapid progress in deep neural networks has given both researchers and practitioners enhanced methods that increase the richness of feature representations, be it from images, text or speech. In this work we show how a constructed latent space can be e…
▽ More
The practice of transforming raw data to a feature space so that inference can be performed in that space has been popular for many years. Recently, rapid progress in deep neural networks has given both researchers and practitioners enhanced methods that increase the richness of feature representations, be it from images, text or speech. In this work we show how a constructed latent space can be explored in a controlled manner and argue that this complements well founded inference methods. For constructing the latent space a Variational Autoencoder is used. We present a novel controller module that allows for smooth traversal in the latent space and construct an end-to-end trainable framework. We explore the applicability of our method for performing spatial transformations as well as kinematics for predicting future latent vectors of a video sequence.
△ Less
Submitted 6 December, 2018;
originally announced December 2018.
-
Generative Adversarial Forests for Better Conditioned Adversarial Learning
Authors:
Yan Zuo,
Gil Avraham,
Tom Drummond
Abstract:
In recent times, many of the breakthroughs in various vision-related tasks have revolved around improving learning of deep models; these methods have ranged from network architectural improvements such as Residual Networks, to various forms of regularisation such as Batch Normalisation. In essence, many of these techniques revolve around better conditioning, allowing for deeper and deeper models t…
▽ More
In recent times, many of the breakthroughs in various vision-related tasks have revolved around improving learning of deep models; these methods have ranged from network architectural improvements such as Residual Networks, to various forms of regularisation such as Batch Normalisation. In essence, many of these techniques revolve around better conditioning, allowing for deeper and deeper models to be successfully learned. In this paper, we look towards better conditioning Generative Adversarial Networks (GANs) in an unsupervised learning setting. Our method embeds the powerful discriminating capabilities of a decision forest into the discriminator of a GAN. This results in a better conditioned model which learns in an extremely stable way. We demonstrate empirical results which show both clear qualitative and quantitative evidence of the effectiveness of our approach, gaining significant performance improvements over several popular GAN-based approaches on the Oxford Flowers and Aligned Celebrity Faces datasets.
△ Less
Submitted 14 May, 2018;
originally announced May 2018.
