-
Freeview Sketching: View-Aware Fine-Grained Sketch-Based Image Retrieval
Authors:
Aneeshan Sain,
Pinaki Nath Chowdhury,
Subhadeep Koley,
Ayan Kumar Bhunia,
Yi-Zhe Song
Abstract:
In this paper, we delve into the intricate dynamics of Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) by addressing a critical yet overlooked aspect -- the choice of viewpoint during sketch creation. Unlike photo systems that seamlessly handle diverse views through extensive datasets, sketch systems, with limited data collected from fixed perspectives, face challenges. Our pilot study, employ…
▽ More
In this paper, we delve into the intricate dynamics of Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) by addressing a critical yet overlooked aspect -- the choice of viewpoint during sketch creation. Unlike photo systems that seamlessly handle diverse views through extensive datasets, sketch systems, with limited data collected from fixed perspectives, face challenges. Our pilot study, employing a pre-trained FG-SBIR model, highlights the system's struggle when query-sketches differ in viewpoint from target instances. Interestingly, a questionnaire however shows users desire autonomy, with a significant percentage favouring view-specific retrieval. To reconcile this, we advocate for a view-aware system, seamlessly accommodating both view-agnostic and view-specific tasks. Overcoming dataset limitations, our first contribution leverages multi-view 2D projections of 3D objects, instilling cross-modal view awareness. The second contribution introduces a customisable cross-modal feature through disentanglement, allowing effortless mode switching. Extensive experiments on standard datasets validate the effectiveness of our method.
△ Less
Submitted 1 July, 2024;
originally announced July 2024.
-
Odd-One-Out: Anomaly Detection by Comparing with Neighbors
Authors:
Ankan Bhunia,
Changjian Li,
Hakan Bilen
Abstract:
This paper introduces a novel anomaly detection (AD) problem that focuses on identifying `odd-looking' objects relative to the other instances within a scene. Unlike the traditional AD benchmarks, in our setting, anomalies in this context are scene-specific, defined by the regular instances that make up the majority. Since object instances are often partly visible from a single viewpoint, our sett…
▽ More
This paper introduces a novel anomaly detection (AD) problem that focuses on identifying `odd-looking' objects relative to the other instances within a scene. Unlike the traditional AD benchmarks, in our setting, anomalies in this context are scene-specific, defined by the regular instances that make up the majority. Since object instances are often partly visible from a single viewpoint, our setting provides multiple views of each scene as input. To provide a testbed for future research in this task, we introduce two benchmarks, ToysAD-8K and PartsAD-15K. We propose a novel method that generates 3D object-centric representations for each instance and detects the anomalous ones through a cross-examination between the instances. We rigorously analyze our method quantitatively and qualitatively in the presented benchmarks.
△ Less
Submitted 28 June, 2024;
originally announced June 2024.
-
Looking 3D: Anomaly Detection with 2D-3D Alignment
Authors:
Ankan Bhunia,
Changjian Li,
Hakan Bilen
Abstract:
Automatic anomaly detection based on visual cues holds practical significance in various domains, such as manufacturing and product quality assessment. This paper introduces a new conditional anomaly detection problem, which involves identifying anomalies in a query image by comparing it to a reference shape. To address this challenge, we have created a large dataset, BrokenChairs-180K, consisting…
▽ More
Automatic anomaly detection based on visual cues holds practical significance in various domains, such as manufacturing and product quality assessment. This paper introduces a new conditional anomaly detection problem, which involves identifying anomalies in a query image by comparing it to a reference shape. To address this challenge, we have created a large dataset, BrokenChairs-180K, consisting of around 180K images, with diverse anomalies, geometries, and textures paired with 8,143 reference 3D shapes. To tackle this task, we have proposed a novel transformer-based approach that explicitly learns the correspondence between the query image and reference 3D shape via feature alignment and leverages a customized attention mechanism for anomaly detection. Our approach has been rigorously evaluated through comprehensive experiments, serving as a benchmark for future research in this domain.
△ Less
Submitted 27 June, 2024;
originally announced June 2024.
-
SketchDeco: Decorating B&W Sketches with Colour
Authors:
Chaitat Utintu,
Pinaki Nath Chowdhury,
Aneeshan Sain,
Subhadeep Koley,
Ayan Kumar Bhunia,
Yi-Zhe Song
Abstract:
This paper introduces a novel approach to sketch colourisation, inspired by the universal childhood activity of colouring and its professional applications in design and story-boarding. Striking a balance between precision and convenience, our method utilises region masks and colour palettes to allow intuitive user control, steering clear of the meticulousness of manual colour assignments or the l…
▽ More
This paper introduces a novel approach to sketch colourisation, inspired by the universal childhood activity of colouring and its professional applications in design and story-boarding. Striking a balance between precision and convenience, our method utilises region masks and colour palettes to allow intuitive user control, steering clear of the meticulousness of manual colour assignments or the limitations of textual prompts. By strategically combining ControlNet and staged generation, incorporating Stable Diffusion v1.5, and leveraging BLIP-2 text prompts, our methodology facilitates faithful image generation and user-directed colourisation. Addressing challenges of local and global consistency, we employ inventive solutions such as an inversion scheme, guided sampling, and a self-attention mechanism with a scaling factor. The resulting tool is not only fast and training-free but also compatible with consumer-grade Nvidia RTX 4090 Super GPUs, making it a valuable asset for both creative professionals and enthusiasts in various fields. Project Page: \url{https://chaitron.github.io/SketchDeco/}
△ Less
Submitted 28 May, 2024;
originally announced May 2024.
-
Strong quantum nonlocality: Unextendible biseparability beyond unextendible product basis
Authors:
Atanu Bhunia,
Subrata Bera,
Indranil Biswas,
Indrani Chattopadhyay,
Debasis Sarkar
Abstract:
An unextendible biseparable basis (UBB) is a set of orthogonal pure biseparable states which span a subspace of a given Hilbert space while the complementary subspace contains only genuinely entangled states. These biseparable bases are useful to produce genuinely entangled subspace in multipartite system. Such a subspace could be more beneficial for information theoretic applications if we are ab…
▽ More
An unextendible biseparable basis (UBB) is a set of orthogonal pure biseparable states which span a subspace of a given Hilbert space while the complementary subspace contains only genuinely entangled states. These biseparable bases are useful to produce genuinely entangled subspace in multipartite system. Such a subspace could be more beneficial for information theoretic applications if we are able to extract distillable entanglement across every bipartition from each state of this subspace. In this manuscript, we have derived a rule for constructing such a class of UBB which exhibits the phenomenon of strong quantum nonlocality. This result positively answers the open problem raised by Agrawal et al. [Phys. Rev. A 99, 032335 (2019)]; that there exists a UBB which can demonstrate the phenomenon of strong quantum nonlocality in the perspective of local irreducibility paradigm.
△ Less
Submitted 8 April, 2024;
originally announced April 2024.
-
What Sketch Explainability Really Means for Downstream Tasks
Authors:
Hmrishav Bandyopadhyay,
Pinaki Nath Chowdhury,
Ayan Kumar Bhunia,
Aneeshan Sain,
Tao Xiang,
Yi-Zhe Song
Abstract:
In this paper, we explore the unique modality of sketch for explainability, emphasising the profound impact of human strokes compared to conventional pixel-oriented studies. Beyond explanations of network behavior, we discern the genuine implications of explainability across diverse downstream sketch-related tasks. We propose a lightweight and portable explainability solution -- a seamless plugin…
▽ More
In this paper, we explore the unique modality of sketch for explainability, emphasising the profound impact of human strokes compared to conventional pixel-oriented studies. Beyond explanations of network behavior, we discern the genuine implications of explainability across diverse downstream sketch-related tasks. We propose a lightweight and portable explainability solution -- a seamless plugin that integrates effortlessly with any pre-trained model, eliminating the need for re-training. Demonstrating its adaptability, we present four applications: highly studied retrieval and generation, and completely novel assisted drawing and sketch adversarial attacks. The centrepiece to our solution is a stroke-level attribution map that takes different forms when linked with downstream tasks. By addressing the inherent non-differentiability of rasterisation, we enable explanations at both coarse stroke level (SLA) and partial stroke level (P-SLA), each with its advantages for specific downstream tasks.
△ Less
Submitted 14 March, 2024;
originally announced March 2024.
-
SketchINR: A First Look into Sketches as Implicit Neural Representations
Authors:
Hmrishav Bandyopadhyay,
Ayan Kumar Bhunia,
Pinaki Nath Chowdhury,
Aneeshan Sain,
Tao Xiang,
Timothy Hospedales,
Yi-Zhe Song
Abstract:
We propose SketchINR, to advance the representation of vector sketches with implicit neural models. A variable length vector sketch is compressed into a latent space of fixed dimension that implicitly encodes the underlying shape as a function of time and strokes. The learned function predicts the $xy$ point coordinates in a sketch at each time and stroke. Despite its simplicity, SketchINR outperf…
▽ More
We propose SketchINR, to advance the representation of vector sketches with implicit neural models. A variable length vector sketch is compressed into a latent space of fixed dimension that implicitly encodes the underlying shape as a function of time and strokes. The learned function predicts the $xy$ point coordinates in a sketch at each time and stroke. Despite its simplicity, SketchINR outperforms existing representations at multiple tasks: (i) Encoding an entire sketch dataset into a fixed size latent vector, SketchINR gives $60\times$ and $10\times$ data compression over raster and vector sketches, respectively. (ii) SketchINR's auto-decoder provides a much higher-fidelity representation than other learned vector sketch representations, and is uniquely able to scale to complex vector sketches such as FS-COCO. (iii) SketchINR supports parallelisation that can decode/render $\sim$$100\times$ faster than other learned vector representations such as SketchRNN. (iv) SketchINR, for the first time, emulates the human ability to reproduce a sketch with varying abstraction in terms of number and complexity of strokes. As a first look at implicit sketches, SketchINR's compact high-fidelity representation will support future work in modelling long and complex sketches.
△ Less
Submitted 14 March, 2024;
originally announced March 2024.
-
Dielectric microwave resonator with large optical apertures for spin-based quantum devices
Authors:
Tatsuki Hamamoto,
Amit Bhunia,
Rupak Kumar Bhattacharya,
Hiroki Takahashi,
Yuimaru Kubo
Abstract:
Towards a spin-based quantum microwave-optical photon transducer, we demonstrate a low-loss dielectric microwave resonator with an internal quality factor of $2.30\times10^4$ while accommodating optical apertures with a diameter of $8\, \mathrm{mm}$. The two seemingly conflicting requirements, high quality factor and large optical apertures, are satisfied thanks to the large dielectric constant of…
▽ More
Towards a spin-based quantum microwave-optical photon transducer, we demonstrate a low-loss dielectric microwave resonator with an internal quality factor of $2.30\times10^4$ while accommodating optical apertures with a diameter of $8\, \mathrm{mm}$. The two seemingly conflicting requirements, high quality factor and large optical apertures, are satisfied thanks to the large dielectric constant of rutile ($\mathrm{TiO_2}$). The quality factor is limited by radiation loss, and we confirmed by numerical simulation that this dielectric resonator can achieve a quality factor exceeding $10^6$ by extending the height of the resonator enclosure. Using this resonator, we performed both continuous-wave (cw) and pulse electron spin resonance (ESR) spectroscopy on 2,2-diphenyl-1-picrylhydrazyl (DPPH) crystalline powder and P1 centers in a diamond crystal in a dilution refrigerator. The cw ESR spectroscopy demonstrated high-cooperativity and strong spin-resonator coupling with the DPPH and P1 centers respectively, while the pulse ESR spectroscopy successfully measured longitudinal and transverse relaxation times.
△ Less
Submitted 13 March, 2024;
originally announced March 2024.
-
It's All About Your Sketch: Democratising Sketch Control in Diffusion Models
Authors:
Subhadeep Koley,
Ayan Kumar Bhunia,
Deeptanshu Sekhri,
Aneeshan Sain,
Pinaki Nath Chowdhury,
Tao Xiang,
Yi-Zhe Song
Abstract:
This paper unravels the potential of sketches for diffusion models, addressing the deceptive promise of direct sketch control in generative AI. We importantly democratise the process, enabling amateur sketches to generate precise images, living up to the commitment of "what you sketch is what you get". A pilot study underscores the necessity, revealing that deformities in existing models stem from…
▽ More
This paper unravels the potential of sketches for diffusion models, addressing the deceptive promise of direct sketch control in generative AI. We importantly democratise the process, enabling amateur sketches to generate precise images, living up to the commitment of "what you sketch is what you get". A pilot study underscores the necessity, revealing that deformities in existing models stem from spatial-conditioning. To rectify this, we propose an abstraction-aware framework, utilising a sketch adapter, adaptive time-step sampling, and discriminative guidance from a pre-trained fine-grained sketch-based image retrieval model, working synergistically to reinforce fine-grained sketch-photo association. Our approach operates seamlessly during inference without the need for textual prompts; a simple, rough sketch akin to what you and I can create suffices! We welcome everyone to examine results presented in the paper and its supplementary. Contributions include democratising sketch control, introducing an abstraction-aware framework, and leveraging discriminative guidance, validated through extensive experiments.
△ Less
Submitted 20 March, 2024; v1 submitted 11 March, 2024;
originally announced March 2024.
-
You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval
Authors:
Subhadeep Koley,
Ayan Kumar Bhunia,
Aneeshan Sain,
Pinaki Nath Chowdhury,
Tao Xiang,
Yi-Zhe Song
Abstract:
Two primary input modalities prevail in image retrieval: sketch and text. While text is widely used for inter-category retrieval tasks, sketches have been established as the sole preferred modality for fine-grained image retrieval due to their ability to capture intricate visual details. In this paper, we question the reliance on sketches alone for fine-grained image retrieval by simultaneously ex…
▽ More
Two primary input modalities prevail in image retrieval: sketch and text. While text is widely used for inter-category retrieval tasks, sketches have been established as the sole preferred modality for fine-grained image retrieval due to their ability to capture intricate visual details. In this paper, we question the reliance on sketches alone for fine-grained image retrieval by simultaneously exploring the fine-grained representation capabilities of both sketch and text, orchestrating a duet between the two. The end result enables precise retrievals previously unattainable, allowing users to pose ever-finer queries and incorporate attributes like colour and contextual cues from text. For this purpose, we introduce a novel compositionality framework, effectively combining sketches and text using pre-trained CLIP models, while eliminating the need for extensive fine-grained textual descriptions. Last but not least, our system extends to novel applications in composed image retrieval, domain attribute transfer, and fine-grained generation, providing solutions for various real-world scenarios.
△ Less
Submitted 20 March, 2024; v1 submitted 11 March, 2024;
originally announced March 2024.
-
Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers
Authors:
Subhadeep Koley,
Ayan Kumar Bhunia,
Aneeshan Sain,
Pinaki Nath Chowdhury,
Tao Xiang,
Yi-Zhe Song
Abstract:
This paper, for the first time, explores text-to-image diffusion models for Zero-Shot Sketch-based Image Retrieval (ZS-SBIR). We highlight a pivotal discovery: the capacity of text-to-image diffusion models to seamlessly bridge the gap between sketches and photos. This proficiency is underpinned by their robust cross-modal capabilities and shape bias, findings that are substantiated through our pi…
▽ More
This paper, for the first time, explores text-to-image diffusion models for Zero-Shot Sketch-based Image Retrieval (ZS-SBIR). We highlight a pivotal discovery: the capacity of text-to-image diffusion models to seamlessly bridge the gap between sketches and photos. This proficiency is underpinned by their robust cross-modal capabilities and shape bias, findings that are substantiated through our pilot studies. In order to harness pre-trained diffusion models effectively, we introduce a straightforward yet powerful strategy focused on two key aspects: selecting optimal feature layers and utilising visual and textual prompts. For the former, we identify which layers are most enriched with information and are best suited for the specific retrieval requirements (category-level or fine-grained). Then we employ visual and textual prompts to guide the model's feature extraction process, enabling it to generate more discriminative and contextually relevant cross-modal representations. Extensive experiments on several benchmark datasets validate significant performance improvements.
△ Less
Submitted 20 March, 2024; v1 submitted 11 March, 2024;
originally announced March 2024.
-
How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval?
Authors:
Subhadeep Koley,
Ayan Kumar Bhunia,
Aneeshan Sain,
Pinaki Nath Chowdhury,
Tao Xiang,
Yi-Zhe Song
Abstract:
In this paper, we propose a novel abstraction-aware sketch-based image retrieval framework capable of handling sketch abstraction at varied levels. Prior works had mainly focused on tackling sub-factors such as drawing style and order, we instead attempt to model abstraction as a whole, and propose feature-level and retrieval granularity-level designs so that the system builds into its DNA the nec…
▽ More
In this paper, we propose a novel abstraction-aware sketch-based image retrieval framework capable of handling sketch abstraction at varied levels. Prior works had mainly focused on tackling sub-factors such as drawing style and order, we instead attempt to model abstraction as a whole, and propose feature-level and retrieval granularity-level designs so that the system builds into its DNA the necessary means to interpret abstraction. On learning abstraction-aware features, we for the first-time harness the rich semantic embedding of pre-trained StyleGAN model, together with a novel abstraction-level mapper that deciphers the level of abstraction and dynamically selects appropriate dimensions in the feature matrix correspondingly, to construct a feature matrix embedding that can be freely traversed to accommodate different levels of abstraction. For granularity-level abstraction understanding, we dictate that the retrieval model should not treat all abstraction-levels equally and introduce a differentiable surrogate Acc.@q loss to inject that understanding into the system. Different to the gold-standard triplet loss, our Acc.@q loss uniquely allows a sketch to narrow/broaden its focus in terms of how stringent the evaluation should be - the more abstract a sketch, the less stringent (higher q). Extensive experiments depict our method to outperform existing state-of-the-arts in standard SBIR tasks along with challenging scenarios like early retrieval, forensic sketch-photo matching, and style-invariant retrieval.
△ Less
Submitted 20 March, 2024; v1 submitted 11 March, 2024;
originally announced March 2024.
-
DemoCaricature: Democratising Caricature Generation with a Rough Sketch
Authors:
Dar-Yen Chen,
Ayan Kumar Bhunia,
Subhadeep Koley,
Aneeshan Sain,
Pinaki Nath Chowdhury,
Yi-Zhe Song
Abstract:
In this paper, we democratise caricature generation, empowering individuals to effortlessly craft personalised caricatures with just a photo and a conceptual sketch. Our objective is to strike a delicate balance between abstraction and identity, while preserving the creativity and subjectivity inherent in a sketch. To achieve this, we present Explicit Rank-1 Model Editing alongside single-image pe…
▽ More
In this paper, we democratise caricature generation, empowering individuals to effortlessly craft personalised caricatures with just a photo and a conceptual sketch. Our objective is to strike a delicate balance between abstraction and identity, while preserving the creativity and subjectivity inherent in a sketch. To achieve this, we present Explicit Rank-1 Model Editing alongside single-image personalisation, selectively applying nuanced edits to cross-attention layers for a seamless merge of identity and style. Additionally, we propose Random Mask Reconstruction to enhance robustness, directing the model to focus on distinctive identity and style features. Crucially, our aim is not to replace artists but to eliminate accessibility barriers, allowing enthusiasts to engage in the artistry.
△ Less
Submitted 24 March, 2024; v1 submitted 7 December, 2023;
originally announced December 2023.
-
Doodle Your 3D: From Abstract Freehand Sketches to Precise 3D Shapes
Authors:
Hmrishav Bandyopadhyay,
Subhadeep Koley,
Ayan Das,
Ayan Kumar Bhunia,
Aneeshan Sain,
Pinaki Nath Chowdhury,
Tao Xiang,
Yi-Zhe Song
Abstract:
In this paper, we democratise 3D content creation, enabling precise generation of 3D shapes from abstract sketches while overcoming limitations tied to drawing skills. We introduce a novel part-level modelling and alignment framework that facilitates abstraction modelling and cross-modal correspondence. Leveraging the same part-level decoder, our approach seamlessly extends to sketch modelling by…
▽ More
In this paper, we democratise 3D content creation, enabling precise generation of 3D shapes from abstract sketches while overcoming limitations tied to drawing skills. We introduce a novel part-level modelling and alignment framework that facilitates abstraction modelling and cross-modal correspondence. Leveraging the same part-level decoder, our approach seamlessly extends to sketch modelling by establishing correspondence between CLIPasso edgemaps and projected 3D part regions, eliminating the need for a dataset pairing human sketches and 3D shapes. Additionally, our method introduces a seamless in-position editing process as a byproduct of cross-modal part-aligned modelling. Operating in a low-dimensional implicit space, our approach significantly reduces computational demands and processing time.
△ Less
Submitted 7 June, 2024; v1 submitted 7 December, 2023;
originally announced December 2023.
-
Site-specific stable deterministic single photon emitters with low Huang-Rhys value in layered hexagonal boron nitride at room temperature
Authors:
Amit Bhunia,
Pragya Joshi,
Nitesh Singh,
Biswanath Chakraborty,
Rajesh V Nair
Abstract:
Development of stable room-temperature bright single-photon emitters using atomic defects in hexagonal-boron nitride flakes (h-BN) provides significant promises for quantum technologies. However, an outstanding challenge in h-BN is creating site-specific, stable, high emission rate single photon emitters with very low Huang-Rhys (HR) factor. Here, we discuss the photonic properties of site-specifi…
▽ More
Development of stable room-temperature bright single-photon emitters using atomic defects in hexagonal-boron nitride flakes (h-BN) provides significant promises for quantum technologies. However, an outstanding challenge in h-BN is creating site-specific, stable, high emission rate single photon emitters with very low Huang-Rhys (HR) factor. Here, we discuss the photonic properties of site-specific, isolated, stable quantum emitter that emit single photons with a high emission rate and unprecedented low HR value of 0.6 at room temperature. Scanning confocal image confirms site-specific single photon emitter with a prominent zero-phonon line at ~578 nm with saturation photon counts of 105 counts/second. The second-order intensity-intensity correlation measurement shows an anti-bunching dip of ~0.25 with an emission lifetime of 2.46 ns. Low-energy electron beam irradiation and subsequent annealing are important to achieve stable single photon emitters.
△ Less
Submitted 21 July, 2023;
originally announced July 2023.
-
Cross-modulated Few-shot Image Generation for Colorectal Tissue Classification
Authors:
Amandeep Kumar,
Ankan kumar Bhunia,
Sanath Narayan,
Hisham Cholakkal,
Rao Muhammad Anwer,
Jorma Laaksonen,
Fahad Shahbaz Khan
Abstract:
In this work, we propose a few-shot colorectal tissue image generation method for addressing the scarcity of histopathological training data for rare cancer tissues. Our few-shot generation method, named XM-GAN, takes one base and a pair of reference tissue images as input and generates high-quality yet diverse images. Within our XM-GAN, a novel controllable fusion block densely aggregates local r…
▽ More
In this work, we propose a few-shot colorectal tissue image generation method for addressing the scarcity of histopathological training data for rare cancer tissues. Our few-shot generation method, named XM-GAN, takes one base and a pair of reference tissue images as input and generates high-quality yet diverse images. Within our XM-GAN, a novel controllable fusion block densely aggregates local regions of reference images based on their similarity to those in the base image, resulting in locally consistent features. To the best of our knowledge, we are the first to investigate few-shot generation in colorectal tissue images. We evaluate our few-shot colorectral tissue image generation by performing extensive qualitative, quantitative and subject specialist (pathologist) based evaluations. Specifically, in specialist-based evaluation, pathologists could differentiate between our XM-GAN generated tissue images and real images only 55% time. Moreover, we utilize these generated images as data augmentation to address the few-shot tissue image classification task, achieving a gain of 4.4% in terms of mean accuracy over the vanilla few-shot classifier. Code: \url{https://github.com/VIROBO-15/XM-GAN}
△ Less
Submitted 4 July, 2023; v1 submitted 4 April, 2023;
originally announced April 2023.
-
Generative Multiplane Neural Radiance for 3D-Aware Image Generation
Authors:
Amandeep Kumar,
Ankan Kumar Bhunia,
Sanath Narayan,
Hisham Cholakkal,
Rao Muhammad Anwer,
Salman Khan,
Ming-Hsuan Yang,
Fahad Shahbaz Khan
Abstract:
We present a method to efficiently generate 3D-aware high-resolution images that are view-consistent across multiple target views. The proposed multiplane neural radiance model, named GMNR, consists of a novel α-guided view-dependent representation (α-VdR) module for learning view-dependent information. The α-VdR module, faciliated by an α-guided pixel sampling technique, computes the view-depende…
▽ More
We present a method to efficiently generate 3D-aware high-resolution images that are view-consistent across multiple target views. The proposed multiplane neural radiance model, named GMNR, consists of a novel α-guided view-dependent representation (α-VdR) module for learning view-dependent information. The α-VdR module, faciliated by an α-guided pixel sampling technique, computes the view-dependent representation efficiently by learning viewing direction and position coefficients. Moreover, we propose a view-consistency loss to enforce photometric similarity across multiple views. The GMNR model can generate 3D-aware high-resolution images that are viewconsistent across multiple camera poses, while maintaining the computational efficiency in terms of both training and inference time. Experiments on three datasets demonstrate the effectiveness of the proposed modules, leading to favorable results in terms of both generation quality and inference time, compared to existing approaches. Our GMNR model generates 3D-aware images of 1024 X 1024 pixels with 17.6 FPS on a single V100. Code : https://github.com/VIROBO-15/GMNR
△ Less
Submitted 3 April, 2023;
originally announced April 2023.
-
What Can Human Sketches Do for Object Detection?
Authors:
Pinaki Nath Chowdhury,
Ayan Kumar Bhunia,
Aneeshan Sain,
Subhadeep Koley,
Tao Xiang,
Yi-Zhe Song
Abstract:
Sketches are highly expressive, inherently capturing subjective and fine-grained visual cues. The exploration of such innate properties of human sketches has, however, been limited to that of image retrieval. In this paper, for the first time, we cultivate the expressiveness of sketches but for the fundamental vision task of object detection. The end result is a sketch-enabled object detection fra…
▽ More
Sketches are highly expressive, inherently capturing subjective and fine-grained visual cues. The exploration of such innate properties of human sketches has, however, been limited to that of image retrieval. In this paper, for the first time, we cultivate the expressiveness of sketches but for the fundamental vision task of object detection. The end result is a sketch-enabled object detection framework that detects based on what \textit{you} sketch -- \textit{that} ``zebra'' (e.g., one that is eating the grass) in a herd of zebras (instance-aware detection), and only the \textit{part} (e.g., ``head" of a ``zebra") that you desire (part-aware detection). We further dictate that our model works without (i) knowing which category to expect at testing (zero-shot) and (ii) not requiring additional bounding boxes (as per fully supervised) and class labels (as per weakly supervised). Instead of devising a model from the ground up, we show an intuitive synergy between foundation models (e.g., CLIP) and existing sketch models build for sketch-based image retrieval (SBIR), which can already elegantly solve the task -- CLIP to provide model generalisation, and SBIR to bridge the (sketch$\rightarrow$photo) gap. In particular, we first perform independent prompting on both sketch and photo branches of an SBIR model to build highly generalisable sketch and photo encoders on the back of the generalisation ability of CLIP. We then devise a training paradigm to adapt the learned encoders for object detection, such that the region embeddings of detected boxes are aligned with the sketch and photo embeddings from SBIR. Evaluating our framework on standard object detection datasets like PASCAL-VOC and MS-COCO outperforms both supervised (SOD) and weakly-supervised object detectors (WSOD) on zero-shot setups. Project Page: \url{https://pinakinathc.github.io/sketch-detect}
△ Less
Submitted 28 October, 2023; v1 submitted 27 March, 2023;
originally announced March 2023.
-
Exploiting Unlabelled Photos for Stronger Fine-Grained SBIR
Authors:
Aneeshan Sain,
Ayan Kumar Bhunia,
Subhadeep Koley,
Pinaki Nath Chowdhury,
Soumitri Chattopadhyay,
Tao Xiang,
Yi-Zhe Song
Abstract:
This paper advances the fine-grained sketch-based image retrieval (FG-SBIR) literature by putting forward a strong baseline that overshoots prior state-of-the-arts by ~11%. This is not via complicated design though, but by addressing two critical issues facing the community (i) the gold standard triplet loss does not enforce holistic latent space geometry, and (ii) there are never enough sketches…
▽ More
This paper advances the fine-grained sketch-based image retrieval (FG-SBIR) literature by putting forward a strong baseline that overshoots prior state-of-the-arts by ~11%. This is not via complicated design though, but by addressing two critical issues facing the community (i) the gold standard triplet loss does not enforce holistic latent space geometry, and (ii) there are never enough sketches to train a high accuracy model. For the former, we propose a simple modification to the standard triplet loss, that explicitly enforces separation amongst photos/sketch instances. For the latter, we put forward a novel knowledge distillation module can leverage photo data for model training. Both modules are then plugged into a novel plug-n-playable training paradigm that allows for more stable training. More specifically, for (i) we employ an intra-modal triplet loss amongst sketches to bring sketches of the same instance closer from others, and one more amongst photos to push away different photo instances while bringing closer a structurally augmented version of the same photo (offering a gain of ~4-6%). To tackle (ii), we first pre-train a teacher on the large set of unlabelled photos over the aforementioned intra-modal photo triplet loss. Then we distill the contextual similarity present amongst the instances in the teacher's embedding space to that in the student's embedding space, by matching the distribution over inter-feature distances of respective samples in both embedding spaces (delivering a further gain of ~4-5%). Apart from outperforming prior arts significantly, our model also yields satisfactory results on generalising to new classes. Project page: https://aneeshan95.github.io/Sketch_PVT/
△ Less
Submitted 23 March, 2023;
originally announced March 2023.
-
More assistance of entanglement, less rounds of classical communication
Authors:
Atanu Bhunia,
Indranil Biswas,
Indrani Chattopadhyay,
Debasis Sarkar
Abstract:
Classical communication plays a crucial role to distinguish locally a class of quantum states. Despite considerable advances, we have very little knowledge about the number of measurement and communication rounds needed to implement a discrimination task by local quantum operations and classical communications (in short, LOCC). In this letter, we are able to show the relation between round numbers…
▽ More
Classical communication plays a crucial role to distinguish locally a class of quantum states. Despite considerable advances, we have very little knowledge about the number of measurement and communication rounds needed to implement a discrimination task by local quantum operations and classical communications (in short, LOCC). In this letter, we are able to show the relation between round numbers with the local discrimination of a set of pure bipartite orthogonal quantum states. To demonstrate the possible strong dependence on the round numbers, we consider a class of orthogonal product states in $d\otimes d$, which require at least $2d-2$ round of classical communications. Curiously the round number can be reduced to $d$ by the assistance of one-ebit of entanglement as resource and can be reduced further by assistance of more entanglement. We are also able to show that the number of LOCC rounds needed for a discrimination task may depend on the amount of entanglement assistances.
△ Less
Submitted 23 March, 2023;
originally announced March 2023.
-
CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Not
Authors:
Aneeshan Sain,
Ayan Kumar Bhunia,
Pinaki Nath Chowdhury,
Subhadeep Koley,
Tao Xiang,
Yi-Zhe Song
Abstract:
In this paper, we leverage CLIP for zero-shot sketch based image retrieval (ZS-SBIR). We are largely inspired by recent advances on foundation models and the unparalleled generalisation ability they seem to offer, but for the first time tailor it to benefit the sketch community. We put forward novel designs on how best to achieve this synergy, for both the category setting and the fine-grained set…
▽ More
In this paper, we leverage CLIP for zero-shot sketch based image retrieval (ZS-SBIR). We are largely inspired by recent advances on foundation models and the unparalleled generalisation ability they seem to offer, but for the first time tailor it to benefit the sketch community. We put forward novel designs on how best to achieve this synergy, for both the category setting and the fine-grained setting ("all"). At the very core of our solution is a prompt learning setup. First we show just via factoring in sketch-specific prompts, we already have a category-level ZS-SBIR system that overshoots all prior arts, by a large margin (24.8%) - a great testimony on studying the CLIP and ZS-SBIR synergy. Moving onto the fine-grained setup is however trickier, and requires a deeper dive into this synergy. For that, we come up with two specific designs to tackle the fine-grained matching nature of the problem: (i) an additional regularisation loss to ensure the relative separation between sketches and photos is uniform across categories, which is not the case for the gold standard standalone triplet loss, and (ii) a clever patch shuffling technique to help establishing instance-level structural correspondences between sketch-photo pairs. With these designs, we again observe significant performance gains in the region of 26.9% over previous state-of-the-art. The take-home message, if any, is the proposed CLIP and prompt learning paradigm carries great promise in tackling other sketch-related tasks (not limited to ZS-SBIR) where data scarcity remains a great challenge. Project page: https://aneeshan95.github.io/Sketch_LVM/
△ Less
Submitted 27 March, 2023; v1 submitted 23 March, 2023;
originally announced March 2023.
-
Sketch2Saliency: Learning to Detect Salient Objects from Human Drawings
Authors:
Ayan Kumar Bhunia,
Subhadeep Koley,
Amandeep Kumar,
Aneeshan Sain,
Pinaki Nath Chowdhury,
Tao Xiang,
Yi-Zhe Song
Abstract:
Human sketch has already proved its worth in various visual understanding tasks (e.g., retrieval, segmentation, image-captioning, etc). In this paper, we reveal a new trait of sketches - that they are also salient. This is intuitive as sketching is a natural attentive process at its core. More specifically, we aim to study how sketches can be used as a weak label to detect salient objects present…
▽ More
Human sketch has already proved its worth in various visual understanding tasks (e.g., retrieval, segmentation, image-captioning, etc). In this paper, we reveal a new trait of sketches - that they are also salient. This is intuitive as sketching is a natural attentive process at its core. More specifically, we aim to study how sketches can be used as a weak label to detect salient objects present in an image. To this end, we propose a novel method that emphasises on how "salient object" could be explained by hand-drawn sketches. To accomplish this, we introduce a photo-to-sketch generation model that aims to generate sequential sketch coordinates corresponding to a given visual photo through a 2D attention mechanism. Attention maps accumulated across the time steps give rise to salient regions in the process. Extensive quantitative and qualitative experiments prove our hypothesis and delineate how our sketch-based saliency detection model gives a competitive performance compared to the state-of-the-art.
△ Less
Submitted 30 March, 2023; v1 submitted 20 March, 2023;
originally announced March 2023.
-
Picture that Sketch: Photorealistic Image Generation from Abstract Sketches
Authors:
Subhadeep Koley,
Ayan Kumar Bhunia,
Aneeshan Sain,
Pinaki Nath Chowdhury,
Tao Xiang,
Yi-Zhe Song
Abstract:
Given an abstract, deformed, ordinary sketch from untrained amateurs like you and me, this paper turns it into a photorealistic image - just like those shown in Fig. 1(a), all non-cherry-picked. We differ significantly from prior art in that we do not dictate an edgemap-like sketch to start with, but aim to work with abstract free-hand human sketches. In doing so, we essentially democratise the sk…
▽ More
Given an abstract, deformed, ordinary sketch from untrained amateurs like you and me, this paper turns it into a photorealistic image - just like those shown in Fig. 1(a), all non-cherry-picked. We differ significantly from prior art in that we do not dictate an edgemap-like sketch to start with, but aim to work with abstract free-hand human sketches. In doing so, we essentially democratise the sketch-to-photo pipeline, "picturing" a sketch regardless of how good you sketch. Our contribution at the outset is a decoupled encoder-decoder training paradigm, where the decoder is a StyleGAN trained on photos only. This importantly ensures that generated results are always photorealistic. The rest is then all centred around how best to deal with the abstraction gap between sketch and photo. For that, we propose an autoregressive sketch mapper trained on sketch-photo pairs that maps a sketch to the StyleGAN latent space. We further introduce specific designs to tackle the abstract nature of human sketches, including a fine-grained discriminative loss on the back of a trained sketch-photo retrieval model, and a partial-aware sketch augmentation strategy. Finally, we showcase a few downstream tasks our generation model enables, amongst them is showing how fine-grained sketch-based image retrieval, a well-studied problem in the sketch community, can be reduced to an image (generated) to image retrieval task, surpassing state-of-the-arts. We put forward generated results in the supplementary for everyone to scrutinise.
△ Less
Submitted 30 March, 2023; v1 submitted 20 March, 2023;
originally announced March 2023.
-
Data-Free Sketch-Based Image Retrieval
Authors:
Abhra Chaudhuri,
Ayan Kumar Bhunia,
Yi-Zhe Song,
Anjan Dutta
Abstract:
Rising concerns about privacy and anonymity preservation of deep learning models have facilitated research in data-free learning (DFL). For the first time, we identify that for data-scarce tasks like Sketch-Based Image Retrieval (SBIR), where the difficulty in acquiring paired photos and hand-drawn sketches limits data-dependent cross-modal learning algorithms, DFL can prove to be a much more prac…
▽ More
Rising concerns about privacy and anonymity preservation of deep learning models have facilitated research in data-free learning (DFL). For the first time, we identify that for data-scarce tasks like Sketch-Based Image Retrieval (SBIR), where the difficulty in acquiring paired photos and hand-drawn sketches limits data-dependent cross-modal learning algorithms, DFL can prove to be a much more practical paradigm. We thus propose Data-Free (DF)-SBIR, where, unlike existing DFL problems, pre-trained, single-modality classification models have to be leveraged to learn a cross-modal metric-space for retrieval without access to any training data. The widespread availability of pre-trained classification models, along with the difficulty in acquiring paired photo-sketch datasets for SBIR justify the practicality of this setting. We present a methodology for DF-SBIR, which can leverage knowledge from models independently trained to perform classification on photos and sketches. We evaluate our model on the Sketchy, TU-Berlin, and QuickDraw benchmarks, designing a variety of baselines based on state-of-the-art DFL literature, and observe that our method surpasses all of them by significant margins. Our method also achieves mAPs competitive with data-dependent approaches, all the while requiring no training data. Implementation is available at \url{https://github.com/abhrac/data-free-sbir}.
△ Less
Submitted 14 March, 2023;
originally announced March 2023.
-
Entangled state distillation from single copy mixed states beyond LOCC
Authors:
Indranil Biswas,
Atanu Bhunia,
Indrani Chattopadhyay,
Debasis Sarkar
Abstract:
No pure entangled state can be distilled from a $2\otimes 2$ or $2\otimes 3$ mixed state by separable operations. In $3\otimes 3$, pure entanglement can be distilled by separable operation but not by LOCC. In this letter, we proved the conjecture [PRL. 103, 110502 (2009)] that it is possible to distill pure entanglement for $2\otimes 4$ system by LOCC and further improve these in higher dimensions…
▽ More
No pure entangled state can be distilled from a $2\otimes 2$ or $2\otimes 3$ mixed state by separable operations. In $3\otimes 3$, pure entanglement can be distilled by separable operation but not by LOCC. In this letter, we proved the conjecture [PRL. 103, 110502 (2009)] that it is possible to distill pure entanglement for $2\otimes 4$ system by LOCC and further improve these in higher dimensions to distill a pure entangled state of Schmidt rank $d$ from a $m\otimes n$ mixed state by separable operation when $m+n \geqslant 3d$. We found results for tripartite systems with target state $d$-level GHZ-type state. These results provide a class of systems where separable operation is strictly stronger than LOCC.
△ Less
Submitted 20 December, 2022;
originally announced December 2022.
-
Person Image Synthesis via Denoising Diffusion Model
Authors:
Ankan Kumar Bhunia,
Salman Khan,
Hisham Cholakkal,
Rao Muhammad Anwer,
Jorma Laaksonen,
Mubarak Shah,
Fahad Shahbaz Khan
Abstract:
The pose-guided person image generation task requires synthesizing photorealistic images of humans in arbitrary poses. The existing approaches use generative adversarial networks that do not necessarily maintain realistic textures or need dense correspondences that struggle to handle complex deformations and severe occlusions. In this work, we show how denoising diffusion models can be applied for…
▽ More
The pose-guided person image generation task requires synthesizing photorealistic images of humans in arbitrary poses. The existing approaches use generative adversarial networks that do not necessarily maintain realistic textures or need dense correspondences that struggle to handle complex deformations and severe occlusions. In this work, we show how denoising diffusion models can be applied for high-fidelity person image synthesis with strong sample diversity and enhanced mode coverage of the learnt data distribution. Our proposed Person Image Diffusion Model (PIDM) disintegrates the complex transfer problem into a series of simpler forward-backward denoising steps. This helps in learning plausible source-to-target transformation trajectories that result in faithful textures and undistorted appearance details. We introduce a 'texture diffusion module' based on cross-attention to accurately model the correspondences between appearance and pose information available in source and target images. Further, we propose 'disentangled classifier-free guidance' to ensure close resemblance between the conditional inputs and the synthesized output in terms of both pose and appearance information. Our extensive results on two large-scale benchmarks and a user study demonstrate the photorealism of our proposed approach under challenging scenarios. We also show how our generated images can help in downstream tasks. Our code and models will be publicly released.
△ Less
Submitted 28 February, 2023; v1 submitted 22 November, 2022;
originally announced November 2022.
-
Towards Practicality of Sketch-Based Visual Understanding
Authors:
Ayan Kumar Bhunia
Abstract:
Sketches have been used to conceptualise and depict visual objects from pre-historic times. Sketch research has flourished in the past decade, particularly with the proliferation of touchscreen devices. Much of the utilisation of sketch has been anchored around the fact that it can be used to delineate visual concepts universally irrespective of age, race, language, or demography. The fine-grained…
▽ More
Sketches have been used to conceptualise and depict visual objects from pre-historic times. Sketch research has flourished in the past decade, particularly with the proliferation of touchscreen devices. Much of the utilisation of sketch has been anchored around the fact that it can be used to delineate visual concepts universally irrespective of age, race, language, or demography. The fine-grained interactive nature of sketches facilitates the application of sketches to various visual understanding tasks, like image retrieval, image-generation or editing, segmentation, 3D-shape modelling etc. However, sketches are highly abstract and subjective based on the perception of individuals. Although most agree that sketches provide fine-grained control to the user to depict a visual object, many consider sketching a tedious process due to their limited sketching skills compared to other query/support modalities like text/tags. Furthermore, collecting fine-grained sketch-photo association is a significant bottleneck to commercialising sketch applications. Therefore, this thesis aims to progress sketch-based visual understanding towards more practicality.
△ Less
Submitted 26 October, 2022;
originally announced October 2022.
-
Adaptive Fine-Grained Sketch-Based Image Retrieval
Authors:
Ayan Kumar Bhunia,
Aneeshan Sain,
Parth Shah,
Animesh Gupta,
Pinaki Nath Chowdhury,
Tao Xiang,
Yi-Zhe Song
Abstract:
The recent focus on Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) has shifted towards generalising a model to new categories without any training data from them. In real-world applications, however, a trained FG-SBIR model is often applied to both new categories and different human sketchers, i.e., different drawing styles. Although this complicates the generalisation problem, fortunately, a…
▽ More
The recent focus on Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) has shifted towards generalising a model to new categories without any training data from them. In real-world applications, however, a trained FG-SBIR model is often applied to both new categories and different human sketchers, i.e., different drawing styles. Although this complicates the generalisation problem, fortunately, a handful of examples are typically available, enabling the model to adapt to the new category/style. In this paper, we offer a novel perspective -- instead of asking for a model that generalises, we advocate for one that quickly adapts, with just very few samples during testing (in a few-shot manner). To solve this new problem, we introduce a novel model-agnostic meta-learning (MAML) based framework with several key modifications: (1) As a retrieval task with a margin-based contrastive loss, we simplify the MAML training in the inner loop to make it more stable and tractable. (2) The margin in our contrastive loss is also meta-learned with the rest of the model. (3) Three additional regularisation losses are introduced in the outer loop, to make the meta-learned FG-SBIR model more effective for category/style adaptation. Extensive experiments on public datasets suggest a large gain over generalisation and zero-shot based approaches, and a few strong few-shot baselines.
△ Less
Submitted 19 August, 2022; v1 submitted 4 July, 2022;
originally announced July 2022.
-
SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text
Authors:
Pinaki Nath Chowdhury,
Ayan Kumar Bhunia,
Aneeshan Sain,
Subhadeep Koley,
Tao Xiang,
Yi-Zhe Song
Abstract:
In this paper, we extend scene understanding to include that of human sketch. The result is a complete trilogy of scene representation from three diverse and complementary modalities -- sketch, photo, and text. Instead of learning a rigid three-way embedding and be done with it, we focus on learning a flexible joint embedding that fully supports the ``optionality" that this complementarity brings.…
▽ More
In this paper, we extend scene understanding to include that of human sketch. The result is a complete trilogy of scene representation from three diverse and complementary modalities -- sketch, photo, and text. Instead of learning a rigid three-way embedding and be done with it, we focus on learning a flexible joint embedding that fully supports the ``optionality" that this complementarity brings. Our embedding supports optionality on two axes: (i) optionality across modalities -- use any combination of modalities as query for downstream tasks like retrieval, (ii) optionality across tasks -- simultaneously utilising the embedding for either discriminative (e.g., retrieval) or generative tasks (e.g., captioning). This provides flexibility to end-users by exploiting the best of each modality, therefore serving the very purpose behind our proposal of a trilogy in the first place. First, a combination of information-bottleneck and conditional invertible neural networks disentangle the modality-specific component from modality-agnostic in sketch, photo, and text. Second, the modality-agnostic instances from sketch, photo, and text are synergised using a modified cross-attention. Once learned, we show our embedding can accommodate a multi-facet of scene-related tasks, including those enabled for the first time by the inclusion of sketch, all without any task-specific modifications. Project Page: \url{http://www.pinakinathc.me/scenetrilogy}
△ Less
Submitted 26 March, 2023; v1 submitted 25 April, 2022;
originally announced April 2022.
-
Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches
Authors:
Ayan Kumar Bhunia,
Viswanatha Reddy Gajjala,
Subhadeep Koley,
Rohit Kundu,
Aneeshan Sain,
Tao Xiang,
Yi-Zhe Song
Abstract:
The human visual system is remarkable in learning new visual concepts from just a few examples. This is precisely the goal behind few-shot class incremental learning (FSCIL), where the emphasis is additionally placed on ensuring the model does not suffer from "forgetting". In this paper, we push the boundary further for FSCIL by addressing two key questions that bottleneck its ubiquitous applicati…
▽ More
The human visual system is remarkable in learning new visual concepts from just a few examples. This is precisely the goal behind few-shot class incremental learning (FSCIL), where the emphasis is additionally placed on ensuring the model does not suffer from "forgetting". In this paper, we push the boundary further for FSCIL by addressing two key questions that bottleneck its ubiquitous application (i) can the model learn from diverse modalities other than just photo (as humans do), and (ii) what if photos are not readily accessible (due to ethical and privacy constraints). Our key innovation lies in advocating the use of sketches as a new modality for class support. The product is a "Doodle It Yourself" (DIY) FSCIL framework where the users can freely sketch a few examples of a novel class for the model to learn to recognize photos of that class. For that, we present a framework that infuses (i) gradient consensus for domain invariant learning, (ii) knowledge distillation for preserving old class information, and (iii) graph attention networks for message passing between old and novel classes. We experimentally show that sketches are better class support than text in the context of FSCIL, echoing findings elsewhere in the sketching literature.
△ Less
Submitted 28 March, 2022;
originally announced March 2022.
-
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval
Authors:
Ayan Kumar Bhunia,
Subhadeep Koley,
Abdullah Faiz Ur Rahman Khilji,
Aneeshan Sain,
Pinaki Nath Chowdhury,
Tao Xiang,
Yi-Zhe Song
Abstract:
Sketching enables many exciting applications, notably, image retrieval. The fear-to-sketch problem (i.e., "I can't sketch") has however proven to be fatal for its widespread adoption. This paper tackles this "fear" head on, and for the first time, proposes an auxiliary module for existing retrieval models that predominantly lets the users sketch without having to worry. We first conducted a pilot…
▽ More
Sketching enables many exciting applications, notably, image retrieval. The fear-to-sketch problem (i.e., "I can't sketch") has however proven to be fatal for its widespread adoption. This paper tackles this "fear" head on, and for the first time, proposes an auxiliary module for existing retrieval models that predominantly lets the users sketch without having to worry. We first conducted a pilot study that revealed the secret lies in the existence of noisy strokes, but not so much of the "I can't sketch". We consequently design a stroke subset selector that {detects noisy strokes, leaving only those} which make a positive contribution towards successful retrieval. Our Reinforcement Learning based formulation quantifies the importance of each stroke present in a given subset, based on the extent to which that stroke contributes to retrieval. When combined with pre-trained retrieval models as a pre-processing module, we achieve a significant gain of 8%-10% over standard baselines and in turn report new state-of-the-art performance. Last but not least, we demonstrate the selector once trained, can also be used in a plug-and-play manner to empower various sketch applications in ways that were not previously possible.
△ Less
Submitted 28 March, 2022;
originally announced March 2022.
-
Partially Does It: Towards Scene-Level FG-SBIR with Partial Input
Authors:
Pinaki Nath Chowdhury,
Ayan Kumar Bhunia,
Viswanatha Reddy Gajjala,
Aneeshan Sain,
Tao Xiang,
Yi-Zhe Song
Abstract:
We scrutinise an important observation plaguing scene-level sketch research -- that a significant portion of scene sketches are "partial". A quick pilot study reveals: (i) a scene sketch does not necessarily contain all objects in the corresponding photo, due to the subjective holistic interpretation of scenes, (ii) there exists significant empty (white) regions as a result of object-level abstrac…
▽ More
We scrutinise an important observation plaguing scene-level sketch research -- that a significant portion of scene sketches are "partial". A quick pilot study reveals: (i) a scene sketch does not necessarily contain all objects in the corresponding photo, due to the subjective holistic interpretation of scenes, (ii) there exists significant empty (white) regions as a result of object-level abstraction, and as a result, (iii) existing scene-level fine-grained sketch-based image retrieval methods collapse as scene sketches become more partial. To solve this "partial" problem, we advocate for a simple set-based approach using optimal transport (OT) to model cross-modal region associativity in a partially-aware fashion. Importantly, we improve upon OT to further account for holistic partialness by comparing intra-modal adjacency matrices. Our proposed method is not only robust to partial scene-sketches but also yields state-of-the-art performance on existing datasets.
△ Less
Submitted 28 March, 2022;
originally announced March 2022.
-
Sketch3T: Test-Time Training for Zero-Shot SBIR
Authors:
Aneeshan Sain,
Ayan Kumar Bhunia,
Vaishnav Potlapalli,
Pinaki Nath Chowdhury,
Tao Xiang,
Yi-Zhe Song
Abstract:
Zero-shot sketch-based image retrieval typically asks for a trained model to be applied as is to unseen categories. In this paper, we question to argue that this setup by definition is not compatible with the inherent abstract and subjective nature of sketches, i.e., the model might transfer well to new categories, but will not understand sketches existing in different test-time distribution as a…
▽ More
Zero-shot sketch-based image retrieval typically asks for a trained model to be applied as is to unseen categories. In this paper, we question to argue that this setup by definition is not compatible with the inherent abstract and subjective nature of sketches, i.e., the model might transfer well to new categories, but will not understand sketches existing in different test-time distribution as a result. We thus extend ZS-SBIR asking it to transfer to both categories and sketch distributions. Our key contribution is a test-time training paradigm that can adapt using just one sketch. Since there is no paired photo, we make use of a sketch raster-vector reconstruction module as a self-supervised auxiliary task. To maintain the fidelity of the trained cross-modal joint embedding during test-time update, we design a novel meta-learning based training paradigm to learn a separation between model updates incurred by this auxiliary task from those off the primary objective of discriminative learning. Extensive experiments show our model to outperform state of-the-arts, thanks to the proposed test-time adaption that not only transfers to new categories but also accommodates to new sketching styles.
△ Less
Submitted 28 March, 2022;
originally announced March 2022.
-
FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context
Authors:
Pinaki Nath Chowdhury,
Aneeshan Sain,
Ayan Kumar Bhunia,
Tao Xiang,
Yulia Gryaditskaya,
Yi-Zhe Song
Abstract:
We advance sketch research to scenes with the first dataset of freehand scene sketches, FS-COCO. With practical applications in mind, we collect sketches that convey scene content well but can be sketched within a few minutes by a person with any sketching skills. Our dataset comprises 10,000 freehand scene vector sketches with per point space-time information by 100 non-expert individuals, offeri…
▽ More
We advance sketch research to scenes with the first dataset of freehand scene sketches, FS-COCO. With practical applications in mind, we collect sketches that convey scene content well but can be sketched within a few minutes by a person with any sketching skills. Our dataset comprises 10,000 freehand scene vector sketches with per point space-time information by 100 non-expert individuals, offering both object- and scene-level abstraction. Each sketch is augmented with its text description. Using our dataset, we study for the first time the problem of fine-grained image retrieval from freehand scene sketches and sketch captions. We draw insights on: (i) Scene salience encoded in sketches using the strokes temporal order; (ii) Performance comparison of image retrieval from a scene sketch and an image caption; (iii) Complementarity of information in sketches and image captions, as well as the potential benefit of combining the two modalities. In addition, we extend a popular vector sketch LSTM-based encoder to handle sketches with larger complexity than was supported by previous work. Namely, we propose a hierarchical sketch decoder, which we leverage at a sketch-specific "pre-text" task. Our dataset enables for the first time research on freehand scene sketch understanding and its practical applications.
△ Less
Submitted 20 July, 2022; v1 submitted 3 March, 2022;
originally announced March 2022.
-
DoodleFormer: Creative Sketch Drawing with Transformers
Authors:
Ankan Kumar Bhunia,
Salman Khan,
Hisham Cholakkal,
Rao Muhammad Anwer,
Fahad Shahbaz Khan,
Jorma Laaksonen,
Michael Felsberg
Abstract:
Creative sketching or doodling is an expressive activity, where imaginative and previously unseen depictions of everyday visual objects are drawn. Creative sketch image generation is a challenging vision problem, where the task is to generate diverse, yet realistic creative sketches possessing the unseen composition of the visual-world objects. Here, we propose a novel coarse-to-fine two-stage fra…
▽ More
Creative sketching or doodling is an expressive activity, where imaginative and previously unseen depictions of everyday visual objects are drawn. Creative sketch image generation is a challenging vision problem, where the task is to generate diverse, yet realistic creative sketches possessing the unseen composition of the visual-world objects. Here, we propose a novel coarse-to-fine two-stage framework, DoodleFormer, that decomposes the creative sketch generation problem into the creation of coarse sketch composition followed by the incorporation of fine-details in the sketch. We introduce graph-aware transformer encoders that effectively capture global dynamic as well as local static structural relations among different body parts. To ensure diversity of the generated creative sketches, we introduce a probabilistic coarse sketch decoder that explicitly models the variations of each sketch body part to be drawn. Experiments are performed on two creative sketch datasets: Creative Birds and Creative Creatures. Our qualitative, quantitative and human-based evaluations show that DoodleFormer outperforms the state-of-the-art on both datasets, yielding realistic and diverse creative sketches. On Creative Creatures, DoodleFormer achieves an absolute gain of 25 in terms of Fr`echet inception distance (FID) over the state-of-the-art. We also demonstrate the effectiveness of DoodleFormer for related applications of text to creative sketch generation and sketch completion.
△ Less
Submitted 15 September, 2022; v1 submitted 6 December, 2021;
originally announced December 2021.
-
Nonlocality without entanglement: Party asymmetric case
Authors:
Atanu Bhunia,
Indrani Chattopadhyay,
Debasis Sarkar
Abstract:
A set of orthogonal product states of a composite Hilbert space is genuinely nonlocal if the states are locally indistinguishable across any bipartition. In this work, we construct a minimal set of party asymmetry genuine nonlocal set in arbitrary large dimensional composite quantum systems $C^d\otimes C^d\otimes C^d$. We provide a local discriminating protocol by using a three qubit GHZ state as…
▽ More
A set of orthogonal product states of a composite Hilbert space is genuinely nonlocal if the states are locally indistinguishable across any bipartition. In this work, we construct a minimal set of party asymmetry genuine nonlocal set in arbitrary large dimensional composite quantum systems $C^d\otimes C^d\otimes C^d$. We provide a local discriminating protocol by using a three qubit GHZ state as a resource. On the contrary, we observe that single-copy of two qubit Bell states provide no advantage for this discrimination task. Recently, Halder et al. [Phys. Rev. Lett. 122, 040403 (2019)], proposed the concept of strong nonlocality without entanglement and ask an open question whether there exist an incomplete strong nonlocal set or not. In [Phys. Rev. A 102, 042228 (2020)], an answer is provided by the authors. Here, we construct an incomplete party asymmetry strong nonlocal set which is more stronger than the set constructed in [Phys. Rev. A 102, 042228 (2020)] with respect to the consumption of entanglement as a resource for their respective discrimination tasks.
△ Less
Submitted 29 November, 2021;
originally announced November 2021.
-
0D-2D Heterostructure for making very Large Quantum Registers using itinerant Bose-Einstein Condensate of Excitons
Authors:
Amit Bhunia,
Mohit Kumar Singh,
Maryam Al Huwayz,
Mohamed Henini,
Shouvik Datta
Abstract:
Presence of coherent resonant tunneling in quantum dot (zero-dimensional) - quantum well (two-dimensional) heterostructure is necessary to explain the collective oscillations of average electrical polarization of excitonic dipoles over a macroscopically large area. This was measured using photo excited capacitance as a function of applied voltage bias. Resonant tunneling in this heterostructure de…
▽ More
Presence of coherent resonant tunneling in quantum dot (zero-dimensional) - quantum well (two-dimensional) heterostructure is necessary to explain the collective oscillations of average electrical polarization of excitonic dipoles over a macroscopically large area. This was measured using photo excited capacitance as a function of applied voltage bias. Resonant tunneling in this heterostructure definitely requires momentum space narrowing of charge carriers inside the quantum well and that of associated indirect excitons, which indicates bias dependent itinerant Bose-Einstein condensation of excitons. Observation of periodic variations in negative quantum capacitance points to in-plane coulomb correlations mediated by long range spatial ordering of indirect, dipolar excitons. Enhanced contrast of quantum interference beats of excitonic polarization waves even under white light and observed Rabi oscillations over a macroscopically large area also support the presence of density driven excitonic condensation having long range order. Periodic presence (absence) of splitting of excitonic peaks in photocapacitance spectra even demonstrate collective coupling (decoupling) between energy levels of the quantum well and quantum dots with applied biases, which can potentially be used for quantum gate operations. All these observations point to experimental control of macroscopically large, quantum state of a two-component Bose-Einstein condensate of excitons in this quantum dot - quantum well heterostructure. Therefore, in principle, millions of two-level excitonic qubits can be intertwined to fabricate large quantum registers using such hybrid heterostructure by controlling the local electric fields and also by varying photoexcitation intensities of overlap** light spots.
△ Less
Submitted 19 June, 2023; v1 submitted 28 July, 2021;
originally announced July 2021.
-
Joint Visual Semantic Reasoning: Multi-Stage Decoder for Text Recognition
Authors:
Ayan Kumar Bhunia,
Aneeshan Sain,
Amandeep Kumar,
Shuvozit Ghose,
Pinaki Nath Chowdhury,
Yi-Zhe Song
Abstract:
Although text recognition has significantly evolved over the years, state-of-the-art (SOTA) models still struggle in the wild scenarios due to complex backgrounds, varying fonts, uncontrolled illuminations, distortions and other artefacts. This is because such models solely depend on visual information for text recognition, thus lacking semantic reasoning capabilities. In this paper, we argue that…
▽ More
Although text recognition has significantly evolved over the years, state-of-the-art (SOTA) models still struggle in the wild scenarios due to complex backgrounds, varying fonts, uncontrolled illuminations, distortions and other artefacts. This is because such models solely depend on visual information for text recognition, thus lacking semantic reasoning capabilities. In this paper, we argue that semantic information offers a complementary role in addition to visual only. More specifically, we additionally utilize semantic information by proposing a multi-stage multi-scale attentional decoder that performs joint visual-semantic reasoning. Our novelty lies in the intuition that for text recognition, the prediction should be refined in a stage-wise manner. Therefore our key contribution is in designing a stage-wise unrolling attentional decoder where non-differentiability, invoked by discretely predicted character labels, needs to be bypassed for end-to-end training. While the first stage predicts using visual features, subsequent stages refine on top of it using joint visual-semantic information. Additionally, we introduce multi-scale 2D attention along with dense and residual connections between different stages to deal with varying scales of character sizes, for better performance and faster convergence during training. Experimental results show our approach to outperform existing SOTA methods by a considerable margin.
△ Less
Submitted 26 July, 2021; v1 submitted 26 July, 2021;
originally announced July 2021.
-
Text is Text, No Matter What: Unifying Text Recognition using Knowledge Distillation
Authors:
Ayan Kumar Bhunia,
Aneeshan Sain,
Pinaki Nath Chowdhury,
Yi-Zhe Song
Abstract:
Text recognition remains a fundamental and extensively researched topic in computer vision, largely owing to its wide array of commercial applications. The challenging nature of the very problem however dictated a fragmentation of research efforts: Scene Text Recognition (STR) that deals with text in everyday scenes, and Handwriting Text Recognition (HTR) that tackles hand-written text. In this pa…
▽ More
Text recognition remains a fundamental and extensively researched topic in computer vision, largely owing to its wide array of commercial applications. The challenging nature of the very problem however dictated a fragmentation of research efforts: Scene Text Recognition (STR) that deals with text in everyday scenes, and Handwriting Text Recognition (HTR) that tackles hand-written text. In this paper, for the first time, we argue for their unification -- we aim for a single model that can compete favourably with two separate state-of-the-art STR and HTR models. We first show that cross-utilisation of STR and HTR models trigger significant performance drops due to differences in their inherent challenges. We then tackle their union by introducing a knowledge distillation (KD) based framework. This is however non-trivial, largely due to the variable-length and sequential nature of text sequences, which renders off-the-shelf KD techniques that mostly works with global fixed-length data inadequate. For that, we propose three distillation losses all of which are specifically designed to cope with the aforementioned unique characteristics of text recognition. Empirical evidence suggests that our proposed unified model performs on par with individual models, even surpassing them in certain cases. Ablative studies demonstrate that naive baselines such as a two-stage framework, and domain adaption/generalisation alternatives do not work as well, further verifying the appropriateness of our design.
△ Less
Submitted 27 July, 2021; v1 submitted 26 July, 2021;
originally announced July 2021.
-
Towards the Unseen: Iterative Text Recognition by Distilling from Errors
Authors:
Ayan Kumar Bhunia,
Pinaki Nath Chowdhury,
Aneeshan Sain,
Yi-Zhe Song
Abstract:
Visual text recognition is undoubtedly one of the most extensively researched topics in computer vision. Great progress have been made to date, with the latest models starting to focus on the more practical "in-the-wild" setting. However, a salient problem still hinders practical deployment -- prior arts mostly struggle with recognising unseen (or rarely seen) character sequences. In this paper, w…
▽ More
Visual text recognition is undoubtedly one of the most extensively researched topics in computer vision. Great progress have been made to date, with the latest models starting to focus on the more practical "in-the-wild" setting. However, a salient problem still hinders practical deployment -- prior arts mostly struggle with recognising unseen (or rarely seen) character sequences. In this paper, we put forward a novel framework to specifically tackle this "unseen" problem. Our framework is iterative in nature, in that it utilises predicted knowledge of character sequences from a previous iteration, to augment the main network in improving the next prediction. Key to our success is a unique cross-modal variational autoencoder to act as a feedback module, which is trained with the presence of textual error distribution data. This module importantly translate a discrete predicted character space, to a continuous affine transformation parameter space used to condition the visual feature map at next iteration. Experiments on common datasets have shown competitive performance over state-of-the-arts under the conventional setting. Most importantly, under the new disjoint setup where train-test labels are mutually exclusive, ours offers the best performance thus showcasing the capability of generalising onto unseen words.
△ Less
Submitted 26 July, 2021;
originally announced July 2021.
-
Handwriting Transformers
Authors:
Ankan Kumar Bhunia,
Salman Khan,
Hisham Cholakkal,
Rao Muhammad Anwer,
Fahad Shahbaz Khan,
Mubarak Shah
Abstract:
We propose a novel transformer-based styled handwritten text image generation approach, HWT, that strives to learn both style-content entanglement as well as global and local writing style patterns. The proposed HWT captures the long and short range relationships within the style examples through a self-attention mechanism, thereby encoding both global and local style patterns. Further, the propos…
▽ More
We propose a novel transformer-based styled handwritten text image generation approach, HWT, that strives to learn both style-content entanglement as well as global and local writing style patterns. The proposed HWT captures the long and short range relationships within the style examples through a self-attention mechanism, thereby encoding both global and local style patterns. Further, the proposed transformer-based HWT comprises an encoder-decoder attention that enables style-content entanglement by gathering the style representation of each query character. To the best of our knowledge, we are the first to introduce a transformer-based generative network for styled handwritten text generation. Our proposed HWT generates realistic styled handwritten text images and significantly outperforms the state-of-the-art demonstrated through extensive qualitative, quantitative and human-based evaluations. The proposed HWT can handle arbitrary length of text and any desired writing style in a few-shot setting. Further, our HWT generalizes well to the challenging scenario where both words and writing style are unseen during training, generating realistic styled handwritten text images.
△ Less
Submitted 8 April, 2021;
originally announced April 2021.
-
MetaHTR: Towards Writer-Adaptive Handwritten Text Recognition
Authors:
Ayan Kumar Bhunia,
Shuvozit Ghose,
Amandeep Kumar,
Pinaki Nath Chowdhury,
Aneeshan Sain,
Yi-Zhe Song
Abstract:
Handwritten Text Recognition (HTR) remains a challenging problem to date, largely due to the varying writing styles that exist amongst us. Prior works however generally operate with the assumption that there is a limited number of styles, most of which have already been captured by existing datasets. In this paper, we take a completely different perspective -- we work on the assumption that there…
▽ More
Handwritten Text Recognition (HTR) remains a challenging problem to date, largely due to the varying writing styles that exist amongst us. Prior works however generally operate with the assumption that there is a limited number of styles, most of which have already been captured by existing datasets. In this paper, we take a completely different perspective -- we work on the assumption that there is always a new style that is drastically different, and that we will only have very limited data during testing to perform adaptation. This results in a commercially viable solution -- the model has the best shot at adaptation being exposed to the new style, and the few samples nature makes it practical to implement. We achieve this via a novel meta-learning framework which exploits additional new-writer data through a support set, and outputs a writer-adapted model via single gradient step update, all during inference. We discover and leverage on the important insight that there exists few key characters per writer that exhibit relatively larger style discrepancies. For that, we additionally propose to meta-learn instance specific weights for a character-wise cross-entropy loss, which is specifically designed to work with the sequential nature of text data. Our writer-adaptive MetaHTR framework can be easily implemented on the top of most state-of-the-art HTR models. Experiments show an average performance gain of 5-7% can be obtained by observing very few new style data. We further demonstrate via a set of ablative studies the advantage of our meta design when compared with alternative adaption mechanisms.
△ Less
Submitted 5 April, 2021;
originally announced April 2021.
-
StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval
Authors:
Aneeshan Sain,
Ayan Kumar Bhunia,
Yongxin Yang,
Tao Xiang,
Yi-Zhe Song
Abstract:
Sketch-based image retrieval (SBIR) is a cross-modal matching problem which is typically solved by learning a joint embedding space where the semantic content shared between photo and sketch modalities are preserved. However, a fundamental challenge in SBIR has been largely ignored so far, that is, sketches are drawn by humans and considerable style variations exist amongst different users. An eff…
▽ More
Sketch-based image retrieval (SBIR) is a cross-modal matching problem which is typically solved by learning a joint embedding space where the semantic content shared between photo and sketch modalities are preserved. However, a fundamental challenge in SBIR has been largely ignored so far, that is, sketches are drawn by humans and considerable style variations exist amongst different users. An effective SBIR model needs to explicitly account for this style diversity, crucially, to generalise to unseen user styles. To this end, a novel style-agnostic SBIR model is proposed. Different from existing models, a cross-modal variational autoencoder (VAE) is employed to explicitly disentangle each sketch into a semantic content part shared with the corresponding photo, and a style part unique to the sketcher. Importantly, to make our model dynamically adaptable to any unseen user styles, we propose to meta-train our cross-modal VAE by adding two style-adaptive components: a set of feature transformation layers to its encoder and a regulariser to the disentangled semantic content latent code. With this meta-learning framework, our model can not only disentangle the cross-modal shared semantic content for SBIR, but can adapt the disentanglement to any unseen user style as well, making the SBIR model truly style-agnostic. Extensive experiments show that our style-agnostic model yields state-of-the-art performance for both category-level and instance-level SBIR.
△ Less
Submitted 31 March, 2021; v1 submitted 29 March, 2021;
originally announced March 2021.
-
More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval
Authors:
Ayan Kumar Bhunia,
Pinaki Nath Chowdhury,
Aneeshan Sain,
Yongxin Yang,
Tao Xiang,
Yi-Zhe Song
Abstract:
A fundamental challenge faced by existing Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) models is the data scarcity -- model performances are largely bottlenecked by the lack of sketch-photo pairs. Whilst the number of photos can be easily scaled, each corresponding sketch still needs to be individually produced. In this paper, we aim to mitigate such an upper-bound on sketch data, and study…
▽ More
A fundamental challenge faced by existing Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) models is the data scarcity -- model performances are largely bottlenecked by the lack of sketch-photo pairs. Whilst the number of photos can be easily scaled, each corresponding sketch still needs to be individually produced. In this paper, we aim to mitigate such an upper-bound on sketch data, and study whether unlabelled photos alone (of which they are many) can be cultivated for performances gain. In particular, we introduce a novel semi-supervised framework for cross-modal retrieval that can additionally leverage large-scale unlabelled photos to account for data scarcity. At the centre of our semi-supervision design is a sequential photo-to-sketch generation model that aims to generate paired sketches for unlabelled photos. Importantly, we further introduce a discriminator guided mechanism to guide against unfaithful generation, together with a distillation loss based regularizer to provide tolerance against noisy training samples. Last but not least, we treat generation and retrieval as two conjugate problems, where a joint learning procedure is devised for each module to mutually benefit from each other. Extensive experiments show that our semi-supervised model yields significant performance boost over the state-of-the-art supervised alternatives, as well as existing methods that can exploit unlabelled photos for FG-SBIR.
△ Less
Submitted 25 March, 2021;
originally announced March 2021.
-
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting
Authors:
Ayan Kumar Bhunia,
Pinaki Nath Chowdhury,
Yongxin Yang,
Timothy M. Hospedales,
Tao Xiang,
Yi-Zhe Song
Abstract:
Self-supervised learning has gained prominence due to its efficacy at learning powerful representations from unlabelled data that achieve excellent performance on many challenging downstream tasks. However supervision-free pre-text tasks are challenging to design and usually modality specific. Although there is a rich literature of self-supervised methods for either spatial (such as images) or tem…
▽ More
Self-supervised learning has gained prominence due to its efficacy at learning powerful representations from unlabelled data that achieve excellent performance on many challenging downstream tasks. However supervision-free pre-text tasks are challenging to design and usually modality specific. Although there is a rich literature of self-supervised methods for either spatial (such as images) or temporal data (sound or text) modalities, a common pre-text task that benefits both modalities is largely missing. In this paper, we are interested in defining a self-supervised pre-text task for sketches and handwriting data. This data is uniquely characterised by its existence in dual modalities of rasterized images and vector coordinate sequences. We address and exploit this dual representation by proposing two novel cross-modal translation pre-text tasks for self-supervised feature learning: Vectorization and Rasterization. Vectorization learns to map image space to vector coordinates and rasterization maps vector coordinates to image space. We show that the our learned encoder modules benefit both raster-based and vector-based downstream approaches to analysing hand-drawn data. Empirical evidence shows that our novel pre-text tasks surpass existing single and multi-modal self-supervision methods.
△ Less
Submitted 25 March, 2021;
originally announced March 2021.
-
Nonlocality of tripartite orthogonal product states
Authors:
Atanu Bhunia,
Indrani Chattopadhyay,
Debasis Sarkar
Abstract:
Local distinguishability of orthogonal product states is an area of active research in quantum information theory. However, most of the relevant results about local distinguishability found in bipartite quantum systems and very few are known in multipartite systems. In this work, we construct a locally indistinguishable subset in…
▽ More
Local distinguishability of orthogonal product states is an area of active research in quantum information theory. However, most of the relevant results about local distinguishability found in bipartite quantum systems and very few are known in multipartite systems. In this work, we construct a locally indistinguishable subset in ${\mathbb{C}}^{2d}\bigotimes{\mathbb{C}}^{2d}\bigotimes{\mathbb{C}}^{2d}$, $d\geq2$ that contains $18(d-1)$ orthogonal product states. Further, we generalize our method to arbitrary tripartite quantum systems ${\mathbb{C}}^{k}\bigotimes{\mathbb{C}}^{l}\bigotimes{\mathbb{C}}^{m}$. This result enables us to understand further the role of nonlocality without entanglement in multipartite quantum systems. Finally, we prove that a three-qubit GHZ state is sufficient as a resource to distinguish each of the above classes of states.
△ Less
Submitted 7 November, 2020;
originally announced November 2020.
-
Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval
Authors:
Aneeshan Sain,
Ayan Kumar Bhunia,
Yongxin Yang,
Tao Xiang,
Yi-Zhe Song
Abstract:
Sketch as an image search query is an ideal alternative to text in capturing the fine-grained visual details. Prior successes on fine-grained sketch-based image retrieval (FG-SBIR) have demonstrated the importance of tackling the unique traits of sketches as opposed to photos, e.g., temporal vs. static, strokes vs. pixels, and abstract vs. pixel-perfect. In this paper, we study a further trait of…
▽ More
Sketch as an image search query is an ideal alternative to text in capturing the fine-grained visual details. Prior successes on fine-grained sketch-based image retrieval (FG-SBIR) have demonstrated the importance of tackling the unique traits of sketches as opposed to photos, e.g., temporal vs. static, strokes vs. pixels, and abstract vs. pixel-perfect. In this paper, we study a further trait of sketches that has been overlooked to date, that is, they are hierarchical in terms of the levels of detail -- a person typically sketches up to various extents of detail to depict an object. This hierarchical structure is often visually distinct. In this paper, we design a novel network that is capable of cultivating sketch-specific hierarchies and exploiting them to match sketch with photo at corresponding hierarchical levels. In particular, features from a sketch and a photo are enriched using cross-modal co-attention, coupled with hierarchical node fusion at every level to form a better embedding space to conduct retrieval. Experiments on common benchmarks show our method to outperform state-of-the-arts by a significant margin.
△ Less
Submitted 11 August, 2020; v1 submitted 29 July, 2020;
originally announced July 2020.
-
Fine-Grained Visual Classification via Progressive Multi-Granularity Training of Jigsaw Patches
Authors:
Ruoyi Du,
Dongliang Chang,
Ayan Kumar Bhunia,
Jiyang Xie,
Zhanyu Ma,
Yi-Zhe Song,
Jun Guo
Abstract:
Fine-grained visual classification (FGVC) is much more challenging than traditional classification tasks due to the inherently subtle intra-class object variations. Recent works mainly tackle this problem by focusing on how to locate the most discriminative parts, more complementary parts, and parts of various granularities. However, less effort has been placed to which granularities are the most…
▽ More
Fine-grained visual classification (FGVC) is much more challenging than traditional classification tasks due to the inherently subtle intra-class object variations. Recent works mainly tackle this problem by focusing on how to locate the most discriminative parts, more complementary parts, and parts of various granularities. However, less effort has been placed to which granularities are the most discriminative and how to fuse information cross multi-granularity. In this work, we propose a novel framework for fine-grained visual classification to tackle these problems. In particular, we propose: (i) a progressive training strategy that effectively fuses features from different granularities, and (ii) a random jigsaw patch generator that encourages the network to learn features at specific granularities. We obtain state-of-the-art performances on several standard FGVC benchmark datasets, where the proposed method consistently outperforms existing methods or delivers competitive results. The code will be available at https://github.com/PRIS-CV/PMG-Progressive-Multi-Granularity-Training.
△ Less
Submitted 19 July, 2020; v1 submitted 8 March, 2020;
originally announced March 2020.
-
Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrieval
Authors:
Ayan Kumar Bhunia,
Yongxin Yang,
Timothy M. Hospedales,
Tao Xiang,
Yi-Zhe Song
Abstract:
Fine-grained sketch-based image retrieval (FG-SBIR) addresses the problem of retrieving a particular photo instance given a user's query sketch. Its widespread applicability is however hindered by the fact that drawing a sketch takes time, and most people struggle to draw a complete and faithful sketch. In this paper, we reformulate the conventional FG-SBIR framework to tackle these challenges, wi…
▽ More
Fine-grained sketch-based image retrieval (FG-SBIR) addresses the problem of retrieving a particular photo instance given a user's query sketch. Its widespread applicability is however hindered by the fact that drawing a sketch takes time, and most people struggle to draw a complete and faithful sketch. In this paper, we reformulate the conventional FG-SBIR framework to tackle these challenges, with the ultimate goal of retrieving the target photo with the least number of strokes possible. We further propose an on-the-fly design that starts retrieving as soon as the user starts drawing. To accomplish this, we devise a reinforcement learning-based cross-modal retrieval framework that directly optimizes rank of the ground-truth photo over a complete sketch drawing episode. Additionally, we introduce a novel reward scheme that circumvents the problems related to irrelevant sketch strokes, and thus provides us with a more consistent rank list during the retrieval. We achieve superior early-retrieval efficiency over state-of-the-art methods and alternative baselines on two publicly available fine-grained sketch retrieval datasets.
△ Less
Submitted 11 May, 2020; v1 submitted 24 February, 2020;
originally announced February 2020.
-
The Devil is in the Channels: Mutual-Channel Loss for Fine-Grained Image Classification
Authors:
Dongliang Chang,
Yifeng Ding,
Jiyang Xie,
Ayan Kumar Bhunia,
Xiaoxu Li,
Zhanyu Ma,
Ming Wu,
Jun Guo,
Yi-Zhe Song
Abstract:
Key for solving fine-grained image categorization is finding discriminate and local regions that correspond to subtle visual traits. Great strides have been made, with complex networks designed specifically to learn part-level discriminate feature representations. In this paper, we show it is possible to cultivate subtle details without the need for overly complicated network designs or training m…
▽ More
Key for solving fine-grained image categorization is finding discriminate and local regions that correspond to subtle visual traits. Great strides have been made, with complex networks designed specifically to learn part-level discriminate feature representations. In this paper, we show it is possible to cultivate subtle details without the need for overly complicated network designs or training mechanisms -- a single loss is all it takes. The main trick lies with how we delve into individual feature channels early on, as opposed to the convention of starting from a consolidated feature map. The proposed loss function, termed as mutual-channel loss (MC-Loss), consists of two channel-specific components: a discriminality component and a diversity component. The discriminality component forces all feature channels belonging to the same class to be discriminative, through a novel channel-wise attention mechanism. The diversity component additionally constraints channels so that they become mutually exclusive on spatial-wise. The end result is therefore a set of feature channels that each reflects different locally discriminative regions for a specific class. The MC-Loss can be trained end-to-end, without the need for any bounding-box/part annotations, and yields highly discriminative regions during inference. Experimental results show our MC-Loss when implemented on top of common base networks can achieve state-of-the-art performance on all four fine-grained categorization datasets (CUB-Birds, FGVC-Aircraft, Flowers-102, and Stanford-Cars). Ablative studies further demonstrate the superiority of MC-Loss when compared with other recently proposed general-purpose losses for visual classification, on two different base networks. Code available at https://github.com/dongliangchang/Mutual-Channel-Loss
△ Less
Submitted 10 August, 2021; v1 submitted 11 February, 2020;
originally announced February 2020.