-
Fine-grained Background Representation for Weakly Supervised Semantic Segmentation
Authors:
Xu Yin,
Woobin Im,
Dongbo Min,
Yuchi Huo,
Fei Pan,
Sung-Eui Yoon
Abstract:
Generating reliable pseudo masks from image-level labels is challenging in the weakly supervised semantic segmentation (WSSS) task due to the lack of spatial information. Prevalent class activation map (CAM)-based solutions are challenged to discriminate the foreground (FG) objects from the suspicious background (BG) pixels (a.k.a. co-occurring) and learn the integral object regions. This paper pr…
▽ More
Generating reliable pseudo masks from image-level labels is challenging in the weakly supervised semantic segmentation (WSSS) task due to the lack of spatial information. Prevalent class activation map (CAM)-based solutions are challenged to discriminate the foreground (FG) objects from the suspicious background (BG) pixels (a.k.a. co-occurring) and learn the integral object regions. This paper proposes a simple fine-grained background representation (FBR) method to discover and represent diverse BG semantics and address the co-occurring problems. We abandon using the class prototype or pixel-level features for BG representation. Instead, we develop a novel primitive, negative region of interest (NROI), to capture the fine-grained BG semantic information and conduct the pixel-to-NROI contrast to distinguish the confusing BG pixels. We also present an active sampling strategy to mine the FG negatives on-the-fly, enabling efficient pixel-to-pixel intra-foreground contrastive learning to activate the entire object region. Thanks to the simplicity of design and convenience in use, our proposed method can be seamlessly plugged into various models, yielding new state-of-the-art results under various WSSS settings across benchmarks. Leveraging solely image-level (I) labels as supervision, our method achieves 73.2 mIoU and 45.6 mIoU segmentation results on Pascal Voc and MS COCO test sets, respectively. Furthermore, by incorporating saliency maps as an additional supervision signal (I+S), we attain 74.9 mIoU on Pascal Voc test set. Concurrently, our FBR approach demonstrates meaningful performance gains in weakly-supervised instance segmentation (WSIS) tasks, showcasing its robustness and strong generalization capabilities across diverse domains.
△ Less
Submitted 22 June, 2024;
originally announced June 2024.
-
Extending Segment Anything Model into Auditory and Temporal Dimensions for Audio-Visual Segmentation
Authors:
Juhyeong Seon,
Woobin Im,
Sebin Lee,
Jumin Lee,
Sung-Eui Yoon
Abstract:
Audio-visual segmentation (AVS) aims to segment sound sources in the video sequence, requiring a pixel-level understanding of audio-visual correspondence. As the Segment Anything Model (SAM) has strongly impacted extensive fields of dense prediction problems, prior works have investigated the introduction of SAM into AVS with audio as a new modality of the prompt. Nevertheless, constrained by SAM'…
▽ More
Audio-visual segmentation (AVS) aims to segment sound sources in the video sequence, requiring a pixel-level understanding of audio-visual correspondence. As the Segment Anything Model (SAM) has strongly impacted extensive fields of dense prediction problems, prior works have investigated the introduction of SAM into AVS with audio as a new modality of the prompt. Nevertheless, constrained by SAM's single-frame segmentation scheme, the temporal context across multiple frames of audio-visual data remains insufficiently utilized. To this end, we study the extension of SAM's capabilities to the sequence of audio-visual scenes by analyzing contextual cross-modal relationships across the frames. To achieve this, we propose a Spatio-Temporal, Bidirectional Audio-Visual Attention (ST-BAVA) module integrated into the middle of SAM's image encoder and mask decoder. It adaptively updates the audio-visual features to convey the spatio-temporal correspondence between the video frames and audio streams. Extensive experiments demonstrate that our proposed model outperforms the state-of-the-art methods on AVS benchmarks, especially with an 8.3% mIoU gain on a challenging multi-sources subset.
△ Less
Submitted 10 June, 2024;
originally announced June 2024.
-
SemCity: Semantic Scene Generation with Triplane Diffusion
Authors:
Jumin Lee,
Sebin Lee,
Changho Jo,
Woobin Im,
Juhyeong Seon,
Sung-Eui Yoon
Abstract:
We present "SemCity," a 3D diffusion model for semantic scene generation in real-world outdoor environments. Most 3D diffusion models focus on generating a single object, synthetic indoor scenes, or synthetic outdoor scenes, while the generation of real-world outdoor scenes is rarely addressed. In this paper, we concentrate on generating a real-outdoor scene through learning a diffusion model on a…
▽ More
We present "SemCity," a 3D diffusion model for semantic scene generation in real-world outdoor environments. Most 3D diffusion models focus on generating a single object, synthetic indoor scenes, or synthetic outdoor scenes, while the generation of real-world outdoor scenes is rarely addressed. In this paper, we concentrate on generating a real-outdoor scene through learning a diffusion model on a real-world outdoor dataset. In contrast to synthetic data, real-outdoor datasets often contain more empty spaces due to sensor limitations, causing challenges in learning real-outdoor distributions. To address this issue, we exploit a triplane representation as a proxy form of scene distributions to be learned by our diffusion model. Furthermore, we propose a triplane manipulation that integrates seamlessly with our triplane diffusion model. The manipulation improves our diffusion model's applicability in a variety of downstream tasks related to outdoor scene generation such as scene inpainting, scene outpainting, and semantic scene completion refinements. In experimental results, we demonstrate that our triplane diffusion model shows meaningful generation results compared with existing work in a real-outdoor dataset, SemanticKITTI. We also show our triplane manipulation facilitates seamlessly adding, removing, or modifying objects within a scene. Further, it also enables the expansion of scenes toward a city-level scale. Finally, we evaluate our method on semantic scene completion refinements where our diffusion model enhances predictions of semantic scene completion networks by learning scene distribution. Our code is available at https://github.com/zoomin-lee/SemCity.
△ Less
Submitted 17 March, 2024; v1 submitted 12 March, 2024;
originally announced March 2024.
-
Harnessing Meta-Learning for Improving Full-Frame Video Stabilization
Authors:
Muhammad Kashif Ali,
Eun Woo Im,
Dong** Kim,
Tae Hyun Kim
Abstract:
Video stabilization is a longstanding computer vision problem, particularly pixel-level synthesis solutions for video stabilization which synthesize full frames add to the complexity of this task. These techniques aim to stabilize videos by synthesizing full frames while enhancing the stability of the considered video. This intensifies the complexity of the task due to the distinct mix of unique m…
▽ More
Video stabilization is a longstanding computer vision problem, particularly pixel-level synthesis solutions for video stabilization which synthesize full frames add to the complexity of this task. These techniques aim to stabilize videos by synthesizing full frames while enhancing the stability of the considered video. This intensifies the complexity of the task due to the distinct mix of unique motion profiles and visual content present in each video sequence, making robust generalization with fixed parameters difficult. In our study, we introduce a novel approach to enhance the performance of pixel-level synthesis solutions for video stabilization by adapting these models to individual input video sequences. The proposed adaptation exploits low-level visual cues accessible during test-time to improve both the stability and quality of resulting videos. We highlight the efficacy of our methodology of "test-time adaptation" through simple fine-tuning of one of these models, followed by significant stability gain via the integration of meta-learning techniques. Notably, significant improvement is achieved with only a single adaptation step. The versatility of the proposed algorithm is demonstrated by consistently improving the performance of various pixel-level synthesis models for video stabilization in real-world scenarios.
△ Less
Submitted 8 April, 2024; v1 submitted 6 March, 2024;
originally announced March 2024.
-
Mesh Represented Recycle Learning for 3D Hand Pose and Mesh Estimation
Authors:
Bosang Kim,
Jonghyun Kim,
Hyotae Lee,
Lanying **,
Jeongwon Ha,
Dowoo Kwon,
Jungpyo Kim,
Wonhyeok Im,
KyungMin **,
Jungho Lee
Abstract:
In general, hand pose estimation aims to improve the robustness of model performance in the real-world scenes. However, it is difficult to enhance the robustness since existing datasets are obtained in restricted environments to annotate 3D information. Although neural networks quantitatively achieve a high estimation accuracy, unsatisfied results can be observed in visual quality. This discrepanc…
▽ More
In general, hand pose estimation aims to improve the robustness of model performance in the real-world scenes. However, it is difficult to enhance the robustness since existing datasets are obtained in restricted environments to annotate 3D information. Although neural networks quantitatively achieve a high estimation accuracy, unsatisfied results can be observed in visual quality. This discrepancy between quantitative results and their visual qualities remains an open issue in the hand pose representation. To this end, we propose a mesh represented recycle learning strategy for 3D hand pose and mesh estimation which reinforces synthesized hand mesh representation in a training phase. To be specific, a hand pose and mesh estimation model first predicts parametric 3D hand annotations (i.e., 3D keypoint positions and vertices for hand mesh) with real-world hand images in the training phase. Second, synthetic hand images are generated with self-estimated hand mesh representations. After that, the synthetic hand images are fed into the same model again. Thus, the proposed learning strategy simultaneously improves quantitative results and visual qualities by reinforcing synthetic mesh representation. To encourage consistency between original model output and its recycled one, we propose self-correlation loss which maximizes the accuracy and reliability of our learning strategy. Consequently, the model effectively conducts self-refinement on hand pose estimation by learning mesh representation from its own output. To demonstrate the effectiveness of our learning strategy, we provide extensive experiments on FreiHAND dataset. Notably, our learning strategy improves the performance on hand pose and mesh estimation without any extra computational burden during the inference.
△ Less
Submitted 18 October, 2023;
originally announced October 2023.
-
Hybrid model for Single-Stage Multi-Person Pose Estimation
Authors:
Jonghyun Kim,
Bosang Kim,
Hyotae Lee,
Jungpyo Kim,
Wonhyeok Im,
Lanying **,
Dowoo Kwon,
Jungho Lee
Abstract:
In general, human pose estimation methods are categorized into two approaches according to their architectures: regression (i.e., heatmap-free) and heatmap-based methods. The former one directly estimates precise coordinates of each keypoint using convolutional and fully-connected layers. Although this approach is able to detect overlapped and dense keypoints, unexpected results can be obtained by…
▽ More
In general, human pose estimation methods are categorized into two approaches according to their architectures: regression (i.e., heatmap-free) and heatmap-based methods. The former one directly estimates precise coordinates of each keypoint using convolutional and fully-connected layers. Although this approach is able to detect overlapped and dense keypoints, unexpected results can be obtained by non-existent keypoints in a scene. On the other hand, the latter one is able to filter the non-existent ones out by utilizing predicted heatmaps for each keypoint. Nevertheless, it suffers from quantization error when obtaining the keypoint coordinates from its heatmaps. In addition, unlike the regression one, it is difficult to distinguish densely placed keypoints in an image. To this end, we propose a hybrid model for single-stage multi-person pose estimation, named HybridPose, which mutually overcomes each drawback of both approaches by maximizing their strengths. Furthermore, we introduce self-correlation loss to inject spatial dependencies between keypoint coordinates and their visibility. Therefore, HybridPose is capable of not only detecting densely placed keypoints, but also filtering the non-existent keypoints in an image. Experimental results demonstrate that proposed HybridPose exhibits the keypoints visibility without performance degradation in terms of the pose estimation accuracy.
△ Less
Submitted 18 June, 2023; v1 submitted 1 May, 2023;
originally announced May 2023.
-
Diffusion Probabilistic Models for Scene-Scale 3D Categorical Data
Authors:
Jumin Lee,
Woobin Im,
Sebin Lee,
Sung-Eui Yoon
Abstract:
In this paper, we learn a diffusion model to generate 3D data on a scene-scale. Specifically, our model crafts a 3D scene consisting of multiple objects, while recent diffusion research has focused on a single object. To realize our goal, we represent a scene with discrete class labels, i.e., categorical distribution, to assign multiple objects into semantic categories. Thus, we extend discrete di…
▽ More
In this paper, we learn a diffusion model to generate 3D data on a scene-scale. Specifically, our model crafts a 3D scene consisting of multiple objects, while recent diffusion research has focused on a single object. To realize our goal, we represent a scene with discrete class labels, i.e., categorical distribution, to assign multiple objects into semantic categories. Thus, we extend discrete diffusion models to learn scene-scale categorical distributions. In addition, we validate that a latent diffusion model can reduce computation costs for training and deploying. To the best of our knowledge, our work is the first to apply discrete and latent diffusion for 3D categorical data on a scene-scale. We further propose to perform semantic scene completion (SSC) by learning a conditional distribution using our diffusion model, where the condition is a partial observation in a sparse point cloud. In experiments, we empirically show that our diffusion models not only generate reasonable scenes, but also perform the scene completion task better than a discriminative model. Our code and models are available at https://github.com/zoomin-lee/scene-scale-diffusion
△ Less
Submitted 2 January, 2023;
originally announced January 2023.
-
Chiral Metafilms and Surface Enhanced Raman Scattering For Enantiomeric Discrimination of Helicoid Nanoparticles
Authors:
Martin Kartau,
Anastasia Skvortsova,
Victor Tabouillot,
Rahul Kumar,
Polina Bainovab,
Vasilii Burtsev,
Vaclav Svorcik,
Nikolaj Gadegaard,
Sang Won Im,
Marie Urbanova,
Oleksiy Lyutakov,
Malcolm Kadodwala,
Affar S. Karimullah
Abstract:
Chiral nanophotonic platforms provide a means of creating near fields with both enhanced asymmetric properties and intensities. They can be exploited for optical measurements that allow enantiomeric discrimination at detection levels greater than 6 orders of magnitude than is achieved with conventional chirally sensitive spectroscopic methods based on circularly polarized light. The optimal approa…
▽ More
Chiral nanophotonic platforms provide a means of creating near fields with both enhanced asymmetric properties and intensities. They can be exploited for optical measurements that allow enantiomeric discrimination at detection levels greater than 6 orders of magnitude than is achieved with conventional chirally sensitive spectroscopic methods based on circularly polarized light. The optimal approach for exploiting nanophotonic platforms for chiral detection would be to use spectroscopic methods that provide a local probe of changes in the near field environment induced by the presence of chiral species. Here we show that surface enhanced Raman spectroscopy (SERS) is such a local probe of the near field environment. We have used it to achieve enantiomeric discrimination of chiral helicoid nanoparticles deposited on left and right-handed enantiomorphs of a chiral metafilm. Hotter electromagnetic hotspots are created for matched combinations of helicoid and metafilms (left-left and right-right), while mismatched combinations leads to significantly cooler electromagnetic hotspots. This large enantiomeric dependency on hotspot intensity is readily detected using SERS with the aid of an achiral Raman reporter molecule. In effect we have used SERS to distinguish between the different EM environments of the plasmonic diastereomers produced by mixing chiral nanoparticles and metafilms. The work demonstrates that by combining chiral nanophotonic platforms with established SERS strategies new avenues in ultrasensitive chiral detection can be opened.
△ Less
Submitted 12 December, 2022;
originally announced December 2022.
-
Semi-Supervised Learning of Optical Flow by Flow Supervisor
Authors:
Woobin Im,
Sebin Lee,
Sung-Eui Yoon
Abstract:
A training pipeline for optical flow CNNs consists of a pretraining stage on a synthetic dataset followed by a fine tuning stage on a target dataset. However, obtaining ground truth flows from a target video requires a tremendous effort. This paper proposes a practical fine tuning method to adapt a pretrained model to a target dataset without ground truth flows, which has not been explored extensi…
▽ More
A training pipeline for optical flow CNNs consists of a pretraining stage on a synthetic dataset followed by a fine tuning stage on a target dataset. However, obtaining ground truth flows from a target video requires a tremendous effort. This paper proposes a practical fine tuning method to adapt a pretrained model to a target dataset without ground truth flows, which has not been explored extensively. Specifically, we propose a flow supervisor for self-supervision, which consists of parameter separation and a student output connection. This design is aimed at stable convergence and better accuracy over conventional self-supervision methods which are unstable on the fine tuning task. Experimental results show the effectiveness of our method compared to different self-supervision methods for semi-supervised learning. In addition, we achieve meaningful improvements over state-of-the-art optical flow models on Sintel and KITTI benchmarks by exploiting additional unlabeled datasets. Code is available at https://github.com/iwbn/flow-supervisor.
△ Less
Submitted 21 July, 2022;
originally announced July 2022.
-
Strain and Crystallographic Identification of the Helically Concaved Surfaces of Nanoparticles
Authors:
Sungwook Choi,
Sang Won Im,
Ji-Hyeok Huh,
Sungwon Kim,
Jaeseung Kim,
Yae-Chan Lim,
Ryeong Myeong Kim,
Jeong Hyun Han,
Hyeohn Kim,
Michael Sprung,
Su Yong Lee,
Wonsuk Cha,
Ross Harder,
Seungwoo Lee,
Ki Tae Nam,
Hyunjung Kim
Abstract:
Identifying the three-dimensional (3D) crystal-plane and strain-field distributions of nanocrystals is essential for optical, catalytic, and electronic applications. Here, we developed a methodology for visualizing the 3D information of chiral gold nanoparticles with concave gap structures by Bragg coherent X-ray diffraction imaging. The distribution of the high-Miller-index planes constituting th…
▽ More
Identifying the three-dimensional (3D) crystal-plane and strain-field distributions of nanocrystals is essential for optical, catalytic, and electronic applications. Here, we developed a methodology for visualizing the 3D information of chiral gold nanoparticles with concave gap structures by Bragg coherent X-ray diffraction imaging. The distribution of the high-Miller-index planes constituting the concave chiral gap was precisely determined. The highly strained region adjacent to the chiral gaps was resolved, which was correlated to the 432-symmetric morphology of the nanoparticles and its corresponding plasmonic properties were numerically predicted from the atomically defined structures. This approach can serve as a general characterization platform for visualizing the 3D crystallographic and strain distributions of nanoparticles, especially for applications where structural complexity and local heterogeneity are major determinants, as exemplified in plasmonics.
△ Less
Submitted 4 July, 2022;
originally announced July 2022.
-
In-N-Out: Towards Good Initialization for Inpainting and Outpainting
Authors:
Changho Jo,
Woobin Im,
Sung-Eui Yoon
Abstract:
In computer vision, recovering spatial information by filling in masked regions, e.g., inpainting, has been widely investigated for its usability and wide applicability to other various applications: image inpainting, image extrapolation, and environment map estimation. Most of them are studied separately depending on the applications. Our focus, however, is on accommodating the opposite task, e.g…
▽ More
In computer vision, recovering spatial information by filling in masked regions, e.g., inpainting, has been widely investigated for its usability and wide applicability to other various applications: image inpainting, image extrapolation, and environment map estimation. Most of them are studied separately depending on the applications. Our focus, however, is on accommodating the opposite task, e.g., image outpainting, which would benefit the target applications, e.g., image inpainting. Our self-supervision method, In-N-Out, is summarized as a training approach that leverages the knowledge of the opposite task into the target model. We empirically show that In-N-Out -- which explores the complementary information -- effectively takes advantage over the traditional pipelines where only task-specific learning takes place in training. In experiments, we compare our method to the traditional procedure and analyze the effectiveness of our method on different applications: image inpainting, image extrapolation, and environment map estimation. For these tasks, we demonstrate that In-N-Out consistently improves the performance of the recent works with In-N-Out self-supervision to their training procedure. Also, we show that our approach achieves better results than an existing training approach for outpainting.
△ Less
Submitted 17 September, 2021; v1 submitted 26 June, 2021;
originally announced June 2021.
-
Two-stream Spatiotemporal Feature for Video QA Task
Authors:
Chiwan Song,
Woobin Im,
Sung-eui Yoon
Abstract:
Understanding the content of videos is one of the core techniques for develo** various helpful applications in the real world, such as recognizing various human actions for surveillance systems or customer behavior analysis in an autonomous shop. However, understanding the content or story of the video still remains a challenging problem due to its sheer amount of data and temporal structure. In…
▽ More
Understanding the content of videos is one of the core techniques for develo** various helpful applications in the real world, such as recognizing various human actions for surveillance systems or customer behavior analysis in an autonomous shop. However, understanding the content or story of the video still remains a challenging problem due to its sheer amount of data and temporal structure. In this paper, we propose a multi-channel neural network structure that adopts a two-stream network structure, which has been shown high performance in human action recognition field, and use it as a spatiotemporal video feature extractor for solving video question and answering task. We also adopt a squeeze-and-excitation structure to two-stream network structure for achieving a channel-wise attended spatiotemporal feature. For jointly modeling the spatiotemporal features from video and the textual features from the question, we design a context matching module with a level adjusting layer to remove the gap of information between visual and textual features by applying attention mechanism on joint modeling. Finally, we adopt a scoring mechanism and smoothed ranking loss objective function for selecting the correct answer from answer candidates. We evaluate our model with TVQA dataset, and our approach shows the improved result in textual only setting, but the result with visual feature shows the limitation and possibility of our approach.
△ Less
Submitted 11 July, 2019;
originally announced July 2019.
-
Content-Based Video-Music Retrieval Using Soft Intra-Modal Structure Constraint
Authors:
Sungeun Hong,
Woobin Im,
Hyun S. Yang
Abstract:
Up to now, only limited research has been conducted on cross-modal retrieval of suitable music for a specified video or vice versa. Moreover, much of the existing research relies on metadata such as keywords, tags, or associated description that must be individually produced and attached posterior. This paper introduces a new content-based, cross-modal retrieval method for video and music that is…
▽ More
Up to now, only limited research has been conducted on cross-modal retrieval of suitable music for a specified video or vice versa. Moreover, much of the existing research relies on metadata such as keywords, tags, or associated description that must be individually produced and attached posterior. This paper introduces a new content-based, cross-modal retrieval method for video and music that is implemented through deep neural networks. We train the network via inter-modal ranking loss such that videos and music with similar semantics end up close together in the embedding space. However, if only the inter-modal ranking constraint is used for embedding, modality-specific characteristics can be lost. To address this problem, we propose a novel soft intra-modal structure loss that leverages the relative distance relationship between intra-modal samples before embedding. We also introduce reasonable quantitative and qualitative experimental protocols to solve the lack of standard protocols for less-mature video-music related tasks. Finally, we construct a large-scale 200K video-music pair benchmark. All the datasets and source code can be found in our online repository (https://github.com/csehong/VM-NET).
△ Less
Submitted 1 September, 2017; v1 submitted 22 April, 2017;
originally announced April 2017.
-
Recognizing Dynamic Scenes with Deep Dual Descriptor based on Key Frames and Key Segments
Authors:
Sungeun Hong,
Jongbin Ryu,
Woobin Im,
Hyun S. Yang
Abstract:
Recognizing dynamic scenes is one of the fundamental problems in scene understanding, which categorizes moving scenes such as a forest fire, landslide, or avalanche. While existing methods focus on reliable capturing of static and dynamic information, few works have explored frame selection from a dynamic scene sequence. In this paper, we propose dynamic scene recognition using a deep dual descrip…
▽ More
Recognizing dynamic scenes is one of the fundamental problems in scene understanding, which categorizes moving scenes such as a forest fire, landslide, or avalanche. While existing methods focus on reliable capturing of static and dynamic information, few works have explored frame selection from a dynamic scene sequence. In this paper, we propose dynamic scene recognition using a deep dual descriptor based on `key frames' and `key segments.' Key frames that reflect the feature distribution of the sequence with a small number are used for capturing salient static appearances. Key segments, which are captured from the area around each key frame, provide an additional discriminative power by dynamic patterns within short time intervals. To this end, two types of transferred convolutional neural network features are used in our approach. A fully connected layer is used to select the key frames and key segments, while the convolutional layer is used to describe them. We conducted experiments using public datasets as well as a new dataset comprised of 23 dynamic scene classes with 10 videos per class. The evaluation results demonstrated the state-of-the-art performance of the proposed method.
△ Less
Submitted 16 February, 2017; v1 submitted 15 February, 2017;
originally announced February 2017.
-
SSPP-DAN: Deep Domain Adaptation Network for Face Recognition with Single Sample Per Person
Authors:
Sungeun Hong,
Woobin Im,
Jongbin Ryu,
Hyun S. Yang
Abstract:
Real-world face recognition using a single sample per person (SSPP) is a challenging task. The problem is exacerbated if the conditions under which the gallery image and the probe set are captured are completely different. To address these issues from the perspective of domain adaptation, we introduce an SSPP domain adaptation network (SSPP-DAN). In the proposed approach, domain adaptation, featur…
▽ More
Real-world face recognition using a single sample per person (SSPP) is a challenging task. The problem is exacerbated if the conditions under which the gallery image and the probe set are captured are completely different. To address these issues from the perspective of domain adaptation, we introduce an SSPP domain adaptation network (SSPP-DAN). In the proposed approach, domain adaptation, feature extraction, and classification are performed jointly using a deep architecture with domain-adversarial training. However, the SSPP characteristic of one training sample per class is insufficient to train the deep architecture. To overcome this shortage, we generate synthetic images with varying poses using a 3D face model. Experimental evaluations using a realistic SSPP dataset show that deep domain adaptation and image synthesis complement each other and dramatically improve accuracy. Experiments on a benchmark dataset using the proposed approach show state-of-the-art performance. All the dataset and the source code can be found in our online repository (https://github.com/csehong/SSPP-DAN).
△ Less
Submitted 28 April, 2018; v1 submitted 13 February, 2017;
originally announced February 2017.
-
Image-Text Multi-Modal Representation Learning by Adversarial Backpropagation
Authors:
Gwangbeen Park,
Woobin Im
Abstract:
We present novel method for image-text multi-modal representation learning. In our knowledge, this work is the first approach of applying adversarial learning concept to multi-modal learning and not exploiting image-text pair information to learn multi-modal feature. We only use category information in contrast with most previous methods using image-text pair information for multi-modal embedding.…
▽ More
We present novel method for image-text multi-modal representation learning. In our knowledge, this work is the first approach of applying adversarial learning concept to multi-modal learning and not exploiting image-text pair information to learn multi-modal feature. We only use category information in contrast with most previous methods using image-text pair information for multi-modal embedding. In this paper, we show that multi-modal feature can be achieved without image-text pair information and our method makes more similar distribution with image and text in multi-modal feature space than other methods which use image-text pair information. And we show our multi-modal feature has universal semantic information, even though it was trained for category prediction. Our model is end-to-end backpropagation, intuitive and easily extended to other multi-modal learning work.
△ Less
Submitted 26 December, 2016;
originally announced December 2016.
