-
SAVE: Segment Audio-Visual Easy way using Segment Anything Model
Authors:
Khanh-Binh Nguyen,
Chae Jung Park
Abstract:
The primary aim of Audio-Visual Segmentation (AVS) is to precisely identify and locate auditory elements within visual scenes by accurately predicting segmentation masks at the pixel level. Achieving this involves comprehensively considering data and model aspects to address this task effectively. This study presents a lightweight approach, SAVE, which efficiently adapts the pre-trained segment an…
▽ More
The primary aim of Audio-Visual Segmentation (AVS) is to precisely identify and locate auditory elements within visual scenes by accurately predicting segmentation masks at the pixel level. Achieving this involves comprehensively considering data and model aspects to address this task effectively. This study presents a lightweight approach, SAVE, which efficiently adapts the pre-trained segment anything model (SAM) to the AVS task. By incorporating an image encoder adapter into the transformer blocks to better capture the distinct dataset information and proposing a residual audio encoder adapter to encode the audio features as a sparse prompt, our proposed model achieves effective audio-visual fusion and interaction during the encoding stage. Our proposed method accelerates the training and inference speed by reducing the input resolution from 1024 to 256 pixels while achieving higher performance compared with the previous SOTA. Extensive experimentation validates our approach, demonstrating that our proposed model outperforms other SOTA methods significantly. Moreover, leveraging the pre-trained model on synthetic data enhances performance on real AVSBench data, achieving 84.59 mIoU on the S4 (V1S) subset and 70.28 mIoU on the MS3 (V1M) set with only 256 pixels for input images. This increases up to 86.16 mIoU on the S4 (V1S) and 70.83 mIoU on the MS3 (V1M) with inputs of 1024 pixels.
△ Less
Submitted 2 July, 2024;
originally announced July 2024.
-
Retro: Reusing teacher projection head for efficient embedding distillation on Lightweight Models via Self-supervised Learning
Authors:
Khanh-Binh Nguyen,
Chae Jung Park
Abstract:
Self-supervised learning (SSL) is gaining attention for its ability to learn effective representations with large amounts of unlabeled data.
Lightweight models can be distilled from larger self-supervised pre-trained models using contrastive and consistency constraints.
Still, the different sizes of the projection heads make it challenging for students to mimic the teacher's embedding accurate…
▽ More
Self-supervised learning (SSL) is gaining attention for its ability to learn effective representations with large amounts of unlabeled data.
Lightweight models can be distilled from larger self-supervised pre-trained models using contrastive and consistency constraints.
Still, the different sizes of the projection heads make it challenging for students to mimic the teacher's embedding accurately.
We propose \textsc{Retro}, which reuses the teacher's projection head for students, and our experimental results demonstrate significant improvements over the state-of-the-art on all lightweight models.
For instance, when training EfficientNet-B0 using ResNet-50/101/152 as teachers, our approach improves the linear result on ImageNet to $66.9\%$, $69.3\%$, and $69.8\%$, respectively, with significantly fewer parameters.
△ Less
Submitted 26 May, 2024; v1 submitted 24 May, 2024;
originally announced May 2024.
-
A human brain atlas of chi-separation for normative iron and myelin distributions
Authors:
Kyeongseon Min,
Beomseok Sohn,
Woo Jung Kim,
Chae Jung Park,
Soohwa Song,
Dong Hoon Shin,
Kyung Won Chang,
Na-Young Shin,
Minjun Kim,
Hyeong-Geol Shin,
Phil Hyu Lee,
Jongho Lee
Abstract:
Iron and myelin are primary susceptibility sources in the human brain. These substances are essential for healthy brain, and their abnormalities are often related to various neurological disorders. Recently, an advanced susceptibility map** technique, which is referred to as chi-separation, has been proposed, successfully disentangling paramagnetic iron from diamagnetic myelin. This method opene…
▽ More
Iron and myelin are primary susceptibility sources in the human brain. These substances are essential for healthy brain, and their abnormalities are often related to various neurological disorders. Recently, an advanced susceptibility map** technique, which is referred to as chi-separation, has been proposed, successfully disentangling paramagnetic iron from diamagnetic myelin. This method opened a potential for generating high resolution iron and myelin maps in the brain. Utilizing this technique, this study constructs a normative chi-separation atlas from 106 healthy human brains. The resulting atlas provides detailed anatomical structures associated with the distributions of iron and myelin, clearly delineating subcortical nuclei, thalamic nuclei, and white matter fiber bundles. Additionally, susceptibility values in a number of regions of interest are reported along with age-dependent changes. This atlas may have direct applications such as localization of subcortical structures for deep brain stimulation or high-intensity focused ultrasound and also serve as a valuable resource for future research.
△ Less
Submitted 2 April, 2024; v1 submitted 8 November, 2023;
originally announced November 2023.
-
Conformal Couplings in Induced Gravity
Authors:
C. J. Park,
Yongsung Yoon
Abstract:
It is found that the induced gravity with conformal couplings requires the conformal invariance in both classical and quantum levels for consistency. This is also true for the induced gravity with an extended conformal coupling interacting with torsion.
It is found that the induced gravity with conformal couplings requires the conformal invariance in both classical and quantum levels for consistency. This is also true for the induced gravity with an extended conformal coupling interacting with torsion.
△ Less
Submitted 22 November, 1996;
originally announced November 1996.
-
The Constraint of a General Effective Potential in Vector Torsion Coupled Conformally Induced Gravity
Authors:
Jewan Kim,
C. J. Park,
Yongsung Yoon
Abstract:
It is found that the deviation of an effective potential from the quartic form is related to the metric and vector torsion dependencies of the effective potential in the vector torsion coupled conformally induced gravity.
It is found that the deviation of an effective potential from the quartic form is related to the metric and vector torsion dependencies of the effective potential in the vector torsion coupled conformally induced gravity.
△ Less
Submitted 6 November, 1994; v1 submitted 6 June, 1994;
originally announced June 1994.
-
Phase Transition in Conformally Induced Gravity with Torsion
Authors:
Jewan Kim,
C. J. Park,
Yongsung Yoon
Abstract:
We have considered the quantum behavior of a conformally induced gravity in the minimal Riemann-Cartan space. The regularized one-loop effective potential considering the quantum fluctuations of the dilaton and the torsion fields in the Coleman-Weinberg sector gives a sensible phase transition for an inflationary phase in De Sitter space. For this effective potential, we have analyzed the semi-c…
▽ More
We have considered the quantum behavior of a conformally induced gravity in the minimal Riemann-Cartan space. The regularized one-loop effective potential considering the quantum fluctuations of the dilaton and the torsion fields in the Coleman-Weinberg sector gives a sensible phase transition for an inflationary phase in De Sitter space. For this effective potential, we have analyzed the semi-classical equation of motion of the dilaton field in the slow-rolling regime.
△ Less
Submitted 25 May, 1994;
originally announced May 1994.
