Showing 101–150 of 9,784 results for author: Li, H

Decentralized Collaborative Pricing and Shunting for Multiple EV Charging Stations Based on Multi-Agent Reinforcement Learning

Authors: Tianhao Bu, Hang Li, Guojie Li

Abstract: The extraordinary electric vehicle (EV) popularization in the recent years has facilitated research studies in alleviating EV energy charging demand. Previous studies primarily focused on the optimizations over charging stations (CS) profit and EV users cost savings through charge/discharge scheduling events. In this work, the random behaviors of EVs are considered, with EV users preferences over… ▽ More The extraordinary electric vehicle (EV) popularization in the recent years has facilitated research studies in alleviating EV energy charging demand. Previous studies primarily focused on the optimizations over charging stations (CS) profit and EV users cost savings through charge/discharge scheduling events. In this work, the random behaviors of EVs are considered, with EV users preferences over multi-CS characteristics modelled to imitate the potential CS selection disequilibrium. A price scheduling strategy under decentralized collaborative framework is proposed to achieve EV shunting in a multi-CS environment, while minimizing the charging cost through multi agent reinforcement learning. The proposed problem is formulated as a Markov Decision Process (MDP) with uncertain transition probability. △ Less

Submitted 17 June, 2024; originally announced June 2024.

Comments: Accepted at PSGEC: PSGEC2024

An Exploration of Length Generalization in Transformer-Based Speech Enhancement

Authors: Qiquan Zhang, Hongxu Zhu, Xinyuan Qian, Eliathamby Ambikairajah, Haizhou Li

Abstract: The use of Transformer architectures has facilitated remarkable progress in speech enhancement. Training Transformers using substantially long speech utterances is often infeasible as self-attention suffers from quadratic complexity. It is a critical and unexplored challenge for a Transformer-based speech enhancement model to learn from short speech utterances and generalize to longer ones. In thi… ▽ More The use of Transformer architectures has facilitated remarkable progress in speech enhancement. Training Transformers using substantially long speech utterances is often infeasible as self-attention suffers from quadratic complexity. It is a critical and unexplored challenge for a Transformer-based speech enhancement model to learn from short speech utterances and generalize to longer ones. In this paper, we conduct comprehensive experiments to explore the length generalization problem in speech enhancement with Transformer. Our findings first establish that position embedding provides an effective instrument to alleviate the impact of utterance length on Transformer-based speech enhancement. Specifically, we explore four different position embedding schemes to enable length generalization. The results confirm the superiority of relative position embeddings (RPEs) over absolute PE (APEs) in length generalization. △ Less

Submitted 17 June, 2024; originally announced June 2024.

Comments: Accepted by INTERSPEECH 2024

Parameter effects on the total intensity of H I Lyα line for a modelled coronal mass ejection and its driven shock

Authors: Beili Ying, Guanglu Shi, Li Feng, Lei Lu, Jianchao Xue, Shuting Li, Weiqun Gan, Hui Li

Abstract: The combination of the H I Lyα (121.6 nm) line formation mechanism with ultraviolet (UV) Lyα and white-light (WL) observations provides an effective method for determining the electron temperature of coronal mass ejections (CMEs). A key to ensuring the accuracy of this diagnostic technique is the precise calculation of theoretical Lyα intensities. This study performs a modelled CME and its driven… ▽ More The combination of the H I Lyα (121.6 nm) line formation mechanism with ultraviolet (UV) Lyα and white-light (WL) observations provides an effective method for determining the electron temperature of coronal mass ejections (CMEs). A key to ensuring the accuracy of this diagnostic technique is the precise calculation of theoretical Lyα intensities. This study performs a modelled CME and its driven shock via the 3D MHD simulation. We generate synthetic UV and WL images of the CME and shock to quantify the impact of different assumptions on theoretical Lyα intensities, such as the incident intensity of the Lyα line (Idisk), the geometric scattering function (p(θ)), and the kinetic temperature (Tn) assumed to be equal to the proton (Tp) or electron (Te) temperatures. By comparing differences of the Lyα intensities under these assumptions, we find that: (1) Using the uniform or Carrington maps of the disk Lyα emission underestimates the corona Lyα intensity (< 10%) compared to the synchronic map, except for a slight overestimate (< 4%) in the partial CME core. The Carrington map yields lower uncertainties than the uniform disk. (2) The geometric scattering process has a minor impact on the Lyα intensity, with a maximum relative uncertainty of < 5%. The Lyα intensity is underestimated for the most part but overestimated in the CME core. (3) Compared to the assumption Tn = Tp, using Tn = Te leads to more complex relative uncertainties in CME Lyα intensity. The CME core and void are both overestimated, with the maximum uncertainty in the core exceeding 50% and the void remaining below 35%. In the CME front, both over- and under-estimates exist with relative uncertainties of < 35%. The electron temperature assumption has a smaller impact on the shock, with an underestimated relative uncertainty of less than 20%. △ Less

Submitted 17 June, 2024; originally announced June 2024.

Comments: 22 pages, 9 figures, accepted by Solar Physics

GoldCoin: Grounding Large Language Models in Privacy Laws via Contextual Integrity Theory

Authors: Wei Fan, Haoran Li, Zheye Deng, Weiqi Wang, Yangqiu Song

Abstract: Privacy issues arise prominently during the inappropriate transmission of information between entities. Existing research primarily studies privacy by exploring various privacy attacks, defenses, and evaluations within narrowly predefined patterns, while neglecting that privacy is not an isolated, context-free concept limited to traditionally sensitive data (e.g., social security numbers), but int… ▽ More Privacy issues arise prominently during the inappropriate transmission of information between entities. Existing research primarily studies privacy by exploring various privacy attacks, defenses, and evaluations within narrowly predefined patterns, while neglecting that privacy is not an isolated, context-free concept limited to traditionally sensitive data (e.g., social security numbers), but intertwined with intricate social contexts that complicate the identification and analysis of potential privacy violations. The advent of Large Language Models (LLMs) offers unprecedented opportunities for incorporating the nuanced scenarios outlined in privacy laws to tackle these complex privacy issues. However, the scarcity of open-source relevant case studies restricts the efficiency of LLMs in aligning with specific legal statutes. To address this challenge, we introduce a novel framework, GoldCoin, designed to efficiently ground LLMs in privacy laws for judicial assessing privacy violations. Our framework leverages the theory of contextual integrity as a bridge, creating numerous synthetic scenarios grounded in relevant privacy statutes (e.g., HIPAA), to assist LLMs in comprehending the complex contexts for identifying privacy risks in the real world. Extensive experimental results demonstrate that GoldCoin markedly enhances LLMs' capabilities in recognizing privacy risks across real court cases, surpassing the baselines on different judicial tasks. △ Less

Submitted 16 June, 2024; originally announced June 2024.

Toward Optimal LLM Alignments Using Two-Player Games

Authors: Rui Zheng, Hongyi Guo, Zhihan Liu, Xiaoying Zhang, Yuanshun Yao, Xiaojun Xu, Zhaoran Wang, Zhiheng Xi, Tao Gui, Qi Zhang, Xuan**g Huang, Hang Li, Yang Liu

Abstract: The standard Reinforcement Learning from Human Feedback (RLHF) framework primarily focuses on optimizing the performance of large language models using pre-collected prompts. However, collecting prompts that provide comprehensive coverage is both tedious and challenging, and often fails to include scenarios that LLMs need to improve on the most. In this paper, we investigate alignment through the… ▽ More The standard Reinforcement Learning from Human Feedback (RLHF) framework primarily focuses on optimizing the performance of large language models using pre-collected prompts. However, collecting prompts that provide comprehensive coverage is both tedious and challenging, and often fails to include scenarios that LLMs need to improve on the most. In this paper, we investigate alignment through the lens of two-agent games, involving iterative interactions between an adversarial and a defensive agent. The adversarial agent's task at each step is to generate prompts that expose the weakness of the defensive agent. In return, the defensive agent seeks to improve its responses to these newly identified prompts it struggled with, based on feedback from the reward model. We theoretically demonstrate that this iterative reinforcement learning optimization converges to a Nash Equilibrium for the game induced by the agents. Experimental results in safety scenarios demonstrate that learning in such a competitive environment not only fully trains agents but also leads to policies with enhanced generalization capabilities for both adversarial and defensive agents. △ Less

Submitted 16 June, 2024; originally announced June 2024.

Comments: Our code is released at https://github.com/ruizheng20/gpo

MSC Class: 68

On the Role of Entity and Event Level Conceptualization in Generalizable Reasoning: A Survey of Tasks, Methods, Applications, and Future Directions

Authors: Weiqi Wang, Tianqing Fang, Haochen Shi, Baixuan Xu, Wenxuan Ding, Liyu Zhang, Wei Fan, Jiaxin Bai, Haoran Li, Xin Liu, Yangqiu Song

Abstract: Entity- and event-level conceptualization, as fundamental elements of human cognition, plays a pivotal role in generalizable reasoning. This process involves abstracting specific instances into higher-level concepts and forming abstract knowledge that can be applied in unfamiliar or novel situations, which can enhance models' inferential capabilities and support the effective transfer of knowledge… ▽ More Entity- and event-level conceptualization, as fundamental elements of human cognition, plays a pivotal role in generalizable reasoning. This process involves abstracting specific instances into higher-level concepts and forming abstract knowledge that can be applied in unfamiliar or novel situations, which can enhance models' inferential capabilities and support the effective transfer of knowledge across various domains. Despite its significance, there is currently a lack of a systematic overview that comprehensively examines existing works in the definition, execution, and application of conceptualization to enhance reasoning tasks. In this paper, we address this gap by presenting the first comprehensive survey of 150+ papers, categorizing various definitions, resources, methods, and downstream applications related to conceptualization into a unified taxonomy, with a focus on the entity and event levels. Furthermore, we shed light on potential future directions in this field and hope to garner more attention from the community. △ Less

Submitted 16 June, 2024; originally announced June 2024.

K-contact manifolds with minimal closed Reeb orbits

Authors: Hui Li

Abstract: We use the Boothby-Wang fibration to construct certain simply connected K-contact manifolds and we give sufficient and necessary conditions on when such K-contact manifolds are homeomorphic to the odd dimensional spheres. For the symplectic base manifold of the fibration which admits a Hamiltonian torus action, we show that the total space of the fibration admits other than the above K-contact str… ▽ More We use the Boothby-Wang fibration to construct certain simply connected K-contact manifolds and we give sufficient and necessary conditions on when such K-contact manifolds are homeomorphic to the odd dimensional spheres. For the symplectic base manifold of the fibration which admits a Hamiltonian torus action, we show that the total space of the fibration admits other than the above K-contact structures whose closed Reeb orbits correspond exactly to the fixed points of the Hamiltonian circle action on the base manifold. Then we give a collection of examples of compact simply connected K-contact manifolds with minimal number of closed Reeb orbits which are not homeomorphic to the odd dimensional spheres, while having the real cohomology ring of the spheres. Finally, we give a family of examples of simply connected K-contact manifolds which have one more than the minimal number of closed Reeb orbits and which do not have the real cohomology ring of the spheres. △ Less

Submitted 16 June, 2024; originally announced June 2024.

MSC Class: Primary: 53D10; 53D05; 53D20; Secondary: 55N10; 57R20

Multi-Scale Accent Modeling with Disentangling for Multi-Speaker Multi-Accent TTS Synthesis

Authors: Xuehao Zhou, Mingyang Zhang, Yi Zhou, Zhizheng Wu, Haizhou Li

Abstract: Synthesizing speech across different accents while preserving the speaker identity is essential for various real-world customer applications. However, the individual and accurate modeling of accents and speakers in a text-to-speech (TTS) system is challenging due to the complexity of accent variations and the intrinsic entanglement between the accent and speaker identity. In this paper, we present… ▽ More Synthesizing speech across different accents while preserving the speaker identity is essential for various real-world customer applications. However, the individual and accurate modeling of accents and speakers in a text-to-speech (TTS) system is challenging due to the complexity of accent variations and the intrinsic entanglement between the accent and speaker identity. In this paper, we present a novel approach for multi-speaker multi-accent TTS synthesis, which aims to synthesize voices of multiple speakers, each with various accents. Our proposed approach employs a multi-scale accent modeling strategy to address accent variations at different levels. Specifically, we introduce both global (utterance level) and local (phoneme level) accent modeling, supervised by individual accent classifiers to capture the overall variation within accented utterances and fine-grained variations between phonemes, respectively. To control accents and speakers separately, speaker-independent accent modeling is necessary, which is achieved by adversarial training with speaker classifiers to disentangle speaker identity within the multi-scale accent modeling. Consequently, we obtain speaker-independent and accent-discriminative multi-scale embeddings as comprehensive accent features. Additionally, we propose a local accent prediction model that allows to generate accented speech directly from phoneme inputs. Extensive experiments are conducted on an accented English speech corpus. Both objective and subjective evaluations show the superiority of our proposed system compared to baselines systems. Detailed component analysis demonstrates the effectiveness of global and local accent modeling, and speaker disentanglement on multi-speaker multi-accent speech synthesis. △ Less

Submitted 16 June, 2024; originally announced June 2024.

Technique Report of CVPR 2024 PBDL Challenges

Authors: Ying Fu, Yu Li, Shaodi You, Boxin Shi, Jose Alvarez, Coert van Gemeren, Linwei Chen, Yunhao Zou, Zichun Wang, Yichen Li, Yuze Han, Yingkai Zhang, Jianan Wang, Qinglin Liu, Wei Yu, Xiaoqian Lv, Jianing Li, Sheng** Zhang, Xiangyang Ji, Yuanpei Chen, Yuhan Zhang, Weihang Peng, Liwen Zhang, Zhe Xu, Dingyong Gou , et al. (77 additional authors not shown)

Abstract: The intersection of physics-based vision and deep learning presents an exciting frontier for advancing computer vision technologies. By leveraging the principles of physics to inform and enhance deep learning models, we can develop more robust and accurate vision systems. Physics-based vision aims to invert the processes to recover scene properties such as shape, reflectance, light distribution, a… ▽ More The intersection of physics-based vision and deep learning presents an exciting frontier for advancing computer vision technologies. By leveraging the principles of physics to inform and enhance deep learning models, we can develop more robust and accurate vision systems. Physics-based vision aims to invert the processes to recover scene properties such as shape, reflectance, light distribution, and medium properties from images. In recent years, deep learning has shown promising improvements for various vision tasks, and when combined with physics-based vision, these approaches can enhance the robustness and accuracy of vision systems. This technical report summarizes the outcomes of the Physics-Based Vision Meets Deep Learning (PBDL) 2024 challenge, held in CVPR 2024 workshop. The challenge consisted of eight tracks, focusing on Low-Light Enhancement and Detection as well as High Dynamic Range (HDR) Imaging. This report details the objectives, methodologies, and results of each track, highlighting the top-performing solutions and their innovative approaches. △ Less

Submitted 27 June, 2024; v1 submitted 15 June, 2024; originally announced June 2024.

Comments: The author list and contents need to be verified by all authors

UniZero: Generalized and Efficient Planning with Scalable Latent World Models

Authors: Yuan Pu, Yazhe Niu, Jiyuan Ren, Zhenjie Yang, Hongsheng Li, Yu Liu

Abstract: Learning predictive world models is essential for enhancing the planning capabilities of reinforcement learning agents. Notably, the MuZero-style algorithms, based on the value equivalence principle and Monte Carlo Tree Search (MCTS), have achieved superhuman performance in various domains. However, in environments that require capturing long-term dependencies, MuZero's performance deteriorates ra… ▽ More Learning predictive world models is essential for enhancing the planning capabilities of reinforcement learning agents. Notably, the MuZero-style algorithms, based on the value equivalence principle and Monte Carlo Tree Search (MCTS), have achieved superhuman performance in various domains. However, in environments that require capturing long-term dependencies, MuZero's performance deteriorates rapidly. We identify that this is partially due to the \textit{entanglement} of latent representations with historical information, which results in incompatibility with the auxiliary self-supervised state regularization. To overcome this limitation, we present \textit{UniZero}, a novel approach that \textit{disentangles} latent states from implicit latent history using a transformer-based latent world model. By concurrently predicting latent dynamics and decision-oriented quantities conditioned on the learned latent history, UniZero enables joint optimization of the long-horizon world model and policy, facilitating broader and more efficient planning in latent space. We demonstrate that UniZero, even with single-frame inputs, matches or surpasses the performance of MuZero-style algorithms on the Atari 100k benchmark. Furthermore, it significantly outperforms prior baselines in benchmarks that require long-term memory. Lastly, we validate the effectiveness and scalability of our design choices through extensive ablation studies, visual analyses, and multi-task learning results. The code is available at \textcolor{magenta}{https://github.com/opendilab/LightZero}. △ Less

Submitted 15 June, 2024; originally announced June 2024.

Comments: 32 pages, 16 figures

CrossFuse: A Novel Cross Attention Mechanism based Infrared and Visible Image Fusion Approach

Authors: Hui Li, Xiao-Jun Wu

Abstract: Multimodal visual information fusion aims to integrate the multi-sensor data into a single image which contains more complementary information and less redundant features. However the complementary information is hard to extract, especially for infrared and visible images which contain big similarity gap between these two modalities. The common cross attention modules only consider the correlation… ▽ More Multimodal visual information fusion aims to integrate the multi-sensor data into a single image which contains more complementary information and less redundant features. However the complementary information is hard to extract, especially for infrared and visible images which contain big similarity gap between these two modalities. The common cross attention modules only consider the correlation, on the contrary, image fusion tasks need focus on complementarity (uncorrelation). Hence, in this paper, a novel cross attention mechanism (CAM) is proposed to enhance the complementary information. Furthermore, a two-stage training strategy based fusion scheme is presented to generate the fused images. For the first stage, two auto-encoder networks with same architecture are trained for each modality. Then, with the fixed encoders, the CAM and a decoder are trained in the second stage. With the trained CAM, features extracted from two modalities are integrated into one fused feature in which the complementary information is enhanced and the redundant features are reduced. Finally, the fused image can be generated by the trained decoder. The experimental results illustrate that our proposed fusion method obtains the SOTA fusion performance compared with the existing fusion networks. The codes are available at https://github.com/hli1221/CrossFuse △ Less

Submitted 15 June, 2024; originally announced June 2024.

Comments: 11 pages, 16 fuigures

Universal materials model of deep-learning density functional theory Hamiltonian

Authors: Yuxiang Wang, Yang Li, Zechen Tang, He Li, Zilong Yuan, Honggeng Tao, Nianlong Zou, Ting Bao, Xinghao Liang, Zezhou Chen, Shanghua Xu, Ce Bian, Zhiming Xu, Chong Wang, Chen Si, Wenhui Duan, Yong Xu

Abstract: Realizing large materials models has emerged as a critical endeavor for materials research in the new era of artificial intelligence, but how to achieve this fantastic and challenging objective remains elusive. Here, we propose a feasible pathway to address this paramount pursuit by develo** universal materials models of deep-learning density functional theory Hamiltonian (DeepH), enabling compu… ▽ More Realizing large materials models has emerged as a critical endeavor for materials research in the new era of artificial intelligence, but how to achieve this fantastic and challenging objective remains elusive. Here, we propose a feasible pathway to address this paramount pursuit by develo** universal materials models of deep-learning density functional theory Hamiltonian (DeepH), enabling computational modeling of the complicated structure-property relationship of materials in general. By constructing a large materials database and substantially improving the DeepH method, we obtain a universal materials model of DeepH capable of handling diverse elemental compositions and material structures, achieving remarkable accuracy in predicting material properties. We further showcase a promising application of fine-tuning universal materials models for enhancing specific materials models. This work not only demonstrates the concept of DeepH's universal materials model but also lays the groundwork for develo** large materials models, opening up significant opportunities for advancing artificial intelligence-driven materials discovery. △ Less

Submitted 15 June, 2024; originally announced June 2024.

Panoptic-FlashOcc: An Efficient Baseline to Marry Semantic Occupancy with Panoptic via Instance Center

Authors: Zichen Yu, Changyong Shu, Qianpu Sun, Junjie Linghu, Xiaobao Wei, Jiangyong Yu, Zongdai Liu, Dawei Yang, Hui Li, Yan Chen

Abstract: Panoptic occupancy poses a novel challenge by aiming to integrate instance occupancy and semantic occupancy within a unified framework. However, there is still a lack of efficient solutions for panoptic occupancy. In this paper, we propose Panoptic-FlashOcc, a straightforward yet robust 2D feature framework that enables realtime panoptic occupancy. Building upon the lightweight design of FlashOcc,… ▽ More Panoptic occupancy poses a novel challenge by aiming to integrate instance occupancy and semantic occupancy within a unified framework. However, there is still a lack of efficient solutions for panoptic occupancy. In this paper, we propose Panoptic-FlashOcc, a straightforward yet robust 2D feature framework that enables realtime panoptic occupancy. Building upon the lightweight design of FlashOcc, our approach simultaneously learns semantic occupancy and class-aware instance clustering in a single network, these outputs are jointly incorporated through panoptic occupancy procession for panoptic occupancy. This approach effectively addresses the drawbacks of high memory and computation requirements associated with three-dimensional voxel-level representations. With its straightforward and efficient design that facilitates easy deployment, Panoptic-FlashOcc demonstrates remarkable achievements in panoptic occupancy prediction. On the Occ3D-nuScenes benchmark, it achieves exceptional performance, with 38.5 RayIoU and 29.1 mIoU for semantic occupancy, operating at a rapid speed of 43.9 FPS. Furthermore, it attains a notable score of 16.0 RayPQ for panoptic occupancy, accompanied by a fast inference speed of 30.2 FPS. These results surpass the performance of existing methodologies in terms of both speed and accuracy. The source code and trained models can be found at the following github repository: https://github.com/Yzichen/FlashOCC. △ Less

Submitted 15 June, 2024; originally announced June 2024.

Self Pre-training with Topology- and Spatiality-aware Masked Autoencoders for 3D Medical Image Segmentation

Authors: Pengfei Gu, Yejia Zhang, Huimin Li, Hongxiao Wang, Yizhe Zhang, Chaoli Wang, Danny Z. Chen

Abstract: Masked Autoencoders (MAEs) have been shown to be effective in pre-training Vision Transformers (ViTs) for natural and medical image analysis problems. By reconstructing missing pixel/voxel information in visible patches, a ViT encoder can aggregate contextual information for downstream tasks. But, existing MAE pre-training methods, which were specifically developed with the ViT architecture, lack… ▽ More Masked Autoencoders (MAEs) have been shown to be effective in pre-training Vision Transformers (ViTs) for natural and medical image analysis problems. By reconstructing missing pixel/voxel information in visible patches, a ViT encoder can aggregate contextual information for downstream tasks. But, existing MAE pre-training methods, which were specifically developed with the ViT architecture, lack the ability to capture geometric shape and spatial information, which is critical for medical image segmentation tasks. In this paper, we propose a novel extension of known MAEs for self pre-training (i.e., models pre-trained on the same target dataset) for 3D medical image segmentation. (1) We propose a new topological loss to preserve geometric shape information by computing topological signatures of both the input and reconstructed volumes, learning geometric shape information. (2) We introduce a pre-text task that predicts the positions of the centers and eight corners of 3D crops, enabling the MAE to aggregate spatial information. (3) We extend the MAE pre-training strategy to a hybrid state-of-the-art (SOTA) medical image segmentation architecture and co-pretrain it alongside the ViT. (4) We develop a fine-tuned model for downstream segmentation tasks by complementing the pre-trained ViT encoder with our pre-trained SOTA model. Extensive experiments on five public 3D segmentation datasets show the effectiveness of our new approach. △ Less

Submitted 15 June, 2024; originally announced June 2024.

Task Facet Learning: A Structured Approach to Prompt Optimization

Authors: Gurusha Juneja, Nagarajan Natarajan, Hua Li, Jian Jiao, Amit Sharma

Abstract: Given a task in the form of a basic description and its training examples, prompt optimization is the problem of synthesizing the given information into a text prompt for a large language model (LLM). Humans solve this problem by also considering the different facets that define a task (e.g., counter-examples, explanations, analogies) and including them in the prompt. However, it is unclear whethe… ▽ More Given a task in the form of a basic description and its training examples, prompt optimization is the problem of synthesizing the given information into a text prompt for a large language model (LLM). Humans solve this problem by also considering the different facets that define a task (e.g., counter-examples, explanations, analogies) and including them in the prompt. However, it is unclear whether existing algorithmic approaches, based on iteratively editing a given prompt or automatically selecting a few in-context examples, can cover the multiple facets required to solve a complex task. In this work, we view prompt optimization as that of learning multiple facets of a task from a set of training examples. We identify and exploit structure in the prompt optimization problem -- first, we find that prompts can be broken down into loosely coupled semantic sections that have a relatively independent effect on the prompt's performance; second, we cluster the input space and use clustered batches so that the optimization procedure can learn the different facets of a task across batches. The resulting algorithm, UniPrompt, consists of a generative model to generate initial candidates for each prompt section; and a feedback mechanism that aggregates suggested edits from multiple mini-batches into a conceptual description for the section. Empirical evaluation on multiple datasets and a real-world task shows that prompts generated using UniPrompt obtain higher accuracy than human-tuned prompts and those from state-of-the-art methods. In particular, our algorithm can generate long, complex prompts that existing methods are unable to generate. Code for UniPrompt will be available at \url{https://aka.ms/uniprompt}. △ Less

Submitted 15 June, 2024; originally announced June 2024.

Self-Supervised Representation Learning with Spatial-Temporal Consistency for Sign Language Recognition

Authors: Weichao Zhao, Wengang Zhou, Hezhen Hu, Min Wang, Houqiang Li

Abstract: Recently, there have been efforts to improve the performance in sign language recognition by designing self-supervised learning methods. However, these methods capture limited information from sign pose data in a frame-wise learning manner, leading to sub-optimal solutions. To this end, we propose a simple yet effective self-supervised contrastive learning framework to excavate rich context via sp… ▽ More Recently, there have been efforts to improve the performance in sign language recognition by designing self-supervised learning methods. However, these methods capture limited information from sign pose data in a frame-wise learning manner, leading to sub-optimal solutions. To this end, we propose a simple yet effective self-supervised contrastive learning framework to excavate rich context via spatial-temporal consistency from two distinct perspectives and learn instance discriminative representation for sign language recognition. On one hand, since the semantics of sign language are expressed by the cooperation of fine-grained hands and coarse-grained trunks, we utilize both granularity information and encode them into latent spaces. The consistency between hand and trunk features is constrained to encourage learning consistent representation of instance samples. On the other hand, inspired by the complementary property of motion and joint modalities, we first introduce first-order motion information into sign language modeling. Additionally, we further bridge the interaction between the embedding spaces of both modalities, facilitating bidirectional knowledge transfer to enhance sign language representation. Our method is evaluated with extensive experiments on four public benchmarks, and achieves new state-of-the-art performance with a notable margin. The source code is publicly available at https://github.com/sakura/Code. △ Less

Submitted 15 June, 2024; originally announced June 2024.

Comments: Accepted by TIP2023

Bursty Star Formation in Dwarfs is Sensitive to Numerical Choices in Supernova Feedback Models

Authors: Eric Zhang, Laura V Sales, Federico Marinacci, Paul Torrey, Mark Vogelsberger, Volker Springel, Hui Li, Rüdiger Pakmor, Thales A Gutcke

Abstract: Simulations of galaxy formation are mostly unable to resolve the energy-conserving phase of individual supernova events, having to resort to subgrid models to distribute the energy and momentum resulting from stellar feedback. However, the properties of these simulated galaxies, including the morphology, stellar mass formed and the burstiness of the star formation history, are highly sensitive to… ▽ More Simulations of galaxy formation are mostly unable to resolve the energy-conserving phase of individual supernova events, having to resort to subgrid models to distribute the energy and momentum resulting from stellar feedback. However, the properties of these simulated galaxies, including the morphology, stellar mass formed and the burstiness of the star formation history, are highly sensitive to numerical choices adopted in these subgrid models. Using the {\small SMUGGLE} stellar feedback model, we compute idealized simulations of a $M_{\rm vir} \sim 10^{10} \, \msun$ dwarf galaxy, a regime where most simulation codes predict significant burstiness in star formation, resulting in strong gas flows that lead to the formation of dark matter cores. We find that by varying only the directional distribution of momentum imparted from supernovae to the surrounding gas, while holding the total momentum per supernova constant, bursty star formation may be amplified or completely suppressed, and the total stellar mass formed can vary by as much as a factor of $\sim 3$. In particular, when momentum is primarily directed perpendicular to the gas disk, less bursty and lower overall star formation rates result, yielding less gas turbulence, more disky morphologies and a retention of cuspy dark matter density profiles. An improved understanding of the non-linear coupling of stellar feedback into inhomogeneous gaseous media is thus needed to make robust predictions for stellar morphologies and dark matter core formation in dwarfs independent of uncertain numerical choices in the baryonic treatment. △ Less

Submitted 14 June, 2024; originally announced June 2024.

Comments: Submitted ApJ; 15 pages, 12 figures; comments welcome

SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages

Authors: Holy Lovenia, Rahmad Mahendra, Salsabil Maulana Akbar, Lester James V. Miranda, Jennifer Santoso, Elyanah Aco, Akhdan Fadhilah, Jonibek Mansurov, Joseph Marvin Imperial, Onno P. Kampman, Joel Ruben Antony Moniz, Muhammad Ravi Shulthan Habibi, Frederikus Hudi, Railey Montalan, Ryan Ignatius, Joanito Agili Lopo, William Nixon, Börje F. Karlsson, James Jaya, Ryandito Diandaru, Yuze Gao, Patrick Amadeus, Bin Wang, Jan Christian Blaise Cruz, Chenxi Whitehouse , et al. (36 additional authors not shown)

Abstract: Southeast Asia (SEA) is a region rich in linguistic diversity and cultural variety, with over 1,300 indigenous languages and a population of 671 million people. However, prevailing AI models suffer from a significant lack of representation of texts, images, and audio datasets from SEA, compromising the quality of AI models for SEA languages. Evaluating models for SEA languages is challenging due t… ▽ More Southeast Asia (SEA) is a region rich in linguistic diversity and cultural variety, with over 1,300 indigenous languages and a population of 671 million people. However, prevailing AI models suffer from a significant lack of representation of texts, images, and audio datasets from SEA, compromising the quality of AI models for SEA languages. Evaluating models for SEA languages is challenging due to the scarcity of high-quality datasets, compounded by the dominance of English training data, raising concerns about potential cultural misrepresentation. To address these challenges, we introduce SEACrowd, a collaborative initiative that consolidates a comprehensive resource hub that fills the resource gap by providing standardized corpora in nearly 1,000 SEA languages across three modalities. Through our SEACrowd benchmarks, we assess the quality of AI models on 36 indigenous languages across 13 tasks, offering valuable insights into the current AI landscape in SEA. Furthermore, we propose strategies to facilitate greater AI advancements, maximizing potential utility and resource equity for the future of AI in SEA. △ Less

Submitted 14 June, 2024; originally announced June 2024.

Comments: https://github.com/SEACrowd

Flight-Scope: microscopy with microfluidics in microgravity

Authors: Thomas Wareing, Alexander Stokes, Katrina Crompton, Koren Murphy, Jack Dawson, Yusuf Furkan Ugurluoglu, Connor Richardson, Hongquan Li, Manu Prakash, Adam J. M. Wollman

Abstract: With the European Space Agency (ESA) and NASA working to return humans to the moon and onwards to Mars, it has never been more important to study the impact of altered gravity conditions on biological organisms. These include astronauts but also useful micro-organisms they may bring with them to produce food, medicine, and other useful compounds by synthetic biology. Parabolic flights are one of t… ▽ More With the European Space Agency (ESA) and NASA working to return humans to the moon and onwards to Mars, it has never been more important to study the impact of altered gravity conditions on biological organisms. These include astronauts but also useful micro-organisms they may bring with them to produce food, medicine, and other useful compounds by synthetic biology. Parabolic flights are one of the most accessible microgravity research platforms but present their own challenges: relatively short periods of altered gravity (~20s) and aircraft vibration. Live-imaging is necessary in these altered-gravity conditions to readout any real-time phenotypes. Here we present Flight-Scope, a new microscopy and microfluidics platform to study dynamic cellular processes during the short, altered gravity periods on parabolic flights. We demonstrated Flight-Scopes capability by performing live and dynamic imaging of fluorescent glucose uptake by yeast, S. cerevisiae, on board an ESA parabolic flight. Flight-Scope operated well in this challenging environment, opening the way for future microgravity experiments on biological organisms. △ Less

Submitted 14 June, 2024; originally announced June 2024.

Comments: 22 pages, 5 figures

IFA: Interaction Fidelity Attention for Entire Lifelong Behaviour Sequence Modeling

Authors: Wenhui Yu, Chao Feng, Yanze Zhang, Lantao Hu, Peng Jiang, Han Li

Abstract: The lifelong user behavior sequence provides abundant information of user preference and gains impressive improvement in the recommendation task, however increases computational consumption significantly. To meet the severe latency requirement in online service, a short sub-sequence is sampled based on similarity to the target item. Unfortunately, items not in the sub-sequence are abandoned, leadi… ▽ More The lifelong user behavior sequence provides abundant information of user preference and gains impressive improvement in the recommendation task, however increases computational consumption significantly. To meet the severe latency requirement in online service, a short sub-sequence is sampled based on similarity to the target item. Unfortunately, items not in the sub-sequence are abandoned, leading to serious information loss. In this paper, we propose a new efficient paradigm to model the full lifelong sequence, which is named as \textbf{I}nteraction \textbf{F}idelity \textbf{A}ttention (\textbf{IFA}). In IFA, we input all target items in the candidate set into the model at once, and leverage linear transformer to reduce the time complexity of the cross attention between the candidate set and the sequence without any interaction information loss. We also additionally model the relationship of all target items for optimal set generation, and design loss function for better consistency of training and inference. We demonstrate the effectiveness and efficiency of our model by off-line and online experiments in the recommender system of Kuaishou. △ Less

Submitted 14 June, 2024; originally announced June 2024.

Comments: 7 pages, 2 figures

Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapters

Authors: Yuhang Zhou, Zihua Zhao, Haolin Li, Siyuan Du, Jiangchao Yao, Ya Zhang, Yanfeng Wang

Abstract: Training a unified model to take multiple targets into account is a trend towards artificial general intelligence. However, how to efficiently mitigate the training conflicts among heterogeneous data collected from different domains or tasks remains under-explored. In this study, we explore to leverage Mixture of Low-rank Adapters (MoLA) to mitigate conflicts in heterogeneous data training, which… ▽ More Training a unified model to take multiple targets into account is a trend towards artificial general intelligence. However, how to efficiently mitigate the training conflicts among heterogeneous data collected from different domains or tasks remains under-explored. In this study, we explore to leverage Mixture of Low-rank Adapters (MoLA) to mitigate conflicts in heterogeneous data training, which requires to jointly train the multiple low-rank adapters and their shared backbone. Specifically, we introduce two variants of MoLA, namely, MoLA-Grad and MoLA-Router, to respectively handle the target-aware and target-agnostic scenarios during inference. The former uses task identifiers to assign personalized low-rank adapters to each task, disentangling task-specific knowledge towards their adapters, thereby mitigating heterogeneity conflicts. The latter uses a novel Task-wise Decorrelation (TwD) loss to intervene the router to learn oriented weight combinations of adapters to homogeneous tasks, achieving similar effects. We conduct comprehensive experiments to verify the superiority of MoLA over previous state-of-the-art methods and present in-depth analysis on its working mechanism. Source code is available at: https://github.com/MediaBrain-SJTU/MoLA △ Less

Submitted 13 June, 2024; originally announced June 2024.

Comments: ICML2024

Search for $X(1870)$ via the decay $J/ψ\to ωK^+ K^-η$

Authors: BESIII Collaboration, M. Ablikim, M. N. Achasov, P. Adlarson, O. Afedulidis, X. C. Ai, R. Aliberti, A. Amoroso, Q. An, Y. Bai, O. Bakina, I. Balossino, Y. Ban, H. -R. Bao, V. Batozskaya, K. Begzsuren, N. Berger, M. Berlowski, M. Bertani, D. Bettoni, F. Bianchi, E. Bianco, A. Bortone, I. Boyko, R. A. Briere , et al. (644 additional authors not shown)

Abstract: Using a sample of $(10087\pm 44)\times10^{6}$ $J/ψ$ events collected by the BESIII detector at the BEPCII collider, we search for the decay $X(1870)\to K^+ K^-η$ via the $J/ψ\to ωK^+ K^- η$ process for the first time. No significant $X(1870)$ signal is observed. The upper limit on the branching fraction of the decay $ J/ψ\to ωX(1870) \toωK^+ K^- η$ is determined to be $9.55\times 10^{-7}$ at the… ▽ More Using a sample of $(10087\pm 44)\times10^{6}$ $J/ψ$ events collected by the BESIII detector at the BEPCII collider, we search for the decay $X(1870)\to K^+ K^-η$ via the $J/ψ\to ωK^+ K^- η$ process for the first time. No significant $X(1870)$ signal is observed. The upper limit on the branching fraction of the decay $ J/ψ\to ωX(1870) \toωK^+ K^- η$ is determined to be $9.55\times 10^{-7}$ at the $90\%$ confidence level. In addition, the branching faction $B(J/ψ\toωK^+ K^- η)$ is measured to be $(3.33\pm0.02(\rm{stat.})\pm 0.12(\rm{syst.}))\times 10^{-4}$. △ Less

Submitted 13 June, 2024; originally announced June 2024.

Explore the Limits of Omni-modal Pretraining at Scale

Authors: Yiyuan Zhang, Handong Li, **g Liu, Xiangyu Yue

Abstract: We propose to build omni-modal intelligence, which is capable of understanding any modality and learning universal representations. In specific, we propose a scalable pretraining paradigm, named Multimodal Context (MiCo), which can scale up the numbers of modalities and amount of data, together with the model parameters, in the pretraining process. With MiCo, the pretrained models show significant… ▽ More We propose to build omni-modal intelligence, which is capable of understanding any modality and learning universal representations. In specific, we propose a scalable pretraining paradigm, named Multimodal Context (MiCo), which can scale up the numbers of modalities and amount of data, together with the model parameters, in the pretraining process. With MiCo, the pretrained models show significant emergent abilities in multimodal learning, which are evaluated on the following tasks: i) single-modality perception benchmarks of 10 different modalities, ii) 25 cross-modality understanding tasks of retrieval, question-answering, captioning, and iii) 18 multimodal large language model benchmarks. Our models establish 37 new records for state-of-the-art performance. We hope that our research could contribute to the development of omni-modal intelligence. Code and Models are at https://github.com/invictus717/MiCo △ Less

Submitted 13 June, 2024; originally announced June 2024.

Comments: Project Website: https://invictus717.github.io/MiCo/

Scene Graph Generation in Large-Size VHR Satellite Imagery: A Large-Scale Dataset and A Context-Aware Approach

Authors: Yansheng Li, Linlin Wang, Tingzhu Wang, Xue Yang, Junwei Luo, Qi Wang, Youming Deng, Wenbin Wang, Xian Sun, Haifeng Li, Bo Dang, Yongjun Zhang, Yi Yu, Junchi Yan

Abstract: Scene graph generation (SGG) in satellite imagery (SAI) benefits promoting intelligent understanding of geospatial scenarios from perception to cognition. In SAI, objects exhibit great variations in scales and aspect ratios, and there exist rich relationships between objects (even between spatially disjoint objects), which makes it necessary to holistically conduct SGG in large-size very-high-reso… ▽ More Scene graph generation (SGG) in satellite imagery (SAI) benefits promoting intelligent understanding of geospatial scenarios from perception to cognition. In SAI, objects exhibit great variations in scales and aspect ratios, and there exist rich relationships between objects (even between spatially disjoint objects), which makes it necessary to holistically conduct SGG in large-size very-high-resolution (VHR) SAI. However, the lack of SGG datasets with large-size VHR SAI has constrained the advancement of SGG in SAI. Due to the complexity of large-size VHR SAI, mining triplets <subject, relationship, object> in large-size VHR SAI heavily relies on long-range contextual reasoning. Consequently, SGG models designed for small-size natural imagery are not directly applicable to large-size VHR SAI. To address the scarcity of datasets, this paper constructs a large-scale dataset for SGG in large-size VHR SAI with image sizes ranging from 512 x 768 to 27,860 x 31,096 pixels, named RSG, encompassing over 210,000 objects and more than 400,000 triplets. To realize SGG in large-size VHR SAI, we propose a context-aware cascade cognition (CAC) framework to understand SAI at three levels: object detection (OBD), pair pruning and relationship prediction. As a fundamental prerequisite for SGG in large-size SAI, a holistic multi-class object detection network (HOD-Net) that can flexibly integrate multi-scale contexts is proposed. With the consideration that there exist a huge amount of object pairs in large-size SAI but only a minority of object pairs contain meaningful relationships, we design a pair proposal generation (PPG) network via adversarial reconstruction to select high-value pairs. Furthermore, a relationship prediction network with context-aware messaging (RPCM) is proposed to predict the relationship types of these pairs. △ Less

Submitted 30 June, 2024; v1 submitted 13 June, 2024; originally announced June 2024.

Comments: This paper releases a SAI-oriented SGG toolkit with about 30 OBD methods and 10 SGG methods, and develops a benchmark based on RSG where our HOD-Net and RPCM significantly outperform the state-of-the-art methods in both OBD and SGG tasks. The RSG dataset and SAI-oriented toolkit will be made publicly available at https://linlin-dev.github.io/project/RSG

Enhanced Object Detection: A Study on Vast Vocabulary Object Detection Track for V3Det Challenge 2024

Authors: Peixi Wu, Bosong Chai, Xuan Nie, Longquan Yan, Zeyu Wang, Qifan Zhou, Boning Wang, Yansong Peng, Hebei Li

Abstract: In this technical report, we present our findings from the research conducted on the Vast Vocabulary Visual Detection (V3Det) dataset for Supervised Vast Vocabulary Visual Detection task. How to deal with complex categories and detection boxes has become a difficulty in this track. The original supervised detector is not suitable for this task. We have designed a series of improvements, including… ▽ More In this technical report, we present our findings from the research conducted on the Vast Vocabulary Visual Detection (V3Det) dataset for Supervised Vast Vocabulary Visual Detection task. How to deal with complex categories and detection boxes has become a difficulty in this track. The original supervised detector is not suitable for this task. We have designed a series of improvements, including adjustments to the network structure, changes to the loss function, and design of training strategies. Our model has shown improvement over the baseline and achieved excellent rankings on the Leaderboard for both the Vast Vocabulary Object Detection (Supervised) track and the Open Vocabulary Object Detection (OVD) track of the V3Det Challenge 2024. △ Less

Submitted 21 June, 2024; v1 submitted 13 June, 2024; originally announced June 2024.

Journal ref: Second Place in CVPR 2024 Vast Vocabulary Visual Detection Challenge

Dissipative Superfluidity in a Molecular Bose-Einstein Condensate

Authors: Hongchao Li, Xie-Hang Yu, Masaya Nakagawa, Masahito Ueda

Abstract: Motivated by recent experimental realization of a Bose-Einstein condensate (BEC) of dipolar molecules, we develop superfluid transport theory for a dissipative BEC to show that a weak uniform two-body loss can induce phase rigidity, leading to superfluid transport of bosons. A generalized f-sum rule is shown to hold for a dissipative superfluid as a consequence of weak U(1) symmetry. It is also de… ▽ More Motivated by recent experimental realization of a Bose-Einstein condensate (BEC) of dipolar molecules, we develop superfluid transport theory for a dissipative BEC to show that a weak uniform two-body loss can induce phase rigidity, leading to superfluid transport of bosons. A generalized f-sum rule is shown to hold for a dissipative superfluid as a consequence of weak U(1) symmetry. It is also demonstrated that dissipation enhances the stability of a molecular BEC with dipolar interactions. Possible experimental situations for measuring the superfluid fraction and the spectral function are discussed. △ Less

Submitted 13 June, 2024; originally announced June 2024.

Comments: 6 pages, 1 figure+27 pages 2 figures

Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation

Authors: Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, **gdong Wang, Yao Yao, Siyu Zhu

Abstract: The field of portrait image animation, driven by speech audio input, has experienced significant advancements in the generation of realistic and dynamic portraits. This research delves into the complexities of synchronizing facial movements and creating visually appealing, temporally consistent animations within the framework of diffusion-based methodologies. Moving away from traditional paradigms… ▽ More The field of portrait image animation, driven by speech audio input, has experienced significant advancements in the generation of realistic and dynamic portraits. This research delves into the complexities of synchronizing facial movements and creating visually appealing, temporally consistent animations within the framework of diffusion-based methodologies. Moving away from traditional paradigms that rely on parametric models for intermediate facial representations, our innovative approach embraces the end-to-end diffusion paradigm and introduces a hierarchical audio-driven visual synthesis module to enhance the precision of alignment between audio inputs and visual outputs, encompassing lip, expression, and pose motion. Our proposed network architecture seamlessly integrates diffusion-based generative models, a UNet-based denoiser, temporal alignment techniques, and a reference network. The proposed hierarchical audio-driven visual synthesis offers adaptive control over expression and pose diversity, enabling more effective personalization tailored to different identities. Through a comprehensive evaluation that incorporates both qualitative and quantitative analyses, our approach demonstrates obvious enhancements in image and video quality, lip synchronization precision, and motion diversity. Further visualization and access to the source code can be found at: https://fudan-generative-vision.github.io/hallo. △ Less

Submitted 16 June, 2024; v1 submitted 13 June, 2024; originally announced June 2024.

Comments: 20 pages

Solving Fractional Differential Equations on a Quantum Computer: A Variational Approach

Authors: Fong Yew Leong, Dax Enshan Koh, Jian Feng Kong, Siong Thye Goh, Jun Yong Khoo, Wei-Bin Ewe, Hongying Li, Jayne Thompson, Dario Poletti

Abstract: We introduce an efficient variational hybrid quantum-classical algorithm designed for solving Caputo time-fractional partial differential equations. Our method employs an iterable cost function incorporating a linear combination of overlap history states. The proposed algorithm is not only efficient in time complexity, but has lower memory costs compared to classical methods. Our results indicate… ▽ More We introduce an efficient variational hybrid quantum-classical algorithm designed for solving Caputo time-fractional partial differential equations. Our method employs an iterable cost function incorporating a linear combination of overlap history states. The proposed algorithm is not only efficient in time complexity, but has lower memory costs compared to classical methods. Our results indicate that solution fidelity is insensitive to the fractional index and that gradient evaluation cost scales economically with the number of time steps. As a proof of concept, we apply our algorithm to solve a range of fractional partial differential equations commonly encountered in engineering applications, such as the sub-diffusion equation, the non-linear Burgers' equation and a coupled diffusive epidemic model. We assess quantum hardware performance under realistic noise conditions, further validating the practical utility of our algorithm. △ Less

Submitted 12 June, 2024; originally announced June 2024.

Constraints on Ultra Heavy Dark Matter Properties from Dwarf Spheroidal Galaxies with LHAASO Observations

Authors: Zhen Cao, F. Aharonian, Q. An, Axikegu, Y. X. Bai, Y. W. Bao, D. Bastieri, X. J. Bi, Y. J. Bi, J. T. Cai, Q. Cao, W. Y. Cao, Zhe Cao, J. Chang, J. F. Chang, A. M. Chen, E. S. Chen, Liang Chen, Lin Chen, Long Chen, M. J. Chen, M. L. Chen, Q. H. Chen, S. H. Chen, S. Z. Chen , et al. (255 additional authors not shown)

Abstract: In this work we try to search for signals generated by ultra-heavy dark matter at the Large High Altitude Air Shower Observatory (LHAASO) data. We look for possible gamma-ray by dark matter annihilation or decay from 16 dwarf spheroidal galaxies in the field of view of LHAASO. Dwarf spheroidal galaxies are among the most promising targets for indirect detection of dark matter which have low fluxes… ▽ More In this work we try to search for signals generated by ultra-heavy dark matter at the Large High Altitude Air Shower Observatory (LHAASO) data. We look for possible gamma-ray by dark matter annihilation or decay from 16 dwarf spheroidal galaxies in the field of view of LHAASO. Dwarf spheroidal galaxies are among the most promising targets for indirect detection of dark matter which have low fluxes of astrophysical $γ$-ray background while large amount of dark matter. By analyzing more than 700 days observational data at LHAASO, no significant dark matter signal from 1 TeV to 1 EeV is detected. Accordingly we derive the most stringent constraints on the ultra-heavy dark matter annihilation cross-section up to EeV. The constraints on the lifetime of dark matter in decay mode are also derived. △ Less

Submitted 12 June, 2024; originally announced June 2024.

Comments: 17 pages, 12 figures, accepted by PRL

Refining Self-Supervised Learnt Speech Representation using Brain Activations

Authors: Hengyu Li, Kangdi Mei, Zhaoci Liu, Yang Ai, Li** Chen, Jie Zhang, Zhenhua Ling

Abstract: It was shown in literature that speech representations extracted by self-supervised pre-trained models exhibit similarities with brain activations of human for speech perception and fine-tuning speech representation models on downstream tasks can further improve the similarity. However, it still remains unclear if this similarity can be used to optimize the pre-trained speech models. In this work,… ▽ More It was shown in literature that speech representations extracted by self-supervised pre-trained models exhibit similarities with brain activations of human for speech perception and fine-tuning speech representation models on downstream tasks can further improve the similarity. However, it still remains unclear if this similarity can be used to optimize the pre-trained speech models. In this work, we therefore propose to use the brain activations recorded by fMRI to refine the often-used wav2vec2.0 model by aligning model representations toward human neural responses. Experimental results on SUPERB reveal that this operation is beneficial for several downstream tasks, e.g., speaker verification, automatic speech recognition, intent classification.One can then consider the proposed method as a new alternative to improve self-supervised speech models. △ Less

Submitted 13 June, 2024; v1 submitted 12 June, 2024; originally announced June 2024.

Comments: accpeted by Interspeech2024

M3T: A New Benchmark Dataset for Multi-Modal Document-Level Machine Translation

Authors: Benjamin Hsu, Xiaoyu Liu, Huayang Li, Yoshinari Fu**uma, Maria Nadejde, Xing Niu, Yair Kittenplon, Ron Litman, Raghavendra Pappagari

Abstract: Document translation poses a challenge for Neural Machine Translation (NMT) systems. Most document-level NMT systems rely on meticulously curated sentence-level parallel data, assuming flawless extraction of text from documents along with their precise reading order. These systems also tend to disregard additional visual cues such as the document layout, deeming it irrelevant. However, real-world… ▽ More Document translation poses a challenge for Neural Machine Translation (NMT) systems. Most document-level NMT systems rely on meticulously curated sentence-level parallel data, assuming flawless extraction of text from documents along with their precise reading order. These systems also tend to disregard additional visual cues such as the document layout, deeming it irrelevant. However, real-world documents often possess intricate text layouts that defy these assumptions. Extracting information from Optical Character Recognition (OCR) or heuristic rules can result in errors, and the layout (e.g., paragraphs, headers) may convey relationships between distant sections of text. This complexity is particularly evident in widely used PDF documents, which represent information visually. This paper addresses this gap by introducing M3T, a novel benchmark dataset tailored to evaluate NMT systems on the comprehensive task of translating semi-structured documents. This dataset aims to bridge the evaluation gap in document-level NMT systems, acknowledging the challenges posed by rich text layouts in real-world applications. △ Less

Submitted 12 June, 2024; originally announced June 2024.

Comments: NAACL 2024, dataset at https://github.com/amazon-science/m3t-multi-modal-translation-bench

Observation of $η_{c}$(1S, 2S) and $χ_{cJ}$ decays to 2$(π^{+}π^{-})η$ via $ψ$(3686) radiative transitions

Authors: BESIII Collaboration, M. Ablikim, M. N. Achasov, P. Adlarson, O. Afedulidis, X. C. Ai, R. Aliberti, A. Amoroso, Q. An, Y. Bai, O. Bakina, I. Balossino, Y. Ban, H. -R. Bao, V. Batozskaya, K. Begzsuren, N. Berger, M. Berlowski, M. Bertani, D. Bettoni, F. Bianchi, E. Bianco, A. Bortone, I. Boyko, R. A. Briere , et al. (636 additional authors not shown)

Abstract: Based on $2.7 \times 10^9~ψ(3686)$ decays collected with the BESIII detector, the radiative decay $ψ(3686)\to\gamma2(π^{+}π^{-})η$ is investigated to measure properties of S- and P-wave charmonium states. The branching fraction of the decay $η_{c}(1S) \to 2(π^{+}π^{-})η$, which is found to have a strong dependence on the interference pattern between $η_c(1S)$ and non-$η_c(1S)$ processes, is measur… ▽ More Based on $2.7 \times 10^9~ψ(3686)$ decays collected with the BESIII detector, the radiative decay $ψ(3686)\to\gamma2(π^{+}π^{-})η$ is investigated to measure properties of S- and P-wave charmonium states. The branching fraction of the decay $η_{c}(1S) \to 2(π^{+}π^{-})η$, which is found to have a strong dependence on the interference pattern between $η_c(1S)$ and non-$η_c(1S)$ processes, is measured in both destructive and constructive interference scenarios for the first time. The mass and width of the $η_{c}(1S)$ are measured to be $M=(2984.14 \pm 0.13 \pm 0.38)$ MeV/$c^{2}$ and $Γ=(28.82 \pm 0.11 \pm 0.82)$ MeV, respectively. Clear signals for the decays of the $χ_{cJ}(J=0,1,2)$ and the $η_{c}(2S)$ to $2(π^{+}π^{-})η$ are also observed for the first time, and the corresponding branching fractions are measured. The ratio of the branching fractions between the $η_{c}(2S)$ and $η_{c}(1S)$ decays is significantly lower than the theoretical prediction, which might suggest different dynamics in their decays. △ Less

Submitted 12 June, 2024; originally announced June 2024.

FreeV: Free Lunch For Vocoders Through Pseudo Inversed Mel Filter

Authors: Yuanjun Lv, Hai Li, Ying Yan, Junhui Liu, Danming Xie, Lei Xie

Abstract: Vocoders reconstruct speech waveforms from acoustic features and play a pivotal role in modern TTS systems. Frequent-domain GAN vocoders like Vocos and APNet2 have recently seen rapid advancements, outperforming time-domain models in inference speed while achieving comparable audio quality. However, these frequency-domain vocoders suffer from large parameter sizes, thus introducing extra memory bu… ▽ More Vocoders reconstruct speech waveforms from acoustic features and play a pivotal role in modern TTS systems. Frequent-domain GAN vocoders like Vocos and APNet2 have recently seen rapid advancements, outperforming time-domain models in inference speed while achieving comparable audio quality. However, these frequency-domain vocoders suffer from large parameter sizes, thus introducing extra memory burden. Inspired by PriorGrad and SpecGrad, we employ pseudo-inverse to estimate the amplitude spectrum as the initialization roughly. This simple initialization significantly mitigates the parameter demand for vocoder. Based on APNet2 and our streamlined Amplitude prediction branch, we propose our FreeV, compared with its counterpart APNet2, our FreeV achieves 1.8 times inference speed improvement with nearly half parameters. Meanwhile, our FreeV outperforms APNet2 in resynthesis quality, marking a step forward in pursuing real-time, high-fidelity speech synthesis. Code and checkpoints is available at: https://github.com/BakerBunker/FreeV △ Less

Submitted 12 June, 2024; originally announced June 2024.

Comments: Accepted by InterSpeech 2024; 5 pages, 5 figures

Semi-Supervised Spoken Language Glossification

Authors: Huijie Yao, Wengang Zhou, Hao Zhou, Houqiang Li

Abstract: Spoken language glossification (SLG) aims to translate the spoken language text into the sign language gloss, i.e., a written record of sign language. In this work, we present a framework named $S$emi-$S$upervised $S$poken $L$anguage $G$lossification ($S^3$LG) for SLG. To tackle the bottleneck of limited parallel data in SLG, our $S^3$LG incorporates large-scale monolingual spoken language text in… ▽ More Spoken language glossification (SLG) aims to translate the spoken language text into the sign language gloss, i.e., a written record of sign language. In this work, we present a framework named $S$emi-$S$upervised $S$poken $L$anguage $G$lossification ($S^3$LG) for SLG. To tackle the bottleneck of limited parallel data in SLG, our $S^3$LG incorporates large-scale monolingual spoken language text into SLG training. The proposed framework follows the self-training structure that iteratively annotates and learns from pseudo labels. Considering the lexical similarity and syntactic difference between sign language and spoken language, our $S^3$LG adopts both the rule-based heuristic and model-based approach for auto-annotation. During training, we randomly mix these complementary synthetic datasets and mark their differences with a special token. As the synthetic data may be less quality, the $S^3$LG further leverages consistency regularization to reduce the negative impact of noise in the synthetic data. Extensive experiments are conducted on public benchmarks to demonstrate the effectiveness of the $S^3$LG. Our code is available at \url{https://github.com/yaohj11/S3LG}. △ Less

Submitted 12 June, 2024; originally announced June 2024.

Comments: Accepted to ACL2024 main

Unveiling potential neutron halos in intermediate-mass nuclei: an \textit{ab initio} study

Authors: H. H. Li, J. G. Li, M. R. Xie, W. Zuo

Abstract: Halos epitomize the fascinating interplay between weak binding, shell evolution, and deformation effects, especially in nuclei near the drip line. In this Letter, we apply the state-of-the-art \textit{ab initio} valence-space in-medium similarity renormalization group approach to predict potential candidates for one- and two-neutron halo in the intermediate-mass region. Notably, we use spectroscop… ▽ More Halos epitomize the fascinating interplay between weak binding, shell evolution, and deformation effects, especially in nuclei near the drip line. In this Letter, we apply the state-of-the-art \textit{ab initio} valence-space in-medium similarity renormalization group approach to predict potential candidates for one- and two-neutron halo in the intermediate-mass region. Notably, we use spectroscopic factors (SF) and two-nucleon amplitudes (TNA) as criteria for suggesting one- and two-neutron halo candidates, respectively. This approach is not only theoretically sound but also amenable to experimental validation. Our research focuses on Mg, Al, Si, P, and S neutron-drip-line nuclei, offering systematic predictions of neutron halo candidates in terms of separation energies, SF (TNA), and average occupation. The calculation suggests the ground states of $^{40,42,44,46}$Al, $^{41,43,45,47}$Si, $^{46,48}$P, and $^{47,49}$S are promising candidates for one-neutron halos, while $^{40,42,44,46}$Mg, $^{45,47}$Al, $^{46,48}$Si, $^{49}$P, and $^{50}$S may harbor two-neutron halos. In addition, the relative mean-square neutron radius between halo nuclei and \textit{inner core} is calculated for suggested potential neutron halos. Finally, the relations of halo formations and shell evolution are discussed. △ Less

Submitted 12 June, 2024; originally announced June 2024.

Real-time, chirped-pulse heterodyne detection at room-temperature with 100GHz 3dB-bandwidth mid-infrared quantum-well photodetectors

Authors: Quyang Lin, Michael Hakl, Sylvie Lepillet, Hua Li, Jean-Francois Lampin, Emilien Peytavit, Stefano Barbieri

Abstract: Thanks to intrinsically short electronic relaxation on the ps time scale, III-V semiconductor unipolar devices are ideal candidates for ultrahigh-speed operation at mid-infrared frequencies. In this work, antenna-coupled, GaAs-based multi quantum-well photodetectors operating in the 10-11um range are demonstrated, with a responsivity of 0.3A/W and a 3dB-cutoff bandwidth of 100GHz at room-temperatu… ▽ More Thanks to intrinsically short electronic relaxation on the ps time scale, III-V semiconductor unipolar devices are ideal candidates for ultrahigh-speed operation at mid-infrared frequencies. In this work, antenna-coupled, GaAs-based multi quantum-well photodetectors operating in the 10-11um range are demonstrated, with a responsivity of 0.3A/W and a 3dB-cutoff bandwidth of 100GHz at room-temperature. The frequency response is measured up to 220GHz: beyond 100GHz we find a roll-off dominated by the 2.5 ps-long recombination time of the photo-excited electrons. The potential of the detectors is illustrated by setting up an experiment where the time dependent emission frequency of a quantum cascade laser operated in pulsed mode is measured electronically and in real-time, over a frequency range >60GHz. By exploiting broadband electronics, and thanks to its high signal-to-noise ratio, this technique allows the acquisition, in a single-shot, of frequency-calibrated, mid-infrared molecular spectra spanning up to 100GHz and beyond, which is particularly attractive for fast, active remote sensing applications in fields such as environmental or combustion monitoring. △ Less

Submitted 12 June, 2024; originally announced June 2024.

Holographic Superfluid Ring with a Weak Link

Authors: Zhi-Hong li, Huai-Fan Li

Abstract: We explore the generation of topological defects in the course of a dynamical phase transition in a ring with a weak link, i.e., a SSS Josephson junction, from the AdS/CFT correspondence. By setting different parameters of the junction (width, steepness, depth) and the final temperature of the quench, the configurations of the charge density and condensate of the order parameters of the dual field… ▽ More We explore the generation of topological defects in the course of a dynamical phase transition in a ring with a weak link, i.e., a SSS Josephson junction, from the AdS/CFT correspondence. By setting different parameters of the junction (width, steepness, depth) and the final temperature of the quench, the configurations of the charge density and condensate of the order parameters of the dual field theory are presented. Meanwhile, we observe that in the final equilibrium state, variations in parameters of the junctions only affect the configurations of the charge density and condensate of the order parameters, without altering their values outside the junction. However, variations in the final temperature will directly affect the values of the charge density and condensate of the order parameters outside of the junction. Moreover, in the final equilibrium state, we propose an analytic relation between the gauge-invariant velocity in the two superconducting states in the SSS Josephson junction, which agrees well with the numerical results. △ Less

Submitted 12 June, 2024; originally announced June 2024.

Comments: 16 pages, 8 figures

A3VLM: Actionable Articulation-Aware Vision Language Model

Authors: Siyuan Huang, Haonan Chang, Yuhan Liu, Yimeng Zhu, Hao Dong, Peng Gao, Abdeslam Boularias, Hongsheng Li

Abstract: Vision Language Models (VLMs) have received significant attention in recent years in the robotics community. VLMs are shown to be able to perform complex visual reasoning and scene understanding tasks, which makes them regarded as a potential universal solution for general robotics problems such as manipulation and navigation. However, previous VLMs for robotics such as RT-1, RT-2, and ManipLLM ha… ▽ More Vision Language Models (VLMs) have received significant attention in recent years in the robotics community. VLMs are shown to be able to perform complex visual reasoning and scene understanding tasks, which makes them regarded as a potential universal solution for general robotics problems such as manipulation and navigation. However, previous VLMs for robotics such as RT-1, RT-2, and ManipLLM have focused on directly learning robot-centric actions. Such approaches require collecting a significant amount of robot interaction data, which is extremely costly in the real world. Thus, we propose A3VLM, an object-centric, actionable, articulation-aware vision language model. A3VLM focuses on the articulation structure and action affordances of objects. Its representation is robot-agnostic and can be translated into robot actions using simple action primitives. Extensive experiments in both simulation benchmarks and real-world settings demonstrate the effectiveness and stability of A3VLM. We release our code and other materials at https://github.com/changhaonan/A3VLM. △ Less

Submitted 13 June, 2024; v1 submitted 11 June, 2024; originally announced June 2024.

Trim 3D Gaussian Splatting for Accurate Geometry Representation

Authors: Lue Fan, Yuxue Yang, Minxing Li, Hongsheng Li, Zhaoxiang Zhang

Abstract: In this paper, we introduce Trim 3D Gaussian Splatting (TrimGS) to reconstruct accurate 3D geometry from images. Previous arts for geometry reconstruction from 3D Gaussians mainly focus on exploring strong geometry regularization. Instead, from a fresh perspective, we propose to obtain accurate 3D geometry of a scene by Gaussian trimming, which selectively removes the inaccurate geometry while pre… ▽ More In this paper, we introduce Trim 3D Gaussian Splatting (TrimGS) to reconstruct accurate 3D geometry from images. Previous arts for geometry reconstruction from 3D Gaussians mainly focus on exploring strong geometry regularization. Instead, from a fresh perspective, we propose to obtain accurate 3D geometry of a scene by Gaussian trimming, which selectively removes the inaccurate geometry while preserving accurate structures. To achieve this, we analyze the contributions of individual 3D Gaussians and propose a contribution-based trimming strategy to remove the redundant or inaccurate Gaussians. Furthermore, our experimental and theoretical analyses reveal that a relatively small Gaussian scale is a non-negligible factor in representing and optimizing the intricate details. Therefore the proposed TrimGS maintains relatively small Gaussian scales. In addition, TrimGS is also compatible with the effective geometry regularization strategies in previous arts. When combined with the original 3DGS and the state-of-the-art 2DGS, TrimGS consistently yields more accurate geometry and higher perceptual quality. Our project page is https://trimgs.github.io △ Less

Submitted 11 June, 2024; originally announced June 2024.

Comments: Project page: https://trimgs.github.io/

Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation

Authors: Hanzhao Li, Liumeng Xue, Haohan Guo, Xinfa Zhu, Yuanjun Lv, Lei Xie, Yunlin Chen, Hao Yin, Zhifei Li

Abstract: The multi-codebook speech codec enables the application of large language models (LLM) in TTS but bottlenecks efficiency and robustness due to multi-sequence prediction. To avoid this obstacle, we propose Single-Codec, a single-codebook single-sequence codec, which employs a disentangled VQ-VAE to decouple speech into a time-invariant embedding and a phonetically-rich discrete sequence. Furthermor… ▽ More The multi-codebook speech codec enables the application of large language models (LLM) in TTS but bottlenecks efficiency and robustness due to multi-sequence prediction. To avoid this obstacle, we propose Single-Codec, a single-codebook single-sequence codec, which employs a disentangled VQ-VAE to decouple speech into a time-invariant embedding and a phonetically-rich discrete sequence. Furthermore, the encoder is enhanced with 1) contextual modeling with a BLSTM module to exploit the temporal information, 2) a hybrid sampling module to alleviate distortion from upsampling and downsampling, and 3) a resampling module to encourage discrete units to carry more phonetic information. Compared with multi-codebook codecs, e.g., EnCodec and TiCodec, Single-Codec demonstrates higher reconstruction quality with a lower bandwidth of only 304bps. The effectiveness of Single-Code is further validated by LLM-TTS experiments, showing improved naturalness and intelligibility. △ Less

Submitted 11 June, 2024; originally announced June 2024.

Comments: Accepted by Interspeech 2024

AI.vs.Clinician: Unveiling Intricate Interactions Between AI and Clinicians through an Open-Access Database

Authors: Wanling Gao, Yuan Liu, Zhuoming Yu, Dandan Cui, Wen**g Liu, Xiaoshuang Liang, Jiahui Zhao, Jiyue Xie, Hao Li, Li Ma, Ning Ye, Yumiao Kang, Dingfeng Luo, Peng Pan, Wei Huang, Zhongmou Liu, Jizhong Hu, Fan Huang, Gangyuan Zhao, Chongrong Jiang, Tianyi Wei, Zhifei Zhang, Yunyou Huang, Jianfeng Zhan

Abstract: Artificial Intelligence (AI) plays a crucial role in medical field and has the potential to revolutionize healthcare practices. However, the success of AI models and their impacts hinge on the synergy between AI and medical specialists, with clinicians assuming a dominant role. Unfortunately, the intricate dynamics and interactions between AI and clinicians remain undiscovered and thus hinder AI f… ▽ More Artificial Intelligence (AI) plays a crucial role in medical field and has the potential to revolutionize healthcare practices. However, the success of AI models and their impacts hinge on the synergy between AI and medical specialists, with clinicians assuming a dominant role. Unfortunately, the intricate dynamics and interactions between AI and clinicians remain undiscovered and thus hinder AI from being translated into medical practice. To address this gap, we have curated a groundbreaking database called AI.vs.Clinician. This database is the first of its kind for studying the interactions between AI and clinicians. It derives from 7,500 collaborative diagnosis records on a life-threatening medical emergency -- Sepsis -- from 14 medical centers across China. For the patient cohorts well-chosen from MIMIC databases, the AI-related information comprises the model property, feature input, diagnosis decision, and inferred probabilities of sepsis onset presently and within next three hours. The clinician-related information includes the viewed examination data and sequence, viewed time, preliminary and final diagnosis decisions with or without AI assistance, and recommended treatment. △ Less

Submitted 15 June, 2024; v1 submitted 11 June, 2024; originally announced June 2024.

Comments: 12 pages

Analytical Delta-V Approximation for Nonlinear Programming of Multi-target Rendezvous and Flyby Trajectories

Authors: An-Yi Huang, Heng-Nian Li, Ya-Zhong Luo

Abstract: This study proposes an analytical Delta-V approximation of short-time transfers based on the linear relative motion and a gradient-based nonlinear programming model of multi-target rendezvous and flyby trajectories. In previous studies, the Lambert's solution is commonly used to evaluate Delta-V of short-duration transfers. In this study, to avoid the iteration process for obtaining the Lambert's… ▽ More This study proposes an analytical Delta-V approximation of short-time transfers based on the linear relative motion and a gradient-based nonlinear programming model of multi-target rendezvous and flyby trajectories. In previous studies, the Lambert's solution is commonly used to evaluate Delta-V of short-duration transfers. In this study, to avoid the iteration process for obtaining the Lambert's solution and its gradient, the linear relative motion equations are applied to form an analytical two-point boundary value model for the near-circular orbit rendezvous problems. Although the relative motion equations are usually applicable when the two orbits are close enough, and the position and velocity errors would become more significant as the orbital differences increase, the errors of the velocity increments were proved acceptable in our simulations. Moreover, the analytical formula facilitates the calculation of the gradients to the start epoch and flight time, which are used to establish a nonlinear programming model for sequence optimization that gradient-based algorithms can easily solve. Simulation results demonstrated that the analytical Delta-V approximation requires much less calculation than the Lambert's solution, and the proposed gradient-based nonlinear programming algorithms can obtain similar results in less time than previous methods. △ Less

Submitted 11 June, 2024; originally announced June 2024.

Advancing Grounded Multimodal Named Entity Recognition via LLM-Based Reformulation and Box-Based Segmentation

Authors: **yuan Li, Ziyan Li, Han Li, Jianfei Yu, Rui Xia, Di Sun, Gang Pan

Abstract: Grounded Multimodal Named Entity Recognition (GMNER) task aims to identify named entities, entity types and their corresponding visual regions. GMNER task exhibits two challenging attributes: 1) The tenuous correlation between images and text on social media contributes to a notable proportion of named entities being ungroundable. 2) There exists a distinction between coarse-grained noun phrases u… ▽ More Grounded Multimodal Named Entity Recognition (GMNER) task aims to identify named entities, entity types and their corresponding visual regions. GMNER task exhibits two challenging attributes: 1) The tenuous correlation between images and text on social media contributes to a notable proportion of named entities being ungroundable. 2) There exists a distinction between coarse-grained noun phrases used in similar tasks (e.g., phrase localization) and fine-grained named entities. In this paper, we propose RiVEG, a unified framework that reformulates GMNER into a joint MNER-VE-VG task by leveraging large language models (LLMs) as connecting bridges. This reformulation brings two benefits: 1) It enables us to optimize the MNER module for optimal MNER performance and eliminates the need to pre-extract region features using object detection methods, thus naturally addressing the two major limitations of existing GMNER methods. 2) The introduction of Entity Expansion Expression module and Visual Entailment (VE) module unifies Visual Grounding (VG) and Entity Grounding (EG). This endows the proposed framework with unlimited data and model scalability. Furthermore, to address the potential ambiguity stemming from the coarse-grained bounding box output in GMNER, we further construct the new Segmented Multimodal Named Entity Recognition (SMNER) task and corresponding Twitter-SMNER dataset aimed at generating fine-grained segmentation masks, and experimentally demonstrate the feasibility and effectiveness of using box prompt-based Segment Anything Model (SAM) to empower any GMNER model with the ability to accomplish the SMNER task. Extensive experiments demonstrate that RiVEG significantly outperforms SoTA methods on four datasets across the MNER, GMNER, and SMNER tasks. △ Less

Submitted 11 June, 2024; originally announced June 2024.

Comments: Extension of our Findings of EMNLP 2023 & ACL 2024 paper

Target Speech Diarization with Multimodal Prompts

Authors: Yidi Jiang, Ruijie Tao, Zhengyang Chen, Yanmin Qian, Haizhou Li

Abstract: Traditional speaker diarization seeks to detect ``who spoke when'' according to speaker characteristics. Extending to target speech diarization, we detect ``when target event occurs'' according to the semantic characteristics of speech. We propose a novel Multimodal Target Speech Diarization (MM-TSD) framework, which accommodates diverse and multi-modal prompts to specify target events in a flexib… ▽ More Traditional speaker diarization seeks to detect ``who spoke when'' according to speaker characteristics. Extending to target speech diarization, we detect ``when target event occurs'' according to the semantic characteristics of speech. We propose a novel Multimodal Target Speech Diarization (MM-TSD) framework, which accommodates diverse and multi-modal prompts to specify target events in a flexible and user-friendly manner, including semantic language description, pre-enrolled speech, pre-registered face image, and audio-language logical prompts. We further propose a voice-face aligner module to project human voice and face representation into a shared space. We develop a multi-modal dataset based on VoxCeleb2 for MM-TSD training and evaluation. Additionally, we conduct comparative analysis and ablation studies for each category of prompts to validate the efficacy of each component in the proposed framework. Furthermore, our framework demonstrates versatility in performing various signal processing tasks, including speaker diarization and overlap speech detection, using task-specific prompts. MM-TSD achieves robust and comparable performance as a unified system compared to specialized models. Moreover, MM-TSD shows capability to handle complex conversations for real-world dataset. △ Less

Submitted 11 June, 2024; originally announced June 2024.

Comments: 13 pages, 7 figures

T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Text

Authors: Aoxiong Yin, Haoyuan Li, Kai Shen, Siliang Tang, Yueting Zhuang

Abstract: In this work, we propose a two-stage sign language production (SLP) paradigm that first encodes sign language sequences into discrete codes and then autoregressively generates sign language from text based on the learned codebook. However, existing vector quantization (VQ) methods are fixed-length encodings, overlooking the uneven information density in sign language, which leads to under-encoding… ▽ More In this work, we propose a two-stage sign language production (SLP) paradigm that first encodes sign language sequences into discrete codes and then autoregressively generates sign language from text based on the learned codebook. However, existing vector quantization (VQ) methods are fixed-length encodings, overlooking the uneven information density in sign language, which leads to under-encoding of important regions and over-encoding of unimportant regions. To address this issue, we propose a novel dynamic vector quantization (DVA-VAE) model that can dynamically adjust the encoding length based on the information density in sign language to achieve accurate and compact encoding. Then, a GPT-like model learns to generate code sequences and their corresponding durations from spoken language text. Extensive experiments conducted on the PHOENIX14T dataset demonstrate the effectiveness of our proposed method. To promote sign language research, we propose a new large German sign language dataset, PHOENIX-News, which contains 486 hours of sign language videos, audio, and transcription texts.Experimental analysis on PHOENIX-News shows that the performance of our model can be further improved by increasing the size of the training data. Our project homepage is https://t2sgpt-demo.yinaoxiong.cn. △ Less

Submitted 11 June, 2024; originally announced June 2024.

Comments: Accepted by ACL 2024

Invariant regimes of Spencer scaling law for magnetic compression of rotating FRC plasma

Authors: Yiming Ma, ** Zhu, Bo Rao, Haolong Li

Abstract: The scaling laws for the magnetic compression of a toroidally rotating field reversed configuration (FRC) have been investigated in this work. The magnetohydrodynamics (MHD) simulations of the magnetic compression on rotating FRCs employing the NIMROD code [C. R. Sovinec \textit{et al.}, J. Comput. Phys. \textbf{195}, 355 (2004)], are compared with the Spencer's one-dimensional (1D) theory [R. L.… ▽ More The scaling laws for the magnetic compression of a toroidally rotating field reversed configuration (FRC) have been investigated in this work. The magnetohydrodynamics (MHD) simulations of the magnetic compression on rotating FRCs employing the NIMROD code [C. R. Sovinec \textit{et al.}, J. Comput. Phys. \textbf{195}, 355 (2004)], are compared with the Spencer's one-dimensional (1D) theory [R. L. Spencer \textit{et al.}, Phys. Fluids \textbf{26}, 1564 (1983)] for a wide range of initial flow speeds and profiles. The toroidal flow can influence the scalings directly through the alteration of the compressional work as also evidenced in the 1D adiabatic model, and indirectly by resha** the initial equilibrium. However, in comparison to the static initial FRC equilibrium cases, the pressure and the radius scalings remain invariant for the magnetic compression ratio $B_{w2}/B_{w1}$ up to 6 in presence of the initial equilibrium flow, suggesting a broader applicable regime of the Spencer scaling law for FRC magnetic compression. The invariant scaling has been proven a natural consequence of the conservation of angular momentum of both fluid and magnetic field during the dynamic compression process. △ Less

Submitted 11 June, 2024; originally announced June 2024.

Evolving Subnetwork Training for Large Language Models

Authors: Hanqi Li, Lu Chen, Da Ma, Zijian Wu, Su Zhu, Kai Yu

Abstract: Large language models have ushered in a new era of artificial intelligence research. However, their substantial training costs hinder further development and widespread adoption. In this paper, inspired by the redundancy in the parameters of large language models, we propose a novel training paradigm: Evolving Subnetwork Training (EST). EST samples subnetworks from the layers of the large language… ▽ More Large language models have ushered in a new era of artificial intelligence research. However, their substantial training costs hinder further development and widespread adoption. In this paper, inspired by the redundancy in the parameters of large language models, we propose a novel training paradigm: Evolving Subnetwork Training (EST). EST samples subnetworks from the layers of the large language model and from commonly used modules within each layer, Multi-Head Attention (MHA) and Multi-Layer Perceptron (MLP). By gradually increasing the size of the subnetworks during the training process, EST can save the cost of training. We apply EST to train GPT2 model and TinyLlama model, resulting in 26.7\% FLOPs saving for GPT2 and 25.0\% for TinyLlama without an increase in loss on the pre-training dataset. Moreover, EST leads to performance improvements in downstream tasks, indicating that it benefits generalization. Additionally, we provide intuitive theoretical studies based on training dynamics and Dropout theory to ensure the feasibility of EST. Our code is available at https://github.com/OpenDFM/EST. △ Less

Submitted 11 June, 2024; originally announced June 2024.

Comments: Accepted to ICML 2024

Efficient combination of observational and experimental datasets under general restrictions on outcome mean functions

Authors: Harrison H. Li

Abstract: A researcher collecting data from a randomized controlled trial (RCT) often has access to an auxiliary observational dataset that may be confounded or otherwise biased for estimating causal effects. Common modeling assumptions impose restrictions on the outcome mean function - the conditional expectation of the outcome of interest given observed covariates - in the two datasets. Running examples f… ▽ More A researcher collecting data from a randomized controlled trial (RCT) often has access to an auxiliary observational dataset that may be confounded or otherwise biased for estimating causal effects. Common modeling assumptions impose restrictions on the outcome mean function - the conditional expectation of the outcome of interest given observed covariates - in the two datasets. Running examples from the literature include settings where the observational dataset is subject to outcome-mediated selection bias or to confounding bias taking an assumed parametric form. We propose a succinct framework to derive the efficient influence function for any identifiable pathwise differentiable estimand under a general class of restrictions on the outcome mean function. This uncovers surprising results that with homoskedastic outcomes and a constant propensity score in the RCT, even strong parametric assumptions cannot improve the semiparametric lower bound for estimating various average treatment effects. We then leverage double machine learning to construct a one-step estimator that achieves the semiparametric efficiency bound even in cases when the outcome mean function and other nuisance parameters are estimated nonparametrically. The goal is to empower a researcher with custom, previously unstudied modeling restrictions on the outcome mean function to systematically construct causal estimators that maximially leverage their assumptions for variance reduction. We demonstrate the finite sample precision gains of our estimator over existing approaches in extensions of various numerical studies and data examples from the literature. △ Less

Submitted 11 June, 2024; originally announced June 2024.

Comments: 52 pages, 4 figures

LinkQ: An LLM-Assisted Visual Interface for Knowledge Graph Question-Answering

Authors: Harry Li, Gabriel Appleby, Ashley Suh

Abstract: We present LinkQ, a system that leverages a large language model (LLM) to facilitate knowledge graph (KG) query construction through natural language question-answering. Traditional approaches often require detailed knowledge of complex graph querying languages, limiting the ability for users -- even experts -- to acquire valuable insights from KG data. LinkQ simplifies this process by first inter… ▽ More We present LinkQ, a system that leverages a large language model (LLM) to facilitate knowledge graph (KG) query construction through natural language question-answering. Traditional approaches often require detailed knowledge of complex graph querying languages, limiting the ability for users -- even experts -- to acquire valuable insights from KG data. LinkQ simplifies this process by first interpreting a user's question, then converting it into a well-formed KG query. By using the LLM to construct a query instead of directly answering the user's question, LinkQ guards against the LLM hallucinating or generating false, erroneous information. By integrating an LLM into LinkQ, users are able to conduct both exploratory and confirmatory data analysis, with the LLM hel** to iteratively refine open-ended questions into precise ones. To demonstrate the efficacy of LinkQ, we conducted a qualitative study with five KG practitioners and distill their feedback. Our results indicate that practitioners find LinkQ effective for KG question-answering, and desire future LLM-assisted systems for the exploratory analysis of graph databases. △ Less

Submitted 7 June, 2024; originally announced June 2024.

HORAE: A Domain-Agnostic Modeling Language for Automating Multimodal Service Regulation

Authors: Yutao Sun, Mingshuai Chen, Kangjia Zhao, He Li, **tao Chen, Linyu Yang, Zhongyi Wang, Tiancheng Zhao, Jianwei Yin

Abstract: Artificial intelligence is rapidly encroaching on the field of service regulation. This work presents the design principles behind HORAE, a unified specification language to model multimodal regulation rules across a diverse set of domains. We show how HORAE facilitates an intelligent service regulation pipeline by further exploiting a fine-tuned large language model named HORAE that automates the… ▽ More Artificial intelligence is rapidly encroaching on the field of service regulation. This work presents the design principles behind HORAE, a unified specification language to model multimodal regulation rules across a diverse set of domains. We show how HORAE facilitates an intelligent service regulation pipeline by further exploiting a fine-tuned large language model named HORAE that automates the HORAE modeling process, thereby yielding an end-to-end framework for fully automated intelligent service regulation. △ Less

Submitted 6 June, 2024; originally announced June 2024.