-
Expressive Gaussian Human Avatars from Monocular RGB Video
Authors:
Hezhen Hu,
Zhiwen Fan,
Tianhao Wu,
Yihan Xi,
Seoyoung Lee,
Georgios Pavlakos,
Zhangyang Wang
Abstract:
Nuanced expressiveness, particularly through fine-grained hand and facial expressions, is pivotal for enhancing the realism and vitality of digital human representations. In this work, we focus on investigating the expressiveness of human avatars when learned from monocular RGB video; a setting that introduces new challenges in capturing and animating fine-grained details. To this end, we introduc…
▽ More
Nuanced expressiveness, particularly through fine-grained hand and facial expressions, is pivotal for enhancing the realism and vitality of digital human representations. In this work, we focus on investigating the expressiveness of human avatars when learned from monocular RGB video; a setting that introduces new challenges in capturing and animating fine-grained details. To this end, we introduce EVA, a drivable human model that meticulously sculpts fine details based on 3D Gaussians and SMPL-X, an expressive parametric human model. Focused on enhancing expressiveness, our work makes three key contributions. First, we highlight the critical importance of aligning the SMPL-X model with RGB frames for effective avatar learning. Recognizing the limitations of current SMPL-X prediction methods for in-the-wild videos, we introduce a plug-and-play module that significantly ameliorates misalignment issues. Second, we propose a context-aware adaptive density control strategy, which is adaptively adjusting the gradient thresholds to accommodate the varied granularity across body parts. Last but not least, we develop a feedback mechanism that predicts per-pixel confidence to better guide the learning of 3D Gaussians. Extensive experiments on two benchmarks demonstrate the superiority of our framework both quantitatively and qualitatively, especially on the fine-grained hand and facial details. See the project website at \url{https://evahuman.github.io}
△ Less
Submitted 3 July, 2024;
originally announced July 2024.
-
Revealing the Electronic Structure of NiPS$_3$ through Synchrotron-Based ARPES and Alkali Metal Dosing
Authors:
Yifeng Cao,
Qishuo Tan,
Yucheng Guo,
Clóvis Guerim Vieira,
Mário S. C. Mazzon,
Jude Laverock,
Nicholas Russo,
Hongze Gao,
Chris Jozwiak,
Aaron Bostwick,
Eli Rotenberg,
**ghua Guo,
Ming Yi,
Matheus J. S. Matos,
Xi Ling,
Kevin E. Smith
Abstract:
This study presents a comprehensive analysis of the band structure in NiPS$_3$, a van der Waals layered antiferromagnet, utilizing high-resolution synchrotron-based angle-resolved photoemission spectroscopy (ARPES) and corroborative density functional theory (DFT) calculations. By tuning the parameters of the light source, we obtained a very clear and wide energy range band structure of NiPS$_3$.…
▽ More
This study presents a comprehensive analysis of the band structure in NiPS$_3$, a van der Waals layered antiferromagnet, utilizing high-resolution synchrotron-based angle-resolved photoemission spectroscopy (ARPES) and corroborative density functional theory (DFT) calculations. By tuning the parameters of the light source, we obtained a very clear and wide energy range band structure of NiPS$_3$. Comparison with DFT calculations allows for the identification of the orbital character of the observed bands. Our DFT calculations perfectly match the experimental results, and no adaptations were made to the calculations based on the experimental outcomes. The appearance of novel electronic structure upon alkali metal dosing (AMD) were also obtained in this ARPES study. Above valence band maximum, structure of conduction bands and bands from defect states were firstly observed in NiPS$_3$. We provide the direct determination of the band gap of NiPS$_3$ as 1.3 eV from the band structure by AMD. In addition, detailed temperature dependent ARPES spectra were obtained across a range that spans both below and above the Néel transition temperature of NiPS$_3$. We found that the paramagnetic and antiferromagnetic states have almost identical spectra, indicating the highly localized nature of Ni $d$ states.
△ Less
Submitted 2 July, 2024;
originally announced July 2024.
-
Camera-LiDAR Cross-modality Gait Recognition
Authors:
Wenxuan Guo,
Ying** Liang,
Zhiyu Pan,
Ziheng Xi,
Jianjiang Feng,
Jie Zhou
Abstract:
Gait recognition is a crucial biometric identification technique. Camera-based gait recognition has been widely applied in both research and industrial fields. LiDAR-based gait recognition has also begun to evolve most recently, due to the provision of 3D structural information. However, in certain applications, cameras fail to recognize persons, such as in low-light environments and long-distance…
▽ More
Gait recognition is a crucial biometric identification technique. Camera-based gait recognition has been widely applied in both research and industrial fields. LiDAR-based gait recognition has also begun to evolve most recently, due to the provision of 3D structural information. However, in certain applications, cameras fail to recognize persons, such as in low-light environments and long-distance recognition scenarios, where LiDARs work well. On the other hand, the deployment cost and complexity of LiDAR systems limit its wider application. Therefore, it is essential to consider cross-modality gait recognition between cameras and LiDARs for a broader range of applications. In this work, we propose the first cross-modality gait recognition framework between Camera and LiDAR, namely CL-Gait. It employs a two-stream network for feature embedding of both modalities. This poses a challenging recognition task due to the inherent matching between 3D and 2D data, exhibiting significant modality discrepancy. To align the feature spaces of the two modalities, i.e., camera silhouettes and LiDAR points, we propose a contrastive pre-training strategy to mitigate modality discrepancy. To make up for the absence of paired camera-LiDAR data for pre-training, we also introduce a strategy for generating data on a large scale. This strategy utilizes monocular depth estimated from single RGB images and virtual cameras to generate pseudo point clouds for contrastive pre-training. Extensive experiments show that the cross-modality gait recognition is very challenging but still contains potential and feasibility with our proposed model and pre-training strategy. To the best of our knowledge, this is the first work to address cross-modality gait recognition.
△ Less
Submitted 2 July, 2024; v1 submitted 2 July, 2024;
originally announced July 2024.
-
GVDIFF: Grounded Text-to-Video Generation with Diffusion Models
Authors:
Huanzhang Dou,
Ruixiang Li,
Wei Su,
Xi Li
Abstract:
In text-to-video (T2V) generation, significant attention has been directed toward its development, yet unifying discrete and continuous grounding conditions in T2V generation remains under-explored. This paper proposes a Grounded text-to-Video generation framework, termed GVDIFF. First, we inject the grounding condition into the self-attention through an uncertainty-based representation to explici…
▽ More
In text-to-video (T2V) generation, significant attention has been directed toward its development, yet unifying discrete and continuous grounding conditions in T2V generation remains under-explored. This paper proposes a Grounded text-to-Video generation framework, termed GVDIFF. First, we inject the grounding condition into the self-attention through an uncertainty-based representation to explicitly guide the focus of the network. Second, we introduce a spatial-temporal grounding layer that connects the grounding condition with target objects and enables the model with the grounded generation capacity in the spatial-temporal domain. Third, our dynamic gate network adaptively skips the redundant grounding process to selectively extract grounding information and semantics while improving efficiency. We extensively evaluate the grounded generation capacity of GVDIFF and demonstrate its versatility in applications, including long-range video generation, sequential prompts, and object-specific editing.
△ Less
Submitted 1 July, 2024;
originally announced July 2024.
-
To Forget or Not? Towards Practical Knowledge Unlearning for Large Language Models
Authors:
Bozhong Tian,
Xiaozhuan Liang,
Siyuan Cheng,
Qingbin Liu,
Mengru Wang,
Dianbo Sui,
Xi Chen,
Huajun Chen,
Ningyu Zhang
Abstract:
Large Language Models (LLMs) trained on extensive corpora inevitably retain sensitive data, such as personal privacy information and copyrighted material. Recent advancements in knowledge unlearning involve updating LLM parameters to erase specific knowledge. However, current unlearning paradigms are mired in vague forgetting boundaries, often erasing knowledge indiscriminately. In this work, we i…
▽ More
Large Language Models (LLMs) trained on extensive corpora inevitably retain sensitive data, such as personal privacy information and copyrighted material. Recent advancements in knowledge unlearning involve updating LLM parameters to erase specific knowledge. However, current unlearning paradigms are mired in vague forgetting boundaries, often erasing knowledge indiscriminately. In this work, we introduce KnowUnDo, a benchmark containing copyrighted content and user privacy domains to evaluate if the unlearning process inadvertently erases essential knowledge. Our findings indicate that existing unlearning methods often suffer from excessive unlearning. To address this, we propose a simple yet effective method, MemFlex, which utilizes gradient information to precisely target and unlearn sensitive parameters. Experimental results show that MemFlex is superior to existing methods in both precise knowledge unlearning and general knowledge retaining of LLMs. Code and dataset will be released at https://github.com/zjunlp/KnowUnDo.
△ Less
Submitted 1 July, 2024;
originally announced July 2024.
-
Securing Distributed Network Digital Twin Systems Against Model Poisoning Attacks
Authors:
Zifan Zhang,
Minghong Fang,
Mingzhe Chen,
Gaolei Li,
Xi Lin,
Yuchen Liu
Abstract:
In the era of 5G and beyond, the increasing complexity of wireless networks necessitates innovative frameworks for efficient management and deployment. Digital twins (DTs), embodying real-time monitoring, predictive configurations, and enhanced decision-making capabilities, stand out as a promising solution in this context. Within a time-series data-driven framework that effectively maps wireless…
▽ More
In the era of 5G and beyond, the increasing complexity of wireless networks necessitates innovative frameworks for efficient management and deployment. Digital twins (DTs), embodying real-time monitoring, predictive configurations, and enhanced decision-making capabilities, stand out as a promising solution in this context. Within a time-series data-driven framework that effectively maps wireless networks into digital counterparts, encapsulated by integrated vertical and horizontal twinning phases, this study investigates the security challenges in distributed network DT systems, which potentially undermine the reliability of subsequent network applications such as wireless traffic forecasting. Specifically, we consider a minimal-knowledge scenario for all attackers, in that they do not have access to network data and other specialized knowledge, yet can interact with previous iterations of server-level models. In this context, we spotlight a novel fake traffic injection attack designed to compromise a distributed network DT system for wireless traffic prediction. In response, we then propose a defense mechanism, termed global-local inconsistency detection (GLID), to counteract various model poisoning threats. GLID strategically removes abnormal model parameters that deviate beyond a particular percentile range, thereby fortifying the security of network twinning process. Through extensive experiments on real-world wireless traffic datasets, our experimental evaluations show that both our attack and defense strategies significantly outperform existing baselines, highlighting the importance of security measures in the design and implementation of DTs for 5G and beyond network systems.
△ Less
Submitted 1 July, 2024;
originally announced July 2024.
-
Spectral evidence for NiPS3 as a Mott-Hubbard insulator
Authors:
Yifeng Cao,
Nicholas Russo,
Qishuo Tan,
Xi Ling,
**ghua Guo,
Yi-de Chuang,
Kevin E. Smith
Abstract:
The layered van der Waals trichalcogenide NiPS3 has attracted widespread attention due to its unique optical, magnetic, and electronic properties. The complexity of NiPS3 itself, however, has also led to ongoing debates regarding its characteristics such as the existence of self-doped ligand holes. In this study, X-ray absorption spectroscopy and resonant inelastic X-ray scattering have been appli…
▽ More
The layered van der Waals trichalcogenide NiPS3 has attracted widespread attention due to its unique optical, magnetic, and electronic properties. The complexity of NiPS3 itself, however, has also led to ongoing debates regarding its characteristics such as the existence of self-doped ligand holes. In this study, X-ray absorption spectroscopy and resonant inelastic X-ray scattering have been applied to investigate the electronic structure of NiPS3. With the aid of theoretical calculations using the charge-transfer multiplet model, we provide experimental evidence for NiPS3 being a Mott-Hubbard insulator rather than a charge-transfer insulator. Moreover, we explain why some previous XAS studies have concluded that NiPS3 is a charge-transfer insulator by comparing surface and bulk sensitive spectra.
△ Less
Submitted 1 July, 2024;
originally announced July 2024.
-
Quantifying cascading power outages during climate extremes considering renewable energy integration
Authors:
Luo Xu,
Ning Lin,
H. Vincent Poor,
Dazhi Xi,
A. T. D. Perera
Abstract:
Climate extremes, such as hurricanes, combined with large-scale integration of environment-sensitive renewables, could exacerbate the risk of widespread power outages. We introduce a coupled climate-energy model for cascading power outages, which comprehensively captures the impacts of evolving climate extremes on renewable generation, and transmission and distribution networks. The model is valid…
▽ More
Climate extremes, such as hurricanes, combined with large-scale integration of environment-sensitive renewables, could exacerbate the risk of widespread power outages. We introduce a coupled climate-energy model for cascading power outages, which comprehensively captures the impacts of evolving climate extremes on renewable generation, and transmission and distribution networks. The model is validated by the 2022 Puerto Rico catastrophic blackout during Hurricane Fiona, the first-ever system-wide blackout event with complete weather-induced outage records. The model presents a novel resilience pattern that was not captured by the present state-of-the-art models and reveals that early failure of certain critical components surprisingly enhances overall system resilience. Sensitivity analysis of various behind-the-meter solar integration scenarios demonstrates that lower integration levels (below 45%, including the current level) exhibit minimal impact on system resilience in this event. However, surpassing this critical level without additional flexibility resources can exacerbate the failure probability due to substantially enlarged energy imbalances.
△ Less
Submitted 1 July, 2024;
originally announced July 2024.
-
MOT: A Mixture of Actors Reinforcement Learning Method by Optimal Transport for Algorithmic Trading
Authors:
Xi Cheng,
**ghao Zhang,
Yunan Zeng,
Wenfang Xue
Abstract:
Algorithmic trading refers to executing buy and sell orders for specific assets based on automatically identified trading opportunities. Strategies based on reinforcement learning (RL) have demonstrated remarkable capabilities in addressing algorithmic trading problems. However, the trading patterns differ among market conditions due to shifted distribution data. Ignoring multiple patterns in the…
▽ More
Algorithmic trading refers to executing buy and sell orders for specific assets based on automatically identified trading opportunities. Strategies based on reinforcement learning (RL) have demonstrated remarkable capabilities in addressing algorithmic trading problems. However, the trading patterns differ among market conditions due to shifted distribution data. Ignoring multiple patterns in the data will undermine the performance of RL. In this paper, we propose MOT,which designs multiple actors with disentangled representation learning to model the different patterns of the market. Furthermore, we incorporate the Optimal Transport (OT) algorithm to allocate samples to the appropriate actor by introducing a regularization loss term. Additionally, we propose Pretrain Module to facilitate imitation learning by aligning the outputs of actors with expert strategy and better balance the exploration and exploitation of RL. Experimental results on real futures market data demonstrate that MOT exhibits excellent profit capabilities while balancing risks. Ablation studies validate the effectiveness of the components of MOT.
△ Less
Submitted 2 June, 2024;
originally announced July 2024.
-
E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awareness
Authors:
Robin Courant,
Nicolas Dufour,
Xi Wang,
Marc Christie,
Vicky Kalogeiton
Abstract:
Stories and emotions in movies emerge through the effect of well-thought-out directing decisions, in particular camera placement and movement over time. Crafting compelling camera trajectories remains a complex iterative process, even for skilful artists. To tackle this, in this paper, we propose a dataset called the Exceptional Trajectories (E.T.) with camera trajectories along with character inf…
▽ More
Stories and emotions in movies emerge through the effect of well-thought-out directing decisions, in particular camera placement and movement over time. Crafting compelling camera trajectories remains a complex iterative process, even for skilful artists. To tackle this, in this paper, we propose a dataset called the Exceptional Trajectories (E.T.) with camera trajectories along with character information and textual captions encompassing descriptions of both camera and character. To our knowledge, this is the first dataset of its kind. To show the potential applications of the E.T. dataset, we propose a diffusion-based approach, named DIRECTOR, which generates complex camera trajectories from textual captions that describe the relation and synchronisation between the camera and characters. To ensure robust and accurate evaluations, we train on the E.T. dataset CLaTr, a Contrastive Language-Trajectory embedding for evaluation metrics. We posit that our proposed dataset and method significantly advance the democratization of cinematography, making it more accessible to common users.
△ Less
Submitted 1 July, 2024;
originally announced July 2024.
-
Data fusion for efficiency gain in ATE estimation: A practical review with simulations
Authors:
Xi Lin,
Jens Magelund Tarp,
Robin J. Evans
Abstract:
The integration of real-world data (RWD) and randomized controlled trials (RCT) is increasingly important for advancing causal inference in scientific research. This combination holds great promise for enhancing the efficiency of causal effect estimation, offering benefits such as reduced trial participant numbers and expedited drug access for patients. Despite the availability of numerous data fu…
▽ More
The integration of real-world data (RWD) and randomized controlled trials (RCT) is increasingly important for advancing causal inference in scientific research. This combination holds great promise for enhancing the efficiency of causal effect estimation, offering benefits such as reduced trial participant numbers and expedited drug access for patients. Despite the availability of numerous data fusion methods, selecting the most appropriate one for a specific research question remains challenging. This paper systematically reviews and compares these methods regarding their assumptions, limitations, and implementation complexities. Through simulations reflecting real-world scenarios, we identify a prevalent risk-reward trade-off across different methods. We investigate and interpret this trade-off, providing key insights into the strengths and weaknesses of various methods; thereby hel** researchers navigate through the application of data fusion for improved causal inference.
△ Less
Submitted 1 July, 2024;
originally announced July 2024.
-
$\text{Memory}^3$: Language Modeling with Explicit Memory
Authors:
Hongkang Yang,
Zehao Lin,
Wen** Wang,
Hao Wu,
Zhiyu Li,
Bo Tang,
Wenqiang Wei,
**bo Wang,
Zeyun Tang,
Shichao Song,
Chenyang Xi,
Yu Yu,
Kai Chen,
Feiyu Xiong,
Linpeng Tang,
Weinan E
Abstract:
The training and inference of large language models (LLMs) are together a costly process that transports knowledge from raw data to meaningful computation. Inspired by the memory hierarchy of the human brain, we reduce this cost by equip** LLMs with explicit memory, a memory format cheaper than model parameters and text retrieval-augmented generation (RAG). Conceptually, with most of its knowled…
▽ More
The training and inference of large language models (LLMs) are together a costly process that transports knowledge from raw data to meaningful computation. Inspired by the memory hierarchy of the human brain, we reduce this cost by equip** LLMs with explicit memory, a memory format cheaper than model parameters and text retrieval-augmented generation (RAG). Conceptually, with most of its knowledge externalized to explicit memories, the LLM can enjoy a smaller parameter size, training cost, and inference cost, all proportional to the amount of remaining "abstract knowledge". As a preliminary proof of concept, we train from scratch a 2.4B LLM, which achieves better performance than much larger LLMs as well as RAG models, and maintains higher decoding speed than RAG. The model is named $\text{Memory}^3$, since explicit memory is the third form of memory in LLMs after implicit memory (model parameters) and working memory (context key-values). We introduce a memory circuitry theory to support the externalization of knowledge, and present novel techniques including a memory sparsification mechanism that makes storage tractable and a two-stage pretraining scheme that facilitates memory formation.
△ Less
Submitted 1 July, 2024;
originally announced July 2024.
-
Randomized linear solvers for computational architectures with straggling workers
Authors:
Vassilis Kalantzis,
Yuanzhe Xi,
Lior Horesh,
Yousef Saad
Abstract:
In this paper, we consider the iterative solution of sparse systems of linear algebraic equations under the condition that sparse matrix-vector products with the coefficient matrix are computed only partially. At the same time, non-computed entries are set to zeros. We assume that both the number of computed entries and their associated row index set are random variables, with the row index set sa…
▽ More
In this paper, we consider the iterative solution of sparse systems of linear algebraic equations under the condition that sparse matrix-vector products with the coefficient matrix are computed only partially. At the same time, non-computed entries are set to zeros. We assume that both the number of computed entries and their associated row index set are random variables, with the row index set sampled uniformly given the number of computed entries. This model of computations is prevalent to that realized in hybrid cloud computing architectures following the controller-worker distributed model under the influence of straggling workers. We propose a randomized Richardson iterative scheme and a randomized Chebyshev semi-iterative method within this model and prove the sufficient conditions for their convergence in expectation. Numerical experiments verify the presented theoretical results as well as the effectiveness of the proposed schemes on a few sparse matrix problems.
△ Less
Submitted 1 July, 2024;
originally announced July 2024.
-
Instruct-IPT: All-in-One Image Processing Transformer via Weight Modulation
Authors:
Yuchuan Tian,
Jianhong Han,
Hanting Chen,
Yuanyuan Xi,
Guoyang Zhang,
Jie Hu,
Chao Xu,
Yunhe Wang
Abstract:
Due to the unaffordable size and intensive computation costs of low-level vision models, All-in-One models that are designed to address a handful of low-level vision tasks simultaneously have been popular. However, existing All-in-One models are limited in terms of the range of tasks and performance. To overcome these limitations, we propose Instruct-IPT -- an All-in-One Image Processing Transform…
▽ More
Due to the unaffordable size and intensive computation costs of low-level vision models, All-in-One models that are designed to address a handful of low-level vision tasks simultaneously have been popular. However, existing All-in-One models are limited in terms of the range of tasks and performance. To overcome these limitations, we propose Instruct-IPT -- an All-in-One Image Processing Transformer that could effectively address manifold image restoration tasks with large inter-task gaps, such as denoising, deblurring, deraining, dehazing, and desnowing. Rather than popular feature adaptation methods, we propose weight modulation that adapts weights to specific tasks. Firstly, we figure out task-sensitive weights via a toy experiment and introduce task-specific biases on top of them. Secondly, we conduct rank analysis for a good compression strategy and perform low-rank decomposition on the biases. Thirdly, we propose synchronous training that updates the task-general backbone model and the task-specific biases simultaneously. In this way, the model is instructed to learn general and task-specific knowledge. Via our simple yet effective method that instructs the IPT to be task experts, Instruct-IPT could better cooperate between tasks with distinct characteristics at humble costs. Further, we propose to maneuver Instruct-IPT with text instructions for better user interfaces. We have conducted experiments on Instruct-IPT to demonstrate the effectiveness of our method on manifold tasks, and we have effectively extended our method to diffusion denoisers as well. The code is available at https://github.com/huawei-noah/Pretrained-IPT.
△ Less
Submitted 30 June, 2024;
originally announced July 2024.
-
Screening of half-Heuslers with temperature-induced band convergence and enhanced thermoelectric properties
Authors:
**yang Xi,
Zirui Dong,
Menghan Gao,
Jun Luo,
Jiong Yang
Abstract:
Enhancing band convergence is an effective way to optimize the thermoelectric (TE) properties of materials. However, the temperature-induced band renormalization is commonly ignored. By employing the recently-developed electron-phonon renormalization (EPR) method, the nature of band renormalization in half-Heusler (HH) compounds TiCoSb and NbFeSb is revealed, and the key factors for temperature-in…
▽ More
Enhancing band convergence is an effective way to optimize the thermoelectric (TE) properties of materials. However, the temperature-induced band renormalization is commonly ignored. By employing the recently-developed electron-phonon renormalization (EPR) method, the nature of band renormalization in half-Heusler (HH) compounds TiCoSb and NbFeSb is revealed, and the key factors for temperature-induced conduction band convergence in HH are found out. Using these as the screening criteria, 3 out of 274 HHs (TiRhBi, TiPtSn, NbPtTl) are then stood out from our MatHub-3d database. Taking TiPtSn as the example, it shows the conduction band convergence at mid-high temperature, and further resulting in enhanced Seebeck coefficient S: e.g., at 600 K with electron concentration 10^20 cm^-3, the predicted S with and without renormalized band is 352.83 uV/K and 289.52 uV/K, respectively. Herein, the former is closer to our measurement value of 338.79 uV/K. Besides, the effective masses obtained from calculation and experiment are both enlarged with temperature, indicating the existence of band convergence. Our work demonstrates for the first time the significance of adding the temperature effect on electronic structure in the design of potential high-performance TE materials.
△ Less
Submitted 29 June, 2024;
originally announced July 2024.
-
The role of lattice thermal conductivity suppression by dopants from a holistic perspective
Authors:
Shengnan Dai,
Shijie Zhang,
Ye Sheng,
Erting Dong,
Sheng Sun,
Lili Xi,
G. Jeffrey Snyder,
**yang Xi,
Jiong Yang
Abstract:
Dopants play an important role in improving electrical and thermal transport. In the traditional perspective, a dopant suppresses lattice thermal conductivity kL by adding point defect (PD) scattering term to the phonon relaxation time, which has been adopted for decades. In this study, we propose an innovative perspective to solve the kL of defective systems-the holistic approach, i.e., treating…
▽ More
Dopants play an important role in improving electrical and thermal transport. In the traditional perspective, a dopant suppresses lattice thermal conductivity kL by adding point defect (PD) scattering term to the phonon relaxation time, which has been adopted for decades. In this study, we propose an innovative perspective to solve the kL of defective systems-the holistic approach, i.e., treating dopant and matrix as a holism. This approach allows us to handle the influences from defects explicitly by the calculations of defective systems, about their changed phonon dispersion, phonon-phonon and electron-phonon interaction, etc, due to the existence of dopants. The kL reduction between defective MxNb1-xFeSb (M=V, Ti) and NbFeSb is used as an example for the holistic approach, and comparable results with experiments are obtained. It is notable that light elemental dopants also induced the avoided-crossing behavior. It can be further rationalized by a one-dimensional atomic chain model. The mass and force constant imbalance generally generates the avoided-crossing phonons, mathematically in a similar way as the coefficients in traditional PD scattering, but along a different direction in kL reduction. Our work provides another perspective for understanding the mechanism of dopants influence in material's thermal transport.
△ Less
Submitted 29 June, 2024;
originally announced July 2024.
-
Futility analyses for the MCP-Mod methodology based on longitudinal models
Authors:
Björn Bornkamp,
Jie Zhou,
Dong Xi,
Weihua Cao
Abstract:
This article discusses futility analyses for the MCP-Mod methodology. Formulas are derived for calculating predictive and conditional power for MCP-Mod, which also cover the case when longitudinal models are used allowing to utilize incomplete data from patients at interim. A simulation study is conducted to evaluate the repeated sampling properties of the proposed decision rules and to assess the…
▽ More
This article discusses futility analyses for the MCP-Mod methodology. Formulas are derived for calculating predictive and conditional power for MCP-Mod, which also cover the case when longitudinal models are used allowing to utilize incomplete data from patients at interim. A simulation study is conducted to evaluate the repeated sampling properties of the proposed decision rules and to assess the benefit of using a longitudinal versus a completer only model for decision making at interim. The results suggest that the proposed methods perform adequately and a longitudinal analysis outperforms a completer only analysis, particularly when the recruitment speed is higher and the correlation over time is larger. The proposed methodology is illustrated using real data from a dose-finding study for severe uncontrolled asthma.
△ Less
Submitted 28 June, 2024;
originally announced June 2024.
-
Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Model
Authors:
Longrong Yang,
Dong Sheng,
Chaoxiang Cai,
Fan Yang,
Size Li,
Di Zhang,
Xi Li
Abstract:
The Mixture-of-Experts (MoE) has gained increasing attention in the study of Large Vision-Language Models (LVLMs). It uses a sparse model to replace the dense model, achieving comparable performance while activating fewer parameters during inference, thus significantly reducing the inference cost. Existing MoE methods in LVLMs encourage different experts to handle different tokens, and thus they e…
▽ More
The Mixture-of-Experts (MoE) has gained increasing attention in the study of Large Vision-Language Models (LVLMs). It uses a sparse model to replace the dense model, achieving comparable performance while activating fewer parameters during inference, thus significantly reducing the inference cost. Existing MoE methods in LVLMs encourage different experts to handle different tokens, and thus they employ a router to predict the routing for each token. However, the predictions are based solely on sample features and do not truly reveal the optimization direction of tokens. This can lead to severe optimization conflicts between different tokens within an expert. To address this problem, this paper proposes a novel method based on token-level gradient analysis. Specifically, we first use token-level gradients to identify conflicting tokens in experts. Then, we add a specialized loss tailored to eliminate conflicts among tokens within each expert. Our method can serve as a plug-in for diverse Large Vision-Language Models, and extensive experimental results demonstrate the effectiveness of our method. The code will be publicly available at https://github.com/longrongyang/STGC.
△ Less
Submitted 28 June, 2024;
originally announced June 2024.
-
EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splatting
Authors:
Daiwei Zhang,
Gengyan Li,
Jiajie Li,
Mickaël Bressieux,
Otmar Hilliges,
Marc Pollefeys,
Luc Van Gool,
Xi Wang
Abstract:
Human activities are inherently complex, and even simple household tasks involve numerous object interactions. To better understand these activities and behaviors, it is crucial to model their dynamic interactions with the environment. The recent availability of affordable head-mounted cameras and egocentric data offers a more accessible and efficient means to understand dynamic human-object inter…
▽ More
Human activities are inherently complex, and even simple household tasks involve numerous object interactions. To better understand these activities and behaviors, it is crucial to model their dynamic interactions with the environment. The recent availability of affordable head-mounted cameras and egocentric data offers a more accessible and efficient means to understand dynamic human-object interactions in 3D environments. However, most existing methods for human activity modeling either focus on reconstructing 3D models of hand-object or human-scene interactions or on map** 3D scenes, neglecting dynamic interactions with objects. The few existing solutions often require inputs from multiple sources, including multi-camera setups, depth-sensing cameras, or kinesthetic sensors. To this end, we introduce EgoGaussian, the first method capable of simultaneously reconstructing 3D scenes and dynamically tracking 3D object motion from RGB egocentric input alone. We leverage the uniquely discrete nature of Gaussian Splatting and segment dynamic interactions from the background. Our approach employs a clip-level online learning pipeline that leverages the dynamic nature of human activities, allowing us to reconstruct the temporal evolution of the scene in chronological order and track rigid object motion. Additionally, our method automatically segments object and background Gaussians, providing 3D representations for both static scenes and dynamic objects. EgoGaussian outperforms previous NeRF and Dynamic Gaussian methods in challenging in-the-wild videos and we also qualitatively demonstrate the high quality of the reconstructed models.
△ Less
Submitted 28 June, 2024;
originally announced June 2024.
-
A Survey on Deep Clustering: From the Prior Perspective
Authors:
Yiding Lu,
Haobin Li,
Yunfan Li,
Yijie Lin,
Xi Peng
Abstract:
Facilitated by the powerful feature extraction ability of neural networks, deep clustering has achieved great success in analyzing high-dimensional and complex real-world data. The performance of deep clustering methods is affected by various factors such as network structures and learning objectives. However, as pointed out in this survey, the essence of deep clustering lies in the incorporation…
▽ More
Facilitated by the powerful feature extraction ability of neural networks, deep clustering has achieved great success in analyzing high-dimensional and complex real-world data. The performance of deep clustering methods is affected by various factors such as network structures and learning objectives. However, as pointed out in this survey, the essence of deep clustering lies in the incorporation and utilization of prior knowledge, which is largely ignored by existing works. From pioneering deep clustering methods based on data structure assumptions to recent contrastive clustering methods based on data augmentation invariances, the development of deep clustering intrinsically corresponds to the evolution of prior knowledge. In this survey, we provide a comprehensive review of deep clustering methods by categorizing them into six types of prior knowledge. We find that in general the prior innovation follows two trends, namely, i) from mining to constructing, and ii) from internal to external. Besides, we provide a benchmark on five widely-used datasets and analyze the performance of methods with diverse priors. By providing a novel prior knowledge perspective, we hope this survey could provide some novel insights and inspire future research in the deep clustering community.
△ Less
Submitted 30 June, 2024; v1 submitted 27 June, 2024;
originally announced June 2024.
-
Non-modal growth analysis of high-speed flows over an inclined cone
Authors:
Xi Chen,
Bingbing Wan,
Guohua Tu,
Maochang Duan,
Xiaohu Li,
Jianqiang Chen
Abstract:
Spatial optimal responses to both inlet disturbances and harmonic external forcing for hypersonic flows over a blunt cone at nonzero angles of attack are obtained by efficiently solving the direct-adjoint equations with a parabolic approach. In either case, the most amplified disturbances initially take the form of localized streamwise vortices on the windward side and will undergo a two-stage evo…
▽ More
Spatial optimal responses to both inlet disturbances and harmonic external forcing for hypersonic flows over a blunt cone at nonzero angles of attack are obtained by efficiently solving the direct-adjoint equations with a parabolic approach. In either case, the most amplified disturbances initially take the form of localized streamwise vortices on the windward side and will undergo a two-stage evolution process when propagating downstream: they first experience a substantial algebraic growth by exploiting the Orr and lift-up mechanisms, and then smoothly transition to a quasi exponential-growth stage driven by the crossflow-instability mechanism, accompanied by an azimuthal advection of the disturbance structure towards the leeward side. The algebraic-growth phase is most receptive to the external forcing, whereas the exponential-growth stage relies on the disturbance frequency and can be significantly strengthened by increasing the angle of attack. The wavemaker delineating the structural sensitivity region for the optimal gain is shown to lie on the windward side immediately downstream of the inlet, implying a potent control strategy. Additionally, considerable non-modal growth is also observed for broadband high-frequency disturbances residing in the entropy layer.
△ Less
Submitted 26 June, 2024;
originally announced June 2024.
-
Bounded asymptotics for high-order moments in wall turbulence
Authors:
Xi Chen,
Katepalli R. Sreenivasan
Abstract:
Turbulent wall-flows are the most important means for understanding the effects of boundary conditions and fluid viscosity on turbulent fluctuations. There has been considerable recent research on mean-square fluctuations. Here, we present expressions for high-order moments of streamwise velocity fluctuation $u$, in the form $ \langle u^{+2q} \rangle^{1/q}=α_q-β_q y^{\ast1/4}$; $q$ is an integer,…
▽ More
Turbulent wall-flows are the most important means for understanding the effects of boundary conditions and fluid viscosity on turbulent fluctuations. There has been considerable recent research on mean-square fluctuations. Here, we present expressions for high-order moments of streamwise velocity fluctuation $u$, in the form $ \langle u^{+2q} \rangle^{1/q}=α_q-β_q y^{\ast1/4}$; $q$ is an integer, $α_q$ and $β_q$ are constants independent of the friction Reynolds number $Re_τ$, and $y^{\ast} = y/δ$ is the distance away from the wall, normalized by the flow thickness $δ$; in particular, $α_q =μ+σq$ according to the `linear q-norm Gaussian' process, where $μ$ and $σ$ are flow-independent constants. Excellent agreement is found between these formulae and available data in pipes, channels and boundary layers for $1 \leq q \leq 5$. For fixed $y^+ = y^*Re_τ$, the present {formulation} leads to the bounded state $\langle u^{+2q} \rangle^{1/q}=α_q$ as $Re_τ\rightarrow\infty$ while the attached eddy model predicts that {the} moments continually grow as log Reynolds number.
△ Less
Submitted 26 June, 2024;
originally announced June 2024.
-
Intrinsic Action Tendency Consistency for Cooperative Multi-Agent Reinforcement Learning
Authors:
Junkai Zhang,
Yifan Zhang,
Xi Sheryl Zhang,
Yifan Zang,
Jian Cheng
Abstract:
Efficient collaboration in the centralized training with decentralized execution (CTDE) paradigm remains a challenge in cooperative multi-agent systems. We identify divergent action tendencies among agents as a significant obstacle to CTDE's training efficiency, requiring a large number of training samples to achieve a unified consensus on agents' policies. This divergence stems from the lack of a…
▽ More
Efficient collaboration in the centralized training with decentralized execution (CTDE) paradigm remains a challenge in cooperative multi-agent systems. We identify divergent action tendencies among agents as a significant obstacle to CTDE's training efficiency, requiring a large number of training samples to achieve a unified consensus on agents' policies. This divergence stems from the lack of adequate team consensus-related guidance signals during credit assignments in CTDE. To address this, we propose Intrinsic Action Tendency Consistency, a novel approach for cooperative multi-agent reinforcement learning. It integrates intrinsic rewards, obtained through an action model, into a reward-additive CTDE (RA-CTDE) framework. We formulate an action model that enables surrounding agents to predict the central agent's action tendency. Leveraging these predictions, we compute a cooperative intrinsic reward that encourages agents to match their actions with their neighbors' predictions. We establish the equivalence between RA-CTDE and CTDE through theoretical analyses, demonstrating that CTDE's training process can be achieved using agents' individual targets. Building on this insight, we introduce a novel method to combine intrinsic rewards and CTDE. Extensive experiments on challenging tasks in SMAC and GRF benchmarks showcase the improved performance of our method.
△ Less
Submitted 26 June, 2024;
originally announced June 2024.
-
ScanFormer: Referring Expression Comprehension by Iteratively Scanning
Authors:
Wei Su,
Peihan Miao,
Huanzhang Dou,
Xi Li
Abstract:
Referring Expression Comprehension (REC) aims to localize the target objects specified by free-form natural language descriptions in images. While state-of-the-art methods achieve impressive performance, they perform a dense perception of images, which incorporates redundant visual regions unrelated to linguistic queries, leading to additional computational overhead. This inspires us to explore a…
▽ More
Referring Expression Comprehension (REC) aims to localize the target objects specified by free-form natural language descriptions in images. While state-of-the-art methods achieve impressive performance, they perform a dense perception of images, which incorporates redundant visual regions unrelated to linguistic queries, leading to additional computational overhead. This inspires us to explore a question: can we eliminate linguistic-irrelevant redundant visual regions to improve the efficiency of the model? Existing relevant methods primarily focus on fundamental visual tasks, with limited exploration in vision-language fields. To address this, we propose a coarse-to-fine iterative perception framework, called ScanFormer. It can iteratively exploit the image scale pyramid to extract linguistic-relevant visual patches from top to bottom. In each iteration, irrelevant patches are discarded by our designed informativeness prediction. Furthermore, we propose a patch selection strategy for discarded patches to accelerate inference. Experiments on widely used datasets, namely RefCOCO, RefCOCO+, RefCOCOg, and ReferItGame, verify the effectiveness of our method, which can strike a balance between accuracy and efficiency.
△ Less
Submitted 25 June, 2024;
originally announced June 2024.
-
Towards Synchronous Memorizability and Generalizability with Site-Modulated Diffusion Replay for Cross-Site Continual Segmentation
Authors:
Dunyuan Xu,
Xi Wang,
**gyang Zhang,
Pheng-Ann Heng
Abstract:
The ability to learn sequentially from different data sites is crucial for a deep network in solving practical medical image diagnosis problems due to privacy restrictions and storage limitations. However, adapting on incoming site leads to catastrophic forgetting on past sites and decreases generalizablity on unseen sites. Existing Continual Learning (CL) and Domain Generalization (DG) methods ha…
▽ More
The ability to learn sequentially from different data sites is crucial for a deep network in solving practical medical image diagnosis problems due to privacy restrictions and storage limitations. However, adapting on incoming site leads to catastrophic forgetting on past sites and decreases generalizablity on unseen sites. Existing Continual Learning (CL) and Domain Generalization (DG) methods have been proposed to solve these two challenges respectively, but none of them can address both simultaneously. Recognizing this limitation, this paper proposes a novel training paradigm, learning towards Synchronous Memorizability and Generalizability (SMG-Learning). To achieve this, we create the orientational gradient alignment to ensure memorizability on previous sites, and arbitrary gradient alignment to enhance generalizability on unseen sites. This approach is named as Parallel Gradient Alignment (PGA). Furthermore, we approximate the PGA as dual meta-objectives using the first-order Taylor expansion to reduce computational cost of aligning gradients. Considering that performing gradient alignments, especially for previous sites, is not feasible due to the privacy constraints, we design a Site-Modulated Diffusion (SMD) model to generate images with site-specific learnable prompts, replaying images have similar data distributions as previous sites. We evaluate our method on two medical image segmentation tasks, where data from different sites arrive sequentially. Experimental results show that our method efficiently enhances both memorizability and generalizablity better than other state-of-the-art methods, delivering satisfactory performance across all sites. Our code will be available at: https://github.com/dyxu-cuhkcse/SMG-Learning.
△ Less
Submitted 25 June, 2024;
originally announced June 2024.
-
HGTDP-DTA: Hybrid Graph-Transformer with Dynamic Prompt for Drug-Target Binding Affinity Prediction
Authors:
Xi Xiao,
Wentao Wang,
Jiacheng Xie,
Li**g Zhu,
Gaofei Chen,
Zhengji Li,
Tianyang Wang,
Min Xu
Abstract:
Drug target binding affinity (DTA) is a key criterion for drug screening. Existing experimental methods are time-consuming and rely on limited structural and domain information. While learning-based methods can model sequence and structural information, they struggle to integrate contextual data and often lack comprehensive modeling of drug-target interactions. In this study, we propose a novel DT…
▽ More
Drug target binding affinity (DTA) is a key criterion for drug screening. Existing experimental methods are time-consuming and rely on limited structural and domain information. While learning-based methods can model sequence and structural information, they struggle to integrate contextual data and often lack comprehensive modeling of drug-target interactions. In this study, we propose a novel DTA prediction method, termed HGTDP-DTA, which utilizes dynamic prompts within a hybrid Graph-Transformer framework. Our method generates context-specific prompts for each drug-target pair, enhancing the model's ability to capture unique interactions. The introduction of prompt tuning further optimizes the prediction process by filtering out irrelevant noise and emphasizing task-relevant information, dynamically adjusting the input features of the molecular graph. The proposed hybrid Graph-Transformer architecture combines structural information from Graph Convolutional Networks (GCNs) with sequence information captured by Transformers, facilitating the interaction between global and local information. Additionally, we adopted the multi-view feature fusion method to project molecular graph views and affinity subgraph views into a common feature space, effectively combining structural and contextual information. Experiments on two widely used public datasets, Davis and KIBA, show that HGTDP-DTA outperforms state-of-the-art DTA prediction methods in both prediction performance and generalization ability.
△ Less
Submitted 25 June, 2024;
originally announced June 2024.
-
Phonon Heat Transport and Anisotropic Tuning of Quantum Fluctuations in a Frustrated Honeycomb Magnet
Authors:
Haoran Fan,
Yue Chen,
Yuchen Gu,
Yuan Li,
Xi Lin
Abstract:
Honeycomb cobalt oxides containing 3$\it{d}$ Co$^{2+}$ ions might realize frustrated magnetism and novel quantum phases. Among candidate materials, Na$_{3}$Co$_{2}$SbO$_{6}$ stands out for its distorted honeycomb lattice and significant in-plane anisotropy, motivating vector-field tuning inside the honeycomb plane. Here we use thermal transport down to the mK regime to study twin-free crystals of…
▽ More
Honeycomb cobalt oxides containing 3$\it{d}$ Co$^{2+}$ ions might realize frustrated magnetism and novel quantum phases. Among candidate materials, Na$_{3}$Co$_{2}$SbO$_{6}$ stands out for its distorted honeycomb lattice and significant in-plane anisotropy, motivating vector-field tuning inside the honeycomb plane. Here we use thermal transport down to the mK regime to study twin-free crystals of Na$_{3}$Co$_{2}$SbO$_{6}$ subject to in-plane vector fields. We find that the thermal conductivity $κ$ never exceeds the heat-transport capability of phonons, rendering its suppression primarily due to phonon scattering off magnetic excitations and/or domain boundaries. The system's field-driven quantum criticality manifests itself as an abundance of magnetic fluctuations hindering the heat transport, which further depends on the field direction in an intriguing manner.
△ Less
Submitted 25 June, 2024;
originally announced June 2024.
-
Robustly Optimized Deep Feature Decoupling Network for Fatty Liver Diseases Detection
Authors:
Peng Huang,
Shu Hu,
Bo Peng,
Jiashu Zhang,
Xi Wu,
Xin Wang
Abstract:
Current medical image classification efforts mainly aim for higher average performance, often neglecting the balance between different classes. This can lead to significant differences in recognition accuracy between classes and obvious recognition weaknesses. Without the support of massive data, deep learning faces challenges in fine-grained classification of fatty liver. In this paper, we propos…
▽ More
Current medical image classification efforts mainly aim for higher average performance, often neglecting the balance between different classes. This can lead to significant differences in recognition accuracy between classes and obvious recognition weaknesses. Without the support of massive data, deep learning faces challenges in fine-grained classification of fatty liver. In this paper, we propose an innovative deep learning framework that combines feature decoupling and adaptive adversarial training. Firstly, we employ two iteratively compressed decouplers to supervised decouple common features and specific features related to fatty liver in abdominal ultrasound images. Subsequently, the decoupled features are concatenated with the original image after transforming the color space and are fed into the classifier. During adversarial training, we adaptively adjust the perturbation and balance the adversarial strength by the accuracy of each class. The model will eliminate recognition weaknesses by correctly classifying adversarial samples, thus improving recognition robustness. Finally, the accuracy of our method improved by 4.16%, achieving 82.95%. As demonstrated by extensive experiments, our method is a generalized learning framework that can be directly used to eliminate the recognition weaknesses of any classifier while improving its average performance. Code is available at https://github.com/HP-ML/MICCAI2024.
△ Less
Submitted 25 June, 2024;
originally announced June 2024.
-
Shortcut to adiabaticity improvement of STIRAP based qubit rotation
Authors:
Khayla Black,
Xi Chen,
Tim Byrnes
Abstract:
Robust quantum control is essential for the development of quantum computers, which rely on precise manipulation of qubits. One form of quantum control is stimulated Raman adiabatic passage (STIRAP), which ordinarily is a state transfer protocol but was extended by Kis and Renzoni (Phys. Rev. A 65, 032318 (2002)) to perform qubit rotations. Shortcut methods to adiabaticity for STIRAP have been sho…
▽ More
Robust quantum control is essential for the development of quantum computers, which rely on precise manipulation of qubits. One form of quantum control is stimulated Raman adiabatic passage (STIRAP), which ordinarily is a state transfer protocol but was extended by Kis and Renzoni (Phys. Rev. A 65, 032318 (2002)) to perform qubit rotations. Shortcut methods to adiabaticity for STIRAP have been shown to speed up adiabatic processes, beyond the adiabatic criterion, with high fidelity. Here, we apply shortcut to adiabaticity methods to the STIRAP qubit rotation scheme to improve the performance of quantum logic gates. The scheme can be implemented via direct connections between ground states in a 4-level $Λ$ system or effective connections in a 5-level $Λ$ system with modified pulses that implement transitionless quantum driving via the addition of a counterdiabatic driving term. We show that the extended shortcut to adiabaticity method serves to improve the fidelity of qubit rotations in the diabatic regime.
△ Less
Submitted 25 June, 2024;
originally announced June 2024.
-
MindSpore Quantum: A User-Friendly, High-Performance, and AI-Compatible Quantum Computing Framework
Authors:
Xusheng Xu,
Jiangyu Cui,
Zidong Cui,
Runhong He,
Qingyu Li,
Xiaowei Li,
Yanling Lin,
Jiale Liu,
Wuxin Liu,
Jiale Lu,
Maolin Luo,
Chufan Lyu,
Shijie Pan,
Mosharev Pavel,
Runqiu Shu,
Jialiang Tang,
Ruoqian Xu,
Shu Xu,
Kang Yang,
Fan Yu,
Qingguo Zeng,
Haiying Zhao,
Qiang Zheng,
Junyuan Zhou,
Xu Zhou
, et al. (14 additional authors not shown)
Abstract:
We introduce MindSpore Quantum, a pioneering hybrid quantum-classical framework with a primary focus on the design and implementation of noisy intermediate-scale quantum (NISQ) algorithms. Leveraging the robust support of MindSpore, an advanced open-source deep learning training/inference framework, MindSpore Quantum exhibits exceptional efficiency in the design and training of variational quantum…
▽ More
We introduce MindSpore Quantum, a pioneering hybrid quantum-classical framework with a primary focus on the design and implementation of noisy intermediate-scale quantum (NISQ) algorithms. Leveraging the robust support of MindSpore, an advanced open-source deep learning training/inference framework, MindSpore Quantum exhibits exceptional efficiency in the design and training of variational quantum algorithms on both CPU and GPU platforms, delivering remarkable performance. Furthermore, this framework places a strong emphasis on enhancing the operational efficiency of quantum algorithms when executed on real quantum hardware. This encompasses the development of algorithms for quantum circuit compilation and qubit map**, crucial components for achieving optimal performance on quantum processors. In addition to the core framework, we introduce QuPack, a meticulously crafted quantum computing acceleration engine. QuPack significantly accelerates the simulation speed of MindSpore Quantum, particularly in variational quantum eigensolver (VQE), quantum approximate optimization algorithm (QAOA), and tensor network simulations, providing astonishing speed. This combination of cutting-edge technologies empowers researchers and practitioners to explore the frontiers of quantum computing with unprecedented efficiency and performance.
△ Less
Submitted 27 June, 2024; v1 submitted 24 June, 2024;
originally announced June 2024.
-
The GPT-WritingPrompts Dataset: A Comparative Analysis of Character Portrayal in Short Stories
Authors:
Xi Yu Huang,
Krishnapriya Vishnubhotla,
Frank Rudzicz
Abstract:
The improved generative capabilities of large language models have made them a powerful tool for creative writing and storytelling. It is therefore important to quantitatively understand the nature of generated stories, and how they differ from human storytelling. We augment the Reddit WritingPrompts dataset with short stories generated by GPT-3.5, given the same prompts. We quantify and compare t…
▽ More
The improved generative capabilities of large language models have made them a powerful tool for creative writing and storytelling. It is therefore important to quantitatively understand the nature of generated stories, and how they differ from human storytelling. We augment the Reddit WritingPrompts dataset with short stories generated by GPT-3.5, given the same prompts. We quantify and compare the emotional and descriptive features of storytelling from both generative processes, human and machine, along a set of six dimensions. We find that generated stories differ significantly from human stories along all six dimensions, and that human and machine generations display similar biases when grouped according to the narrative point-of-view and gender of the main protagonist. We release our dataset and code at https://github.com/KristinHuangg/gpt-writing-prompts.
△ Less
Submitted 24 June, 2024;
originally announced June 2024.
-
Dedicated beam position monitor pair for model-independent lattice characterization at NSLS-II
Authors:
Yongjun Li,
Kiman Ha,
Danny Padrazo,
Bernard Kosciuk,
Belkacem Bacha,
Michael Seegitz,
Robert Rainer,
Joseph Mead,
Xi Yang,
Yuke Tian,
Robert Todd,
Victor Smaluk,
Weixing Cheng
Abstract:
This paper reports recent lattice characterization results obtained at the National Synchrotron Light Source II (NSLS-II) storage ring, conducted without reliance on a lattice model. A pair of beam position monitors (BPMs) with bunch-by-bunch (B$\times$B) resolution, were recently installed in a section of the storage ring free of magnetic fields. The new BPM pair measured the beam, or bunch's tra…
▽ More
This paper reports recent lattice characterization results obtained at the National Synchrotron Light Source II (NSLS-II) storage ring, conducted without reliance on a lattice model. A pair of beam position monitors (BPMs) with bunch-by-bunch (B$\times$B) resolution, were recently installed in a section of the storage ring free of magnetic fields. The new BPM pair measured the beam, or bunch's transverse Poincaré map precisely after the beam was excited. Linear one-turn-matrices (OTM) were then derived, and from these, the 4-dimensional coupled Twiss parameters were extracted at the locations of the BPM pair. By normalizing beam oscillation amplitudes with the Twiss parameters, the global action-variables were obtained. These action-variables facilitated the measurement of the local Twiss parameters observed by other BPMs independent on lattice model. This method is general, and particularly useful in certain scenarios such as a round beam mode in a diffraction-limited light source ring. We applied it to assess both weakly and strongly coupled lattices at the NSLS-II ring. Through analysis of the strongly coupled lattice, the quadrupole tilt errors were estimated to be less than 400 \siμrad. Utilizing the BPMs' B$\times$B resolution, for the first time we observed the variations of the linear lattice along a long bunch-train.
△ Less
Submitted 24 June, 2024;
originally announced June 2024.
-
An Active Search Strategy with Multiple Unmanned Aerial Systems for Multiple Targets
Authors:
Chuanxiang Gao,
Xinyi Wang,
Xi Chen,
Ben M. Chen
Abstract:
The challenge of efficient target searching in vast natural environments has driven the need for advanced multi-UAV active search strategies. This paper introduces a novel method in which global and local information is adeptly merged to avoid issues such as myopia and redundant back-and-forth movements. In addition, a trajectory generation method is used to ensure the search pattern within contin…
▽ More
The challenge of efficient target searching in vast natural environments has driven the need for advanced multi-UAV active search strategies. This paper introduces a novel method in which global and local information is adeptly merged to avoid issues such as myopia and redundant back-and-forth movements. In addition, a trajectory generation method is used to ensure the search pattern within continuous space. To further optimize multi-agent cooperation, the Voronoi partition technique is employed, ensuring a reduction in repetitive flight patterns and making the control of multiple agents in a decentralized way. Through a series of experiments, the evaluation and comparison results demonstrate the efficiency of our approach in various environments. The primary application of this innovative approach is demonstrated in the search for horseshoe crabs within their wild habitats, showcasing its potential to revolutionize ecological survey and conservation efforts.
△ Less
Submitted 24 June, 2024;
originally announced June 2024.
-
MIRReS: Multi-bounce Inverse Rendering using Reservoir Sampling
Authors:
Yuxin Dai,
Qi Wang,
**gsen Zhu,
Dianbing Xi,
Yuchi Huo,
Chen Qian,
Ying He
Abstract:
We present MIRReS, a novel two-stage inverse rendering framework that jointly reconstructs and optimizes the explicit geometry, material, and lighting from multi-view images. Unlike previous methods that rely on implicit irradiance fields or simplified path tracing algorithms, our method extracts an explicit geometry (triangular mesh) in stage one, and introduces a more realistic physically-based…
▽ More
We present MIRReS, a novel two-stage inverse rendering framework that jointly reconstructs and optimizes the explicit geometry, material, and lighting from multi-view images. Unlike previous methods that rely on implicit irradiance fields or simplified path tracing algorithms, our method extracts an explicit geometry (triangular mesh) in stage one, and introduces a more realistic physically-based inverse rendering model that utilizes multi-bounce path tracing and Monte Carlo integration. By leveraging multi-bounce path tracing, our method effectively estimates indirect illumination, including self-shadowing and internal reflections, which improves the intrinsic decomposition of shape, material, and lighting. Moreover, we incorporate reservoir sampling into our framework to address the noise in Monte Carlo integration, enhancing convergence and facilitating gradient-based optimization with low sample counts. Through qualitative and quantitative evaluation of several scenarios, especially in challenging scenarios with complex shadows, we demonstrate that our method achieves state-of-the-art performance on decomposition results. Additionally, our optimized explicit geometry enables applications such as scene editing, relighting, and material editing with modern graphics engines or CAD software. The source code is available at https://brabbitdousha.github.io/MIRReS/
△ Less
Submitted 24 June, 2024; v1 submitted 24 June, 2024;
originally announced June 2024.
-
Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging
Authors:
Deyuan Liu,
Zhanyue Qin,
Hairu Wang,
Zhao Yang,
Zecheng Wang,
Fangying Rong,
Qingbin Liu,
Yanchao Hao,
Xi Chen,
Cunhang Fan,
Zhao Lv,
Zhiying Tu,
Dianhui Chu,
Bo Li,
Dianbo Sui
Abstract:
While large language models (LLMs) excel in many domains, their complexity and scale challenge deployment in resource-limited environments. Current compression techniques, such as parameter pruning, often fail to effectively utilize the knowledge from pruned parameters. To address these challenges, we propose Manifold-Based Knowledge Alignment and Layer Merging Compression (MKA), a novel approach…
▽ More
While large language models (LLMs) excel in many domains, their complexity and scale challenge deployment in resource-limited environments. Current compression techniques, such as parameter pruning, often fail to effectively utilize the knowledge from pruned parameters. To address these challenges, we propose Manifold-Based Knowledge Alignment and Layer Merging Compression (MKA), a novel approach that uses manifold learning and the Normalized Pairwise Information Bottleneck (NPIB) measure to merge similar layers, reducing model size while preserving essential performance. We evaluate MKA on multiple benchmark datasets and various LLMs. Our findings show that MKA not only preserves model performance but also achieves substantial compression ratios, outperforming traditional pruning methods. Moreover, when coupled with quantization, MKA delivers even greater compression. Specifically, on the MMLU dataset using the Llama3-8B model, MKA achieves a compression ratio of 43.75% with a minimal performance decrease of only 2.82\%. The proposed MKA method offers a resource-efficient and performance-preserving model compression technique for LLMs.
△ Less
Submitted 24 June, 2024;
originally announced June 2024.
-
OpticGAI: Generative AI-aided Deep Reinforcement Learning for Optical Networks Optimization
Authors:
Siyuan Li,
Xi Lin,
Yaju Liu,
Gaolei Li,
Jianhua Li
Abstract:
Deep Reinforcement Learning (DRL) is regarded as a promising tool for optical network optimization. However, the flexibility and efficiency of current DRL-based solutions for optical network optimization require further improvement. Currently, generative models have showcased their significant performance advantages across various domains. In this paper, we introduce OpticGAI, the AI-generated pol…
▽ More
Deep Reinforcement Learning (DRL) is regarded as a promising tool for optical network optimization. However, the flexibility and efficiency of current DRL-based solutions for optical network optimization require further improvement. Currently, generative models have showcased their significant performance advantages across various domains. In this paper, we introduce OpticGAI, the AI-generated policy design paradigm for optical networks. In detail, it is implemented as a novel DRL framework that utilizes generative models to learn the optimal policy network. Furthermore, we assess the performance of OpticGAI on two NP-hard optical network problems, Routing and Wavelength Assignment (RWA) and dynamic Routing, Modulation, and Spectrum Allocation (RMSA), to show the feasibility of the AI-generated policy paradigm. Simulation results have shown that OpticGAI achieves the highest reward and the lowest blocking rate of both RWA and RMSA problems. OpticGAI poses a promising direction for future research on generative AI-enhanced flexible optical network optimization.
△ Less
Submitted 22 June, 2024;
originally announced June 2024.
-
ProBE: Proportioning Privacy Budget for Complex Exploratory Decision Support
Authors:
Nada Lahjouji,
Sameera Ghayyur,
Xi He,
Sharad Mehrotra
Abstract:
This paper studies privacy in the context of complex decision support queries composed of multiple conditions on different aggregate statistics combined using disjunction and conjunction operators. Utility requirements for such queries necessitate the need for private mechanisms that guarantee a bound on the false negative and false positive errors. This paper formally defines complex decision sup…
▽ More
This paper studies privacy in the context of complex decision support queries composed of multiple conditions on different aggregate statistics combined using disjunction and conjunction operators. Utility requirements for such queries necessitate the need for private mechanisms that guarantee a bound on the false negative and false positive errors. This paper formally defines complex decision support queries and their accuracy requirements, and provides algorithms that proportion the existing budget to optimally minimize privacy loss while supporting a bounded guarantee on the accuracy. Our experimental results on multiple real-life datasets show that our algorithms successfully maintain such utility guarantees, while also minimizing privacy loss.
△ Less
Submitted 21 June, 2024;
originally announced June 2024.
-
SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation
Authors:
Zeyao Ma,
Bohan Zhang,
**g Zhang,
Jifan Yu,
Xiaokang Zhang,
Xiaohan Zhang,
Sijia Luo,
Xi Wang,
Jie Tang
Abstract:
We introduce SpreadsheetBench, a challenging spreadsheet manipulation benchmark exclusively derived from real-world scenarios, designed to immerse current large language models (LLMs) in the actual workflow of spreadsheet users. Unlike existing benchmarks that rely on synthesized queries and simplified spreadsheet files, SpreadsheetBench is built from 912 real questions gathered from online Excel…
▽ More
We introduce SpreadsheetBench, a challenging spreadsheet manipulation benchmark exclusively derived from real-world scenarios, designed to immerse current large language models (LLMs) in the actual workflow of spreadsheet users. Unlike existing benchmarks that rely on synthesized queries and simplified spreadsheet files, SpreadsheetBench is built from 912 real questions gathered from online Excel forums, which reflect the intricate needs of users. The associated spreadsheets from the forums contain a variety of tabular data such as multiple tables, non-standard relational tables, and abundant non-textual elements. Furthermore, we propose a more reliable evaluation metric akin to online judge platforms, where multiple spreadsheet files are created as test cases for each instruction, ensuring the evaluation of robust solutions capable of handling spreadsheets with varying values. Our comprehensive evaluation of various LLMs under both single-round and multi-round inference settings reveals a substantial gap between the state-of-the-art (SOTA) models and human performance, highlighting the benchmark's difficulty.
△ Less
Submitted 21 June, 2024;
originally announced June 2024.
-
The Fire Thief Is Also the Keeper: Balancing Usability and Privacy in Prompts
Authors:
Zhili Shen,
Zihang Xi,
Ying He,
Wei Tong,
**gyu Hua,
Sheng Zhong
Abstract:
The rapid adoption of online chatbots represents a significant advancement in artificial intelligence. However, this convenience brings considerable privacy concerns, as prompts can inadvertently contain sensitive information exposed to large language models (LLMs). Limited by high computational costs, reduced task usability, and excessive system modifications, previous works based on local deploy…
▽ More
The rapid adoption of online chatbots represents a significant advancement in artificial intelligence. However, this convenience brings considerable privacy concerns, as prompts can inadvertently contain sensitive information exposed to large language models (LLMs). Limited by high computational costs, reduced task usability, and excessive system modifications, previous works based on local deployment, embedding perturbation, and homomorphic encryption are inapplicable to online prompt-based LLM applications.
To address these issues, this paper introduces Prompt Privacy Sanitizer (i.e., ProSan), an end-to-end prompt privacy protection framework that can produce anonymized prompts with contextual privacy removed while maintaining task usability and human readability. It can also be seamlessly integrated into the online LLM service pipeline. To achieve high usability and dynamic anonymity, ProSan flexibly adjusts its protection targets and strength based on the importance of the words and the privacy leakage risk of the prompts. Additionally, ProSan is capable of adapting to diverse computational resource conditions, ensuring privacy protection even for mobile devices with limited computing power. Our experiments demonstrate that ProSan effectively removes private information across various tasks, including question answering, text summarization, and code generation, with minimal reduction in task performance.
△ Less
Submitted 20 June, 2024;
originally announced June 2024.
-
MCAD: Multi-modal Conditioned Adversarial Diffusion Model for High-Quality PET Image Reconstruction
Authors:
Jiaqi Cui,
Xinyi Zeng,
Pinxian Zeng,
Bo Liu,
Xi Wu,
Jiliu Zhou,
Yan Wang
Abstract:
Radiation hazards associated with standard-dose positron emission tomography (SPET) images remain a concern, whereas the quality of low-dose PET (LPET) images fails to meet clinical requirements. Therefore, there is great interest in reconstructing SPET images from LPET images. However, prior studies focus solely on image data, neglecting vital complementary information from other modalities, e.g.…
▽ More
Radiation hazards associated with standard-dose positron emission tomography (SPET) images remain a concern, whereas the quality of low-dose PET (LPET) images fails to meet clinical requirements. Therefore, there is great interest in reconstructing SPET images from LPET images. However, prior studies focus solely on image data, neglecting vital complementary information from other modalities, e.g., patients' clinical tabular, resulting in compromised reconstruction with limited diagnostic utility. Moreover, they often overlook the semantic consistency between real SPET and reconstructed images, leading to distorted semantic contexts. To tackle these problems, we propose a novel Multi-modal Conditioned Adversarial Diffusion model (MCAD) to reconstruct SPET images from multi-modal inputs, including LPET images and clinical tabular. Specifically, our MCAD incorporates a Multi-modal conditional Encoder (Mc-Encoder) to extract multi-modal features, followed by a conditional diffusion process to blend noise with multi-modal features and gradually map blended features to the target SPET images. To balance multi-modal inputs, the Mc-Encoder embeds Optimal Multi-modal Transport co-Attention (OMTA) to narrow the heterogeneity gap between image and tabular while capturing their interactions, providing sufficient guidance for reconstruction. In addition, to mitigate semantic distortions, we introduce the Multi-Modal Masked Text Reconstruction (M3TRec), which leverages semantic knowledge extracted from denoised PET images to restore the masked clinical tabular, thereby compelling the network to maintain accurate semantics during reconstruction. To expedite the diffusion process, we further introduce an adversarial diffusive network with a reduced number of diffusion steps. Experiments show that our method achieves the state-of-the-art performance both qualitatively and quantitatively.
△ Less
Submitted 18 June, 2024;
originally announced June 2024.
-
Text-aware Speech Separation for Multi-talker Keyword Spotting
Authors:
Haoyu Li,
Baochen Yang,
Yu Xi,
Linfeng Yu,
Tian Tan,
Hao Li,
Kai Yu
Abstract:
For noisy environments, ensuring the robustness of keyword spotting (KWS) systems is essential. While much research has focused on noisy KWS, less attention has been paid to multi-talker mixed speech scenarios. Unlike the usual cocktail party problem where multi-talker speech is separated using speaker clues, the key challenge here is to extract the target speech for KWS based on text clues. To ad…
▽ More
For noisy environments, ensuring the robustness of keyword spotting (KWS) systems is essential. While much research has focused on noisy KWS, less attention has been paid to multi-talker mixed speech scenarios. Unlike the usual cocktail party problem where multi-talker speech is separated using speaker clues, the key challenge here is to extract the target speech for KWS based on text clues. To address it, this paper proposes a novel Text-aware Permutation Determinization Training method for multi-talker KWS with a clue-based Speech Separation front-end (TPDT-SS). Our research highlights the critical role of SS front-ends and shows that incorporating keyword-specific clues into these models can greatly enhance the effectiveness. TPDT-SS shows remarkable success in addressing permutation problems in mixed keyword speech, thereby greatly boosting the performance of the backend. Additionally, fine-tuning our system on unseen mixed speech results in further performance improvement.
△ Less
Submitted 18 June, 2024;
originally announced June 2024.
-
When Box Meets Graph Neural Network in Tag-aware Recommendation
Authors:
Fake Lin,
Ziwei Zhao,
Xi Zhu,
Da Zhang,
Shitian Shen,
Xueying Li,
Tong Xu,
Suojuan Zhang,
Enhong Chen
Abstract:
Last year has witnessed the re-flourishment of tag-aware recommender systems supported by the LLM-enriched tags. Unfortunately, though large efforts have been made, current solutions may fail to describe the diversity and uncertainty inherent in user preferences with only tag-driven profiles. Recently, with the development of geometry-based techniques, e.g., box embedding, diversity of user prefer…
▽ More
Last year has witnessed the re-flourishment of tag-aware recommender systems supported by the LLM-enriched tags. Unfortunately, though large efforts have been made, current solutions may fail to describe the diversity and uncertainty inherent in user preferences with only tag-driven profiles. Recently, with the development of geometry-based techniques, e.g., box embedding, diversity of user preferences now could be fully modeled as the range within a box in high dimension space. However, defect still exists as these approaches are incapable of capturing high-order neighbor signals, i.e., semantic-rich multi-hop relations within the user-tag-item tripartite graph, which severely limits the effectiveness of user modeling. To deal with this challenge, in this paper, we propose a novel algorithm, called BoxGNN, to perform the message aggregation via combination of logical operations, thereby incorporating high-order signals. Specifically, we first embed users, items, and tags as hyper-boxes rather than simple points in the representation space, and define two logical operations to facilitate the subsequent process. Next, we perform the message aggregation mechanism via the combination of logical operations, to obtain the corresponding high-order box representations. Finally, we adopt a volume-based learning objective with Gumbel smoothing techniques to refine the representation of boxes. Extensive experiments on two publicly available datasets and one LLM-enhanced e-commerce dataset have validated the superiority of BoxGNN compared with various state-of-the-art baselines. The code is released online
△ Less
Submitted 17 June, 2024;
originally announced June 2024.
-
Job-SDF: A Multi-Granularity Dataset for Job Skill Demand Forecasting and Benchmarking
Authors:
Xi Chen,
Chuan Qin,
Chuyu Fang,
Chao Wang,
Chen Zhu,
Fuzhen Zhuang,
Hengshu Zhu,
Hui Xiong
Abstract:
In a rapidly evolving job market, skill demand forecasting is crucial as it enables policymakers and businesses to anticipate and adapt to changes, ensuring that workforce skills align with market needs, thereby enhancing productivity and competitiveness. Additionally, by identifying emerging skill requirements, it directs individuals towards relevant training and education opportunities, promotin…
▽ More
In a rapidly evolving job market, skill demand forecasting is crucial as it enables policymakers and businesses to anticipate and adapt to changes, ensuring that workforce skills align with market needs, thereby enhancing productivity and competitiveness. Additionally, by identifying emerging skill requirements, it directs individuals towards relevant training and education opportunities, promoting continuous self-learning and development. However, the absence of comprehensive datasets presents a significant challenge, impeding research and the advancement of this field. To bridge this gap, we present Job-SDF, a dataset designed to train and benchmark job-skill demand forecasting models. Based on 10.35 million public job advertisements collected from major online recruitment platforms in China between 2021 and 2023, this dataset encompasses monthly recruitment demand for 2,324 types of skills across 521 companies. Our dataset uniquely enables evaluating skill demand forecasting models at various granularities, including occupation, company, and regional levels. We benchmark a range of models on this dataset, evaluating their performance in standard scenarios, in predictions focused on lower value ranges, and in the presence of structural breaks, providing new insights for further research. Our code and dataset are publicly accessible via the https://github.com/Job-SDF/benchmark.
△ Less
Submitted 19 June, 2024; v1 submitted 17 June, 2024;
originally announced June 2024.
-
Towards Adaptive Neighborhood for Advancing Temporal Interaction Graph Modeling
Authors:
Siwei Zhang,
Xi Chen,
Yun Xiong,
Xixi Wu,
Yao Zhang,
Yongrui Fu,
Yinglong Zhao,
Jiawei Zhang
Abstract:
Temporal Graph Networks (TGNs) have demonstrated their remarkable performance in modeling temporal interaction graphs. These works can generate temporal node representations by encoding the surrounding neighborhoods for the target node. However, an inherent limitation of existing TGNs is their reliance on fixed, hand-crafted rules for neighborhood encoding, overlooking the necessity for an adaptiv…
▽ More
Temporal Graph Networks (TGNs) have demonstrated their remarkable performance in modeling temporal interaction graphs. These works can generate temporal node representations by encoding the surrounding neighborhoods for the target node. However, an inherent limitation of existing TGNs is their reliance on fixed, hand-crafted rules for neighborhood encoding, overlooking the necessity for an adaptive and learnable neighborhood that can accommodate both personalization and temporal evolution across different timestamps. In this paper, we aim to enhance existing TGNs by introducing an adaptive neighborhood encoding mechanism. We present SEAN, a flexible plug-and-play model that can be seamlessly integrated with existing TGNs, effectively boosting their performance. To achieve this, we decompose the adaptive neighborhood encoding process into two phases: (i) representative neighbor selection, and (ii) temporal-aware neighborhood information aggregation. Specifically, we propose the Representative Neighbor Selector component, which automatically pinpoints the most important neighbors for the target node. It offers a tailored understanding of each node's unique surrounding context, facilitating personalization. Subsequently, we propose a Temporal-aware Aggregator, which synthesizes neighborhood aggregation by selectively determining the utilization of aggregation routes and decaying the outdated information, allowing our model to adaptively leverage both the contextually significant and current information during aggregation. We conduct extensive experiments by integrating SEAN into three representative TGNs, evaluating their performance on four public datasets and one financial benchmark dataset introduced in this paper. The results demonstrate that SEAN consistently leads to performance improvements across all models, achieving SOTA performance and exceptional robustness.
△ Less
Submitted 14 June, 2024;
originally announced June 2024.
-
HoLLMwood: Unleashing the Creativity of Large Language Models in Screenwriting via Role Playing
Authors:
**g Chen,
Xinyu Zhu,
Cheng Yang,
Chufan Shi,
Yadong Xi,
Yuxiang Zhang,
Junjie Wang,
Jiashu Pu,
Rongsheng Zhang,
Yujiu Yang,
Tian Feng
Abstract:
Generative AI has demonstrated unprecedented creativity in the field of computer vision, yet such phenomena have not been observed in natural language processing. In particular, large language models (LLMs) can hardly produce written works at the level of human experts due to the extremely high complexity of literature writing. In this paper, we present HoLLMwood, an automated framework for unleas…
▽ More
Generative AI has demonstrated unprecedented creativity in the field of computer vision, yet such phenomena have not been observed in natural language processing. In particular, large language models (LLMs) can hardly produce written works at the level of human experts due to the extremely high complexity of literature writing. In this paper, we present HoLLMwood, an automated framework for unleashing the creativity of LLMs and exploring their potential in screenwriting, which is a highly demanding task. Mimicking the human creative process, we assign LLMs to different roles involved in the real-world scenario. In addition to the common practice of treating LLMs as ${Writer}$, we also apply LLMs as ${Editor}$, who is responsible for providing feedback and revision advice to ${Writer}$. Besides, to enrich the characters and deepen the plots, we introduce a role-playing mechanism and adopt LLMs as ${Actors}$ that can communicate and interact with each other. Evaluations on automatically generated screenplays show that HoLLMwood substantially outperforms strong baselines in terms of coherence, relevance, interestingness and overall quality.
△ Less
Submitted 17 June, 2024;
originally announced June 2024.
-
Hermitian-Einstein equations on noncompact manifolds
Authors:
Di Wu,
Xi Zhang
Abstract:
This paper first investigates solvability of Hermitian-Einstein equation on a Hermitian holomorphic vector bundle on the complement of an arbitrary closed subset in a compact complex manifold. The uniqueness of Hermitian-Einstein metrics on a Zariski open subset in a compact Kähler manifold was only figured out by Takuro Mochizuki recently, the second part of this paper gives an affirmative answer…
▽ More
This paper first investigates solvability of Hermitian-Einstein equation on a Hermitian holomorphic vector bundle on the complement of an arbitrary closed subset in a compact complex manifold. The uniqueness of Hermitian-Einstein metrics on a Zariski open subset in a compact Kähler manifold was only figured out by Takuro Mochizuki recently, the second part of this paper gives an affirmative answer to a question proposed by Takuro Mochizuki and based on which it leads to an alternative approach to the unique issue. We also prove stability from solvability of Hermitian-Einstein equation, which together with the classical existence result of Carlos Simpson in particular establish a Kobayashi-Hitchin bijective correspondence. The argument is also effective for uniqueness result and Kobayashi-Hitchin bijective correspondence on $\mathbb{C}$, as well as on non-Kähler and semi-stable contexts.
△ Less
Submitted 17 June, 2024;
originally announced June 2024.
-
From Pixels to Progress: Generating Road Network from Satellite Imagery for Socioeconomic Insights in Impoverished Areas
Authors:
Yanxin Xi,
Yu Liu,
Zhicheng Liu,
Sasu Tarkoma,
Pan Hui,
Yong Li
Abstract:
The Sustainable Development Goals (SDGs) aim to resolve societal challenges, such as eradicating poverty and improving the lives of vulnerable populations in impoverished areas. Those areas rely on road infrastructure construction to promote accessibility and economic development. Although publicly available data like OpenStreetMap is available to monitor road status, data completeness in impoveri…
▽ More
The Sustainable Development Goals (SDGs) aim to resolve societal challenges, such as eradicating poverty and improving the lives of vulnerable populations in impoverished areas. Those areas rely on road infrastructure construction to promote accessibility and economic development. Although publicly available data like OpenStreetMap is available to monitor road status, data completeness in impoverished areas is limited. Meanwhile, the development of deep learning techniques and satellite imagery shows excellent potential for earth monitoring. To tackle the challenge of road network assessment in impoverished areas, we develop a systematic road extraction framework combining an encoder-decoder architecture and morphological operations on satellite imagery, offering an integrated workflow for interdisciplinary researchers. Extensive experiments of road network extraction on real-world data in impoverished regions achieve a 42.7% enhancement in the F1-score over the baseline methods and reconstruct about 80% of the actual roads. We also propose a comprehensive road network dataset covering approximately 794,178 km2 area and 17.048 million people in 382 impoverished counties in China. The generated dataset is further utilized to conduct socioeconomic analysis in impoverished counties, showing that road network construction positively impacts regional economic development. The technical appendix, code, and generated dataset can be found at https://github.com/tsinghua-fib-lab/Road_network_extraction_impoverished_counties.
△ Less
Submitted 17 June, 2024;
originally announced June 2024.
-
Toward Optimal LLM Alignments Using Two-Player Games
Authors:
Rui Zheng,
Hongyi Guo,
Zhihan Liu,
Xiaoying Zhang,
Yuanshun Yao,
Xiaojun Xu,
Zhaoran Wang,
Zhiheng Xi,
Tao Gui,
Qi Zhang,
Xuan**g Huang,
Hang Li,
Yang Liu
Abstract:
The standard Reinforcement Learning from Human Feedback (RLHF) framework primarily focuses on optimizing the performance of large language models using pre-collected prompts. However, collecting prompts that provide comprehensive coverage is both tedious and challenging, and often fails to include scenarios that LLMs need to improve on the most. In this paper, we investigate alignment through the…
▽ More
The standard Reinforcement Learning from Human Feedback (RLHF) framework primarily focuses on optimizing the performance of large language models using pre-collected prompts. However, collecting prompts that provide comprehensive coverage is both tedious and challenging, and often fails to include scenarios that LLMs need to improve on the most. In this paper, we investigate alignment through the lens of two-agent games, involving iterative interactions between an adversarial and a defensive agent. The adversarial agent's task at each step is to generate prompts that expose the weakness of the defensive agent. In return, the defensive agent seeks to improve its responses to these newly identified prompts it struggled with, based on feedback from the reward model. We theoretically demonstrate that this iterative reinforcement learning optimization converges to a Nash Equilibrium for the game induced by the agents. Experimental results in safety scenarios demonstrate that learning in such a competitive environment not only fully trains agents but also leads to policies with enhanced generalization capabilities for both adversarial and defensive agents.
△ Less
Submitted 16 June, 2024;
originally announced June 2024.
-
Technique Report of CVPR 2024 PBDL Challenges
Authors:
Ying Fu,
Yu Li,
Shaodi You,
Boxin Shi,
Jose Alvarez,
Coert van Gemeren,
Linwei Chen,
Yunhao Zou,
Zichun Wang,
Yichen Li,
Yuze Han,
Yingkai Zhang,
Jianan Wang,
Qinglin Liu,
Wei Yu,
Xiaoqian Lv,
Jianing Li,
Sheng** Zhang,
Xiangyang Ji,
Yuanpei Chen,
Yuhan Zhang,
Weihang Peng,
Liwen Zhang,
Zhe Xu,
Dingyong Gou
, et al. (77 additional authors not shown)
Abstract:
The intersection of physics-based vision and deep learning presents an exciting frontier for advancing computer vision technologies. By leveraging the principles of physics to inform and enhance deep learning models, we can develop more robust and accurate vision systems. Physics-based vision aims to invert the processes to recover scene properties such as shape, reflectance, light distribution, a…
▽ More
The intersection of physics-based vision and deep learning presents an exciting frontier for advancing computer vision technologies. By leveraging the principles of physics to inform and enhance deep learning models, we can develop more robust and accurate vision systems. Physics-based vision aims to invert the processes to recover scene properties such as shape, reflectance, light distribution, and medium properties from images. In recent years, deep learning has shown promising improvements for various vision tasks, and when combined with physics-based vision, these approaches can enhance the robustness and accuracy of vision systems. This technical report summarizes the outcomes of the Physics-Based Vision Meets Deep Learning (PBDL) 2024 challenge, held in CVPR 2024 workshop. The challenge consisted of eight tracks, focusing on Low-Light Enhancement and Detection as well as High Dynamic Range (HDR) Imaging. This report details the objectives, methodologies, and results of each track, highlighting the top-performing solutions and their innovative approaches.
△ Less
Submitted 27 June, 2024; v1 submitted 15 June, 2024;
originally announced June 2024.
-
SemanticMIM: Marring Masked Image Modeling with Semantics Compression for General Visual Representation
Authors:
Yike Yuan,
Huanzhang Dou,
Fengjun Guo,
Xi Li
Abstract:
This paper represents a neat yet effective framework, named SemanticMIM, to integrate the advantages of masked image modeling (MIM) and contrastive learning (CL) for general visual representation. We conduct a thorough comparative analysis between CL and MIM, revealing that their complementary advantages fundamentally stem from two distinct phases, i.e., compression and reconstruction. Specificall…
▽ More
This paper represents a neat yet effective framework, named SemanticMIM, to integrate the advantages of masked image modeling (MIM) and contrastive learning (CL) for general visual representation. We conduct a thorough comparative analysis between CL and MIM, revealing that their complementary advantages fundamentally stem from two distinct phases, i.e., compression and reconstruction. Specifically, SemanticMIM leverages a proxy architecture that customizes interaction between image and mask tokens, bridging these two phases to achieve general visual representation with the property of abundant semantic and positional awareness. Through extensive qualitative and quantitative evaluations, we demonstrate that SemanticMIM effectively amalgamates the benefits of CL and MIM, leading to significant enhancement of performance and feature linear separability. SemanticMIM also offers notable interpretability through attention response visualization. Codes are available at https://github.com/yyk-wew/SemanticMIM.
△ Less
Submitted 15 June, 2024;
originally announced June 2024.