Showing 1–50 of 166 results for author: Du, K

SINKT: A Structure-Aware Inductive Knowledge Tracing Model with Large Language Model

Authors: Lingyue Fu, Hao Guan, Kounianhua Du, Jianghao Lin, Wei Xia, Weinan Zhang, Ruiming Tang, Yasheng Wang, Yong Yu

Abstract: Knowledge Tracing (KT) aims to determine whether students will respond correctly to the next question, which is a crucial task in intelligent tutoring systems (ITS). In educational KT scenarios, transductive ID-based methods often face severe data sparsity and cold start problems, where interactions between individual students and questions are sparse, and new questions and concepts consistently a… ▽ More Knowledge Tracing (KT) aims to determine whether students will respond correctly to the next question, which is a crucial task in intelligent tutoring systems (ITS). In educational KT scenarios, transductive ID-based methods often face severe data sparsity and cold start problems, where interactions between individual students and questions are sparse, and new questions and concepts consistently arrive in the database. In addition, existing KT models only implicitly consider the correlation between concepts and questions, lacking direct modeling of the more complex relationships in the heterogeneous graph of concepts and questions. In this paper, we propose a Structure-aware Inductive Knowledge Tracing model with large language model (dubbed SINKT), which, for the first time, introduces large language models (LLMs) and realizes inductive knowledge tracing. Firstly, SINKT utilizes LLMs to introduce structural relationships between concepts and constructs a heterogeneous graph for concepts and questions. Secondly, by encoding concepts and questions with LLMs, SINKT incorporates semantic information to aid prediction. Finally, SINKT predicts the student's response to the target question by interacting with the student's knowledge state and the question representation. Experiments on four real-world datasets demonstrate that SINKT achieves state-of-the-art performance among 12 existing transductive KT models. Additionally, we explore the performance of SINKT on the inductive KT task and provide insights into various modules. △ Less

Submitted 1 July, 2024; originally announced July 2024.

ELCoRec: Enhance Language Understanding with Co-Propagation of Numerical and Categorical Features for Recommendation

Authors: Jizheng Chen, Kounianhua Du, Jianghao Lin, Bo Chen, Ruiming Tang, Weinan Zhang

Abstract: Large language models have been flourishing in the natural language processing (NLP) domain, and their potential for recommendation has been paid much attention to. Despite the intelligence shown by the recommendation-oriented finetuned models, LLMs struggle to fully understand the user behavior patterns due to their innate weakness in interpreting numerical features and the overhead for long cont… ▽ More Large language models have been flourishing in the natural language processing (NLP) domain, and their potential for recommendation has been paid much attention to. Despite the intelligence shown by the recommendation-oriented finetuned models, LLMs struggle to fully understand the user behavior patterns due to their innate weakness in interpreting numerical features and the overhead for long context, where the temporal relations among user behaviors, subtle quantitative signals among different ratings, and various side features of items are not well explored. Existing works only fine-tune a sole LLM on given text data without introducing that important information to it, leaving these problems unsolved. In this paper, we propose ELCoRec to Enhance Language understanding with CoPropagation of numerical and categorical features for Recommendation. Concretely, we propose to inject the preference understanding capability into LLM via a GAT expert model where the user preference is better encoded by parallelly propagating the temporal relations, and rating signals as well as various side information of historical items. The parallel propagation mechanism could stabilize heterogeneous features and offer an informative user preference encoding, which is then injected into the language models via soft prompting at the cost of a single token embedding. To further obtain the user's recent interests, we proposed a novel Recent interaction Augmented Prompt (RAP) template. Experiment results over three datasets against strong baselines validate the effectiveness of ELCoRec. The code is available at https://anonymous.4open.science/r/CIKM_Code_Repo-E6F5/README.md. △ Less

Submitted 26 June, 2024; originally announced June 2024.

A Novel Quantum LSTM Network

Authors: Yifan Zhou, Chong Cheng Xu, Mingi Song, Yew Kee Wong, Kangsong Du

Abstract: The rapid evolution of artificial intelligence has led to the widespread adoption of Long Short-Term Memory (LSTM) networks, known for their effectiveness in processing sequential data. However, LSTMs are constrained by inherent limitations such as the vanishing gradient problem and substantial computational demands. The advent of quantum computing presents a revolutionary approach to overcoming t… ▽ More The rapid evolution of artificial intelligence has led to the widespread adoption of Long Short-Term Memory (LSTM) networks, known for their effectiveness in processing sequential data. However, LSTMs are constrained by inherent limitations such as the vanishing gradient problem and substantial computational demands. The advent of quantum computing presents a revolutionary approach to overcoming these obstacles. This paper introduces the Quantum LSTM (qLSTM) model, which integrates quantum computing principles with traditional LSTM networks to significantly enhance computational efficiency and model performance in sequence learning tasks. Quantum computing leverages qubits, which can exist in multiple states simultaneously through superposition and entangle these states to represent complex correlations without direct physical interaction, offering a profound advancement over classical binary computing. Our qLSTM model aims to address the limitations of traditional LSTMs, providing a robust framework for more efficient and effective sequential data processing. △ Less

Submitted 13 June, 2024; originally announced June 2024.

Comments: 13 pages, 7 Figures

Extracting Essential and Disentangled Knowledge for Recommendation Enhancement

Authors: Kounianhua Du, Jizheng Chen, Jianghao Lin, Menghui Zhu, Bo Chen, Shuai Li, Ruiming Tang

Abstract: Recommender models play a vital role in various industrial scenarios, while often faced with the catastrophic forgetting problem caused by the fast shifting data distribution, e.g., the evolving user interests, click signals fluctuation during sales promotions, etc. To alleviate this problem, a common approach is to reuse knowledge from the historical data. However, preserving the vast and fast-ac… ▽ More Recommender models play a vital role in various industrial scenarios, while often faced with the catastrophic forgetting problem caused by the fast shifting data distribution, e.g., the evolving user interests, click signals fluctuation during sales promotions, etc. To alleviate this problem, a common approach is to reuse knowledge from the historical data. However, preserving the vast and fast-accumulating data is hard, which causes dramatic storage overhead. Memorizing old data through a parametric knowledge base is then proposed, which compresses the vast amount of raw data into model parameters. Despite the flexibility, how to improve the memorization and generalization capabilities of the parametric knowledge base is challenging. In this paper, we propose two constraints to extract Essential and Disentangled Knowledge from past data for rational and generalized recommendation enhancement, which improves the capabilities of the parametric knowledge base without increasing the size of it. The essential principle helps to compress the input into representative vectors that capture the task-relevant information and filter out the noisy information. The disentanglement principle reduces the redundancy of stored information and pushes the knowledge base to focus on capturing the disentangled invariant patterns. These two rules together promote rational compression of information for robust and generalized knowledge representations. Extensive experiments on two datasets justify the effectiveness of the proposed method. △ Less

Submitted 20 May, 2024; originally announced June 2024.

DisCo: Towards Harmonious Disentanglement and Collaboration between Tabular and Semantic Space for Recommendation

Authors: Kounianhua Du, Jizheng Chen, Jianghao Lin, Yunjia Xi, Hangyu Wang, Xinyi Dai, Bo Chen, Ruiming Tang, Weinan Zhang

Abstract: Recommender systems play important roles in various applications such as e-commerce, social media, etc. Conventional recommendation methods usually model the collaborative signals within the tabular representation space. Despite the personalization modeling and the efficiency, the latent semantic dependencies are omitted. Methods that introduce semantics into recommendation then emerge, injecting… ▽ More Recommender systems play important roles in various applications such as e-commerce, social media, etc. Conventional recommendation methods usually model the collaborative signals within the tabular representation space. Despite the personalization modeling and the efficiency, the latent semantic dependencies are omitted. Methods that introduce semantics into recommendation then emerge, injecting knowledge from the semantic representation space where the general language understanding are compressed. However, existing semantic-enhanced recommendation methods focus on aligning the two spaces, during which the representations of the two spaces tend to get close while the unique patterns are discarded and not well explored. In this paper, we propose DisCo to Disentangle the unique patterns from the two representation spaces and Collaborate the two spaces for recommendation enhancement, where both the specificity and the consistency of the two spaces are captured. Concretely, we propose 1) a dual-side attentive network to capture the intra-domain patterns and the inter-domain patterns, 2) a sufficiency constraint to preserve the task-relevant information of each representation space and filter out the noise, and 3) a disentanglement constraint to avoid the model from discarding the unique information. These modules strike a balance between disentanglement and collaboration of the two representation spaces to produce informative pattern vectors, which could serve as extra features and be appended to arbitrary recommendation backbones for enhancement. Experiment results validate the superiority of our method against different models and the compatibility of DisCo over different backbones. Various ablation studies and efficiency analysis are also conducted to justify each model component. △ Less

Submitted 4 June, 2024; v1 submitted 20 May, 2024; originally announced June 2024.

CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion

Authors: Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, Junchen Jiang

Abstract: Large language models (LLMs) often incorporate multiple text chunks in their inputs to provide the necessary contexts. To speed up the prefill of the long LLM inputs, one can pre-compute the KV cache of a text and re-use the KV cache when the context is reused as the prefix of another LLM input. However, the reused text chunks are not always the input prefix, and when they are not, their precomput… ▽ More Large language models (LLMs) often incorporate multiple text chunks in their inputs to provide the necessary contexts. To speed up the prefill of the long LLM inputs, one can pre-compute the KV cache of a text and re-use the KV cache when the context is reused as the prefix of another LLM input. However, the reused text chunks are not always the input prefix, and when they are not, their precomputed KV caches cannot be directly used since they ignore the text's cross-attention with the preceding text in the LLM input. Thus, the benefits of reusing KV caches remain largely unrealized. This paper tackles just one question: when an LLM input contains multiple text chunks, how to quickly combine their precomputed KV caches in order to achieve the same generation quality as the expensive full prefill (i.e., without reusing KV cache)? We present CacheBlend, a scheme that reuses the pre-computed KV caches, regardless prefix or not, and selectively recomputes the KV values of a small subset of tokens to partially update each reused KV cache. In the meantime,the small extra delay for recomputing some tokens can be pipelined with the retrieval of KV caches within the same job,allowing CacheBlend to store KV caches in slower devices with more storage capacity while retrieving them without increasing the inference delay. By comparing CacheBlend with the state-of-the-art KV cache reusing schemes on three open-source LLMs of various sizes and four popular benchmark datasets of different tasks, we show that CacheBlend reduces time-to-first-token (TTFT) by 2.2-3.3X and increases the inference throughput by 2.8-5X, compared with full KV recompute, without compromising generation quality or incurring more storage cost. △ Less

Submitted 3 June, 2024; v1 submitted 26 May, 2024; originally announced May 2024.

Learning Structure and Knowledge Aware Representation with Large Language Models for Concept Recommendation

Authors: Qingyao Li, Wei Xia, Kounianhua Du, Qiji Zhang, Weinan Zhang, Ruiming Tang, Yong Yu

Abstract: Concept recommendation aims to suggest the next concept for learners to study based on their knowledge states and the human knowledge system. While knowledge states can be predicted using knowledge tracing models, previous approaches have not effectively integrated the human knowledge system into the process of designing these educational models. In the era of rapidly evolving Large Language Model… ▽ More Concept recommendation aims to suggest the next concept for learners to study based on their knowledge states and the human knowledge system. While knowledge states can be predicted using knowledge tracing models, previous approaches have not effectively integrated the human knowledge system into the process of designing these educational models. In the era of rapidly evolving Large Language Models (LLMs), many fields have begun using LLMs to generate and encode text, introducing external knowledge. However, integrating LLMs into concept recommendation presents two urgent challenges: 1) How to construct text for concepts that effectively incorporate the human knowledge system? 2) How to adapt non-smooth, anisotropic text encodings effectively for concept recommendation? In this paper, we propose a novel Structure and Knowledge Aware Representation learning framework for concept Recommendation (SKarREC). We leverage factual knowledge from LLMs as well as the precedence and succession relationships between concepts obtained from the knowledge graph to construct textual representations of concepts. Furthermore, we propose a graph-based adapter to adapt anisotropic text embeddings to the concept recommendation task. This adapter is pre-trained through contrastive learning on the knowledge graph to get a smooth and structure-aware concept representation. Then, it's fine-tuned through the recommendation task, forming a text-to-knowledge-to-recommendation adaptation pipeline, which effectively constructs a structure and knowledge-aware concept representation. Our method does a better job than previous adapters in transforming text encodings for application in concept recommendation. Extensive experiments on real-world datasets demonstrate the effectiveness of the proposed approach. △ Less

Submitted 20 May, 2024; originally announced May 2024.

Comments: 11 pages, 8 figures

Causal Inference from Slowly Varying Nonstationary Processes

Authors: Kang Du, Yu Xiang

Abstract: Causal inference from observational data following the restricted structural causal models (SCM) framework hinges largely on the asymmetry between cause and effect from the data generating mechanisms, such as non-Gaussianity or non-linearity. This methodology can be adapted to stationary time series, yet inferring causal relationships from nonstationary time series remains a challenging task. In t… ▽ More Causal inference from observational data following the restricted structural causal models (SCM) framework hinges largely on the asymmetry between cause and effect from the data generating mechanisms, such as non-Gaussianity or non-linearity. This methodology can be adapted to stationary time series, yet inferring causal relationships from nonstationary time series remains a challenging task. In this work, we propose a new class of restricted SCM, via a time-varying filter and stationary noise, and exploit the asymmetry from nonstationarity for causal identification in both bivariate and network settings. We propose efficient procedures by leveraging powerful estimates of the bivariate evolutionary spectra for slowly varying processes. Various synthetic and real datasets that involve high-order and non-smooth filters are evaluated to demonstrate the effectiveness of our proposed methodology. △ Less

Submitted 29 May, 2024; v1 submitted 11 May, 2024; originally announced May 2024.

Comments: This work was intended as a replacement of arXiv:2012.13025 and any subsequent updates will appear there

CodeGRAG: Extracting Composed Syntax Graphs for Retrieval Augmented Cross-Lingual Code Generation

Authors: Kounianhua Du, Renting Rui, Huacan Chai, Lingyue Fu, Wei Xia, Yasheng Wang, Ruiming Tang, Yong Yu, Weinan Zhang

Abstract: Utilizing large language models to generate codes has shown promising meaning in software development revolution. Despite the intelligence shown by the general large language models, their specificity in code generation can still be improved due to the syntactic gap and mismatched vocabulary existing among natural language and different programming languages. In addition, programming languages are… ▽ More Utilizing large language models to generate codes has shown promising meaning in software development revolution. Despite the intelligence shown by the general large language models, their specificity in code generation can still be improved due to the syntactic gap and mismatched vocabulary existing among natural language and different programming languages. In addition, programming languages are inherently logical and complex, making them hard to be correctly generated. Existing methods rely on multiple prompts to the large language model to explore better solutions, which is expensive. In this paper, we propose Syntax Graph Retrieval Augmented Code Generation (CodeGRAG) to enhance the performance of LLMs in single-round code generation tasks. CodeGRAG extracts and summarizes the control flow and data flow of code blocks to fill the gap between programming languages and natural language. The extracted external structural knowledge models the inherent flows of code blocks, which can facilitate LLMs for better understanding of code syntax and serve as a bridge among different programming languages. CodeGRAG significantly improves the code generation ability of LLMs and can even offer performance gain for cross-lingual code generation, e.g., C++ for Python. △ Less

Submitted 2 May, 2024; originally announced May 2024.

Empirical approximation to invariant measures of mean-field Langevin dynamics

Authors: Wen**g Cao, Kai Du

Abstract: This paper is concerned with the approximation to invariant measures for Langevin dynamics of McKean--Vlasov type. Under dissipativity and Lipschitz conditions, we prove that the empirical measures of both the mean-field and self-interacting Langevin dynamics converge to the invariant measure in the Wasserstein distance. Numerical experiments are conducted to illustrate theoretical results. This paper is concerned with the approximation to invariant measures for Langevin dynamics of McKean--Vlasov type. Under dissipativity and Lipschitz conditions, we prove that the empirical measures of both the mean-field and self-interacting Langevin dynamics converge to the invariant measure in the Wasserstein distance. Numerical experiments are conducted to illustrate theoretical results. △ Less

Submitted 28 April, 2024; originally announced April 2024.

Comments: 21 pages, 1 figure

MSC Class: 60B10; 37M25; 82C31; 60H10

Mining Invariance from Nonlinear Multi-Environment Data: Binary Classification

Authors: Austin Goddard, Kang Du, Yu Xiang

Abstract: Making predictions in an unseen environment given data from multiple training environments is a challenging task. We approach this problem from an invariance perspective, focusing on binary classification to shed light on general nonlinear data generation mechanisms. We identify a unique form of invariance that exists solely in a binary setting that allows us to train models invariant over environ… ▽ More Making predictions in an unseen environment given data from multiple training environments is a challenging task. We approach this problem from an invariance perspective, focusing on binary classification to shed light on general nonlinear data generation mechanisms. We identify a unique form of invariance that exists solely in a binary setting that allows us to train models invariant over environments. We provide sufficient conditions for such invariance and show it is robust even when environmental conditions vary greatly. Our formulation admits a causal interpretation, allowing us to compare it with various frameworks. Finally, we propose a heuristic prediction method and conduct experiments using real and synthetic datasets. △ Less

Submitted 23 April, 2024; originally announced April 2024.

Comments: Accepted to the 2024 International Symposium on Information Theory (ISIT)

Low-energy spin dynamics in a Kitaev material Na3Ni2BiO6 investigated by NMR

Authors: Xinyu Shi, Yi Cui, Yanyan Shangguan, Xiaoyu Xu, Zhanlong Wu, Ze Hu, Shuo Li, Kefan Du, Ying Chen, Long Ma, Zhengxin Liu, **sheng Wen, **shan Zhang, Weiqiang Yu

Abstract: We performed 23Na NMR and magnetization measurements on an S = 1, quasi-2D honeycomb lattice antiferromagnet Na3Ni2BiO6. A large positive Curie-Weiss constant of 22.9 K is observed. The NMR spectra at low fields are consistent with a "zigzag" magnetic order, indicating a large easy-axis anisotropy. With field applied along the c* axis, the NMR spectra confirm the existence of a 1/3-magnetization p… ▽ More We performed 23Na NMR and magnetization measurements on an S = 1, quasi-2D honeycomb lattice antiferromagnet Na3Ni2BiO6. A large positive Curie-Weiss constant of 22.9 K is observed. The NMR spectra at low fields are consistent with a "zigzag" magnetic order, indicating a large easy-axis anisotropy. With field applied along the c* axis, the NMR spectra confirm the existence of a 1/3-magnetization plateau phase between 5.1 T and 7.1 T. The transition from the zigzag order to the 1/3-magnetization plateau phase is also found to be a first-order type. A monotonic decrease of the spin gap is revealed in the 1/3-magnetization plateau phase, which reaches zero at a quantum critical field Hc = 8.35 T before entering the fully polarized phase. These data suggest the existence of exchange frustration in the system along with strong ferromagnetic interactions, hosting the possibility for Kitaev physics. Besides, well below the ordered phase, the 1/T1 at high fields shows either a level off or an enhancement upon cooling below 3 K, which suggests the existence of low-energy fluctuations. △ Less

Submitted 11 April, 2024; originally announced April 2024.

Comments: 7 pages, 7 figures

Context versus Prior Knowledge in Language Models

Authors: Kevin Du, Vésteinn Snæbjarnarson, Niklas Stoehr, Jennifer C. White, Aaron Schein, Ryan Cotterell

Abstract: To answer a question, language models often need to integrate prior knowledge learned during pretraining and new information presented in context. We hypothesize that models perform this integration in a predictable way across different questions and contexts: models will rely more on prior knowledge for questions about entities (e.g., persons, places, etc.) that they are more familiar with due to… ▽ More To answer a question, language models often need to integrate prior knowledge learned during pretraining and new information presented in context. We hypothesize that models perform this integration in a predictable way across different questions and contexts: models will rely more on prior knowledge for questions about entities (e.g., persons, places, etc.) that they are more familiar with due to higher exposure in the training corpus, and be more easily persuaded by some contexts than others. To formalize this problem, we propose two mutual information-based metrics to measure a model's dependency on a context and on its prior about an entity: first, the persuasion score of a given context represents how much a model depends on the context in its decision, and second, the susceptibility score of a given entity represents how much the model can be swayed away from its original answer distribution about an entity. We empirically test our metrics for their validity and reliability. Finally, we explore and find a relationship between the scores and the model's expected familiarity with an entity, and provide two use cases to illustrate their benefits. △ Less

Submitted 16 June, 2024; v1 submitted 6 April, 2024; originally announced April 2024.

Comments: Long paper accepted at ACL 2024

Well-posedness of the obstacle problem for stochastic nonlinear diffusion equations: an entropy formulation

Authors: Kai Du, Ruoyang Liu

Abstract: In this paper, we establish the existence, uniqueness and stability results for the obstacle problem associated with a degenerate nonlinear diffusion equation perturbed by conservative gradient noise. Our approach revolves round introducing a new entropy formulation for stochastic variational inequalities. As a consequence, we obtain a novel well-posedness result for the obstacle problem of determ… ▽ More In this paper, we establish the existence, uniqueness and stability results for the obstacle problem associated with a degenerate nonlinear diffusion equation perturbed by conservative gradient noise. Our approach revolves round introducing a new entropy formulation for stochastic variational inequalities. As a consequence, we obtain a novel well-posedness result for the obstacle problem of deterministic porous medium equations with nonlinear reaction terms. △ Less

Submitted 3 April, 2024; originally announced April 2024.

Comments: 36 pages

MSC Class: 60H15; 35K86; 35K65; 47J20

IPT-V2: Efficient Image Processing Transformer using Hierarchical Attentions

Authors: Zhijun Tu, Kunpeng Du, Hanting Chen, Hailing Wang, Wei Li, Jie Hu, Yunhe Wang

Abstract: Recent advances have demonstrated the powerful capability of transformer architecture in image restoration. However, our analysis indicates that existing transformerbased methods can not establish both exact global and local dependencies simultaneously, which are much critical to restore the details and missing content of degraded images. To this end, we present an efficient image processing trans… ▽ More Recent advances have demonstrated the powerful capability of transformer architecture in image restoration. However, our analysis indicates that existing transformerbased methods can not establish both exact global and local dependencies simultaneously, which are much critical to restore the details and missing content of degraded images. To this end, we present an efficient image processing transformer architecture with hierarchical attentions, called IPTV2, adopting a focal context self-attention (FCSA) and a global grid self-attention (GGSA) to obtain adequate token interactions in local and global receptive fields. Specifically, FCSA applies the shifted window mechanism into the channel self-attention, helps capture the local context and mutual interaction across channels. And GGSA constructs long-range dependencies in the cross-window grid, aggregates global information in spatial dimension. Moreover, we introduce structural re-parameterization technique to feed-forward network to further improve the model capability. Extensive experiments demonstrate that our proposed IPT-V2 achieves state-of-the-art results on various image processing tasks, covering denoising, deblurring, deraining and obtains much better trade-off for performance and computational complexity than previous methods. Besides, we extend our method to image generation as latent diffusion backbone, and significantly outperforms DiTs. △ Less

Submitted 31 March, 2024; originally announced April 2024.

Particle approximation for a conditional McKean--Vlasov stochastic differential equation

Authors: Kai Du, Yunzhang Li, Yuyang Ye

Abstract: In this paper, we construct a type of interacting particle systems to approximate a class of stochastic different equations whose coefficients depend on the conditional probability distributions of the processes given partial observations. After proving the well-posedness and regularity of the particle systems, we establish a quantitative convergence result for the empirical measures of the partic… ▽ More In this paper, we construct a type of interacting particle systems to approximate a class of stochastic different equations whose coefficients depend on the conditional probability distributions of the processes given partial observations. After proving the well-posedness and regularity of the particle systems, we establish a quantitative convergence result for the empirical measures of the particle systems in the Wasserstein space, as the number of particles increases. Moreover, we discuss an Euler--Maruyama scheme of the particle system and validate its strong convergence. A numerical experiment is conducted to illustrate our results. △ Less

Submitted 26 March, 2024; originally announced March 2024.

CMViM: Contrastive Masked Vim Autoencoder for 3D Multi-modal Representation Learning for AD classification

Authors: Guangqian Yang, Kangrui Du, Zhihan Yang, Ye Du, Yong** Zheng, Shujun Wang

Abstract: Alzheimer's disease (AD) is an incurable neurodegenerative condition leading to cognitive and functional deterioration. Given the lack of a cure, prompt and precise AD diagnosis is vital, a complex process dependent on multiple factors and multi-modal data. While successful efforts have been made to integrate multi-modal representation learning into medical datasets, scant attention has been given… ▽ More Alzheimer's disease (AD) is an incurable neurodegenerative condition leading to cognitive and functional deterioration. Given the lack of a cure, prompt and precise AD diagnosis is vital, a complex process dependent on multiple factors and multi-modal data. While successful efforts have been made to integrate multi-modal representation learning into medical datasets, scant attention has been given to 3D medical images. In this paper, we propose Contrastive Masked Vim Autoencoder (CMViM), the first efficient representation learning method tailored for 3D multi-modal data. Our proposed framework is built on a masked Vim autoencoder to learn a unified multi-modal representation and long-dependencies contained in 3D medical images. We also introduce an intra-modal contrastive learning module to enhance the capability of the multi-modal Vim encoder for modeling the discriminative features in the same modality, and an inter-modal contrastive learning module to alleviate misaligned representation among modalities. Our framework consists of two main steps: 1) incorporate the Vision Mamba (Vim) into the mask autoencoder to reconstruct 3D masked multi-modal data efficiently. 2) align the multi-modal representations with contrastive learning mechanisms from both intra-modal and inter-modal aspects. Our framework is pre-trained and validated ADNI2 dataset and validated on the downstream task for AD classification. The proposed CMViM yields 2.7\% AUC performance improvement compared with other state-of-the-art methods. △ Less

Submitted 25 March, 2024; originally announced March 2024.

Comments: 11 pages, 1 figure

Confidence Self-Calibration for Multi-Label Class-Incremental Learning

Authors: Kaile Du, Yifan Zhou, Fan Lyu, Yuyang Li, Chen Lu, Guangcan Liu

Abstract: The partial label challenge in Multi-Label Class-Incremental Learning (MLCIL) arises when only the new classes are labeled during training, while past and future labels remain unavailable. This issue leads to a proliferation of false-positive errors due to erroneously high confidence multi-label predictions, exacerbating catastrophic forgetting within the disjoint label space. In this paper, we ai… ▽ More The partial label challenge in Multi-Label Class-Incremental Learning (MLCIL) arises when only the new classes are labeled during training, while past and future labels remain unavailable. This issue leads to a proliferation of false-positive errors due to erroneously high confidence multi-label predictions, exacerbating catastrophic forgetting within the disjoint label space. In this paper, we aim to refine multi-label confidence calibration in MLCIL and propose a Confidence Self-Calibration (CSC) approach. Firstly, for label relationship calibration, we introduce a class-incremental graph convolutional network that bridges the isolated label spaces by constructing learnable, dynamically extended label relationship graph. Then, for confidence calibration, we present a max-entropy regularization for each multi-label increment, facilitating confidence self-calibration through the penalization of over-confident output distributions. Our approach attains new state-of-the-art results in MLCIL tasks on both MS-COCO and PASCAL VOC datasets, with the calibration of label confidences confirmed through our methodology. △ Less

Submitted 19 March, 2024; originally announced March 2024.

Earth+: on-board satellite imagery compression leveraging historical earth observations

Authors: Kuntai Du, Yihua Cheng, Peder Olsen, Shadi Noghabi, Ranveer Chandra, Junchen Jiang

Abstract: With the increasing deployment of earth observation satellite constellations, the downlink (satellite-to-ground) capacity often limits the freshness, quality, and coverage of the imagery data available to applications on the ground. To overcome the downlink limitation, we present Earth+, a new satellite imagery compression system that, instead of compressing each image individually, pinpoints and… ▽ More With the increasing deployment of earth observation satellite constellations, the downlink (satellite-to-ground) capacity often limits the freshness, quality, and coverage of the imagery data available to applications on the ground. To overcome the downlink limitation, we present Earth+, a new satellite imagery compression system that, instead of compressing each image individually, pinpoints and downloads only recent imagery changes with respect to the history reference images. To minimize the amount of changes, it is critical to make reference images as fresh as possible. Earth+ enables each satellite to choose fresh reference images from not only its own history images but also past images of other satellites from an entire satellite constellation. To share reference images across satellites, Earth+ utilizes the limited capacity of the existing uplink (ground-to-satellite) by judiciously selecting and compressing reference images while still allowing accurate change detection. In short, Earth+ is the first to make reference-based compression efficient, by enabling constellation-wide sharing of fresh reference images across satellites. Our evaluation shows that Earth+ can reduce the downlink usage by a factor of 3.3 compared to state-of-the-art on-board image compression techniques while not sacrificing image quality, or using more on-board computing or storage resources, or more uplink bandwidth than currently available. △ Less

Submitted 17 March, 2024; originally announced March 2024.

Molecular intercalation in the van der Waals antiferromagnets FePS3 and NiPS3

Authors: Cong Li, Ze Hu, Xiaofei Hou, Sheng Xu, Zhanlong Wu, Kefan Du, Shuo Li, Xiaoyu Xu, Ying Chen, Zeyu Wang, Tiancheng Mu, Tian-Long Xia, Yanfeng Guo, B. Normand, Weiqiang Yu, Yi Cui

Abstract: We have performed electrochemical treatment of the van der Waals antiferromagnetic materials FePS$_3$ and NiPS$_3$ with the ionic liquid EMIM-BF$_4$, achieving significant molecular intercalation. Mass analysis of the intercalated compounds, EMIM$_x$-FePS$_3$ and EMIM$_x$-NiPS$_3$, indicated respective intercalation levels, $x$, of approximately 27\% and 37\%, and X-ray diffraction measurements de… ▽ More We have performed electrochemical treatment of the van der Waals antiferromagnetic materials FePS$_3$ and NiPS$_3$ with the ionic liquid EMIM-BF$_4$, achieving significant molecular intercalation. Mass analysis of the intercalated compounds, EMIM$_x$-FePS$_3$ and EMIM$_x$-NiPS$_3$, indicated respective intercalation levels, $x$, of approximately 27\% and 37\%, and X-ray diffraction measurements demonstrated a massive (over 50\%) enhancement of the $c$-axis lattice parameters. To investigate the consequences of these changes for the magnetic properties, we performed magnetic susceptibility and $^{31}$P nuclear magnetic resonance (NMR) studies of both systems. For EMIM$_x$-FePS$_3$, intercalation reduces the magnetic ordering temperature from $T_N = 120$~K to 78~K, and we find a spin gap in the antiferromagnetic phase that drops from 45~K to 30~K. For EMIM$_x$-NiPS$_3$, the ordering temperature is almost unaffected (changing from 148~K to 145~K), but a change towards nearly isotropic spin fluctuations suggests an alteration of the magnetic Hamiltonian. Such relatively modest changes, given that the huge extension of the $c$ axes is expected to cause a very strong suppression any interlayer interactions, point unequivocally to the conclusion that the magnetic properties of both parent compounds are determined solely by two-dimensional (2D), intralayer physics. The changes in transition temperatures and low-temperature spin dynamics in both compounds therefore indicate that intercalation also results in a significant modulation of the intralayer magnetic interactions, which we propose is due to charge do** and localization on the P sites. Our study offers chemical intercalation with ionic liquids as an effective method to control not only the interlayer but also the intralayer interactions in quasi-2D magnetic materials. △ Less

Submitted 3 March, 2024; originally announced March 2024.

Journal ref: Physical Review B 109, 184407(2024)

Exponential Cluster Synchronization in Fast Switching Network Topologies: A Pinning Control Approach with Necessary and Sufficient Conditions

Authors: Ku Du, Yu Kang

Abstract: This research investigates the intricate domain of synchronization problem among multiple agents operating within a dynamic fast switching network topology. We concentrate on cluster synchronization within coupled linear system under pinning control, providing both necessary and sufficient conditions. As a pivotal aspect, this paper aim to president the weakest possible conditions to make the coup… ▽ More This research investigates the intricate domain of synchronization problem among multiple agents operating within a dynamic fast switching network topology. We concentrate on cluster synchronization within coupled linear system under pinning control, providing both necessary and sufficient conditions. As a pivotal aspect, this paper aim to president the weakest possible conditions to make the coupled linear system realize cluster synchronization exponentially. Within the context of fast switching framework, we initially examine the necessary conditions, commencing with the transformation of the consensus problem into a stability problem, introducing a new variable to make the coupled system achieve cluster synchronization if the system is controllable; communication topology switching fast enough and the coupling strength should be sufficiently robust. Then, by using the Lyapunov theorem, we also present that the state matrix controllable is necessary for cluster synchronization. Furthermore, this paper culminating in the incorporation of contraction theory and an invariant manifold, demonstrating that the switching topology has an average is imperative for achieving cluster synchronization. Finally, we introduce three simulations to validate the efficacy of the proposed approach. △ Less

Submitted 18 February, 2024; originally announced February 2024.

Variational Continual Test-Time Adaptation

Authors: Fan Lyu, Kaile Du, Yuyang Li, Hanyu Zhao, Zhang Zhang, Guangcan Liu, Liang Wang

Abstract: The prior drift is crucial in Continual Test-Time Adaptation (CTTA) methods that only use unlabeled test data, as it can cause significant error propagation. In this paper, we introduce VCoTTA, a variational Bayesian approach to measure uncertainties in CTTA. At the source stage, we transform a pre-trained deterministic model into a Bayesian Neural Network (BNN) via a variational warm-up strategy,… ▽ More The prior drift is crucial in Continual Test-Time Adaptation (CTTA) methods that only use unlabeled test data, as it can cause significant error propagation. In this paper, we introduce VCoTTA, a variational Bayesian approach to measure uncertainties in CTTA. At the source stage, we transform a pre-trained deterministic model into a Bayesian Neural Network (BNN) via a variational warm-up strategy, injecting uncertainties into the model. During the testing time, we employ a mean-teacher update strategy using variational inference for the student model and exponential moving average for the teacher model. Our novel approach updates the student model by combining priors from both the source and teacher models. The evidence lower bound is formulated as the cross-entropy between the student and teacher models, along with the Kullback-Leibler (KL) divergence of the prior mixture. Experimental results on three datasets demonstrate the method's effectiveness in mitigating prior drift within the CTTA framework. △ Less

Submitted 12 February, 2024; originally announced February 2024.

Low-Rank Approximation of Structural Redundancy for Self-Supervised Learning

Authors: Kang Du, Yu Xiang

Abstract: We study the data-generating mechanism for reconstructive SSL to shed light on its effectiveness. With an infinite amount of labeled samples, we provide a sufficient and necessary condition for perfect linear approximation. The condition reveals a full-rank component that preserves the label classes of Y, along with a redundant component. Motivated by the condition, we propose to approximate the r… ▽ More We study the data-generating mechanism for reconstructive SSL to shed light on its effectiveness. With an infinite amount of labeled samples, we provide a sufficient and necessary condition for perfect linear approximation. The condition reveals a full-rank component that preserves the label classes of Y, along with a redundant component. Motivated by the condition, we propose to approximate the redundant component by a low-rank factorization and measure the approximation quality by introducing a new quantity $ε_s$, parameterized by the rank of factorization s. We incorporate $ε_s$ into the excess risk analysis under both linear regression and ridge regression settings, where the latter regularization approach is to handle scenarios when the dimension of the learned features is much larger than the number of labeled samples n for downstream tasks. We design three stylized experiments to compare SSL with supervised learning under different settings to support our theoretical findings. △ Less

Submitted 27 May, 2024; v1 submitted 9 February, 2024; originally announced February 2024.

Comments: Accepted to the 3rd Conference on Causal Learning and Reasoning (CLeaR)

Integration of cognitive tasks into artificial general intelligence test for large models

Authors: Youzhi Qu, Chen Wei, Penghui Du, Wenxin Che, Chi Zhang, Wanli Ouyang, Yatao Bian, Feiyang Xu, Bin Hu, Kai Du, Haiyan Wu, Jia Liu, Quanying Liu

Abstract: During the evolution of large models, performance evaluation is necessarily performed to assess their capabilities and ensure safety before practical application. However, current model evaluations mainly rely on specific tasks and datasets, lacking a united framework for assessing the multidimensional intelligence of large models. In this perspective, we advocate for a comprehensive framework of… ▽ More During the evolution of large models, performance evaluation is necessarily performed to assess their capabilities and ensure safety before practical application. However, current model evaluations mainly rely on specific tasks and datasets, lacking a united framework for assessing the multidimensional intelligence of large models. In this perspective, we advocate for a comprehensive framework of cognitive science-inspired artificial general intelligence (AGI) tests, aimed at fulfilling the testing needs of large models with enhanced capabilities. The cognitive science-inspired AGI tests encompass the full spectrum of intelligence facets, including crystallized intelligence, fluid intelligence, social intelligence, and embodied intelligence. To assess the multidimensional intelligence of large models, the AGI tests consist of a battery of well-designed cognitive tests adopted from human intelligence tests, and then naturally encapsulates into an immersive virtual community. We propose increasing the complexity of AGI testing tasks commensurate with advancements in large models and emphasizing the necessity for the interpretation of test results to avoid false negatives and false positives. We believe that cognitive science-inspired AGI tests will effectively guide the targeted improvement of large models in specific dimensions of intelligence and accelerate the integration of large models into human society. △ Less

Submitted 5 March, 2024; v1 submitted 4 February, 2024; originally announced February 2024.

Eloquent: A More Robust Transmission Scheme for LLM Token Streaming

Authors: Hanchen Li, Yuhan Liu, Yihua Cheng, Siddhant Ray, Kuntai Du, Junchen Jiang

Abstract: To render each generated token in real-time for users, the Large Language Model (LLM) server generates tokens one by one and streams each token (or group of a few tokens) through the network to the user right after generation, which we refer to as LLM token streaming. However, under unstable network conditions, the LLM token streaming experience could suffer greatly from stalls since one packet lo… ▽ More To render each generated token in real-time for users, the Large Language Model (LLM) server generates tokens one by one and streams each token (or group of a few tokens) through the network to the user right after generation, which we refer to as LLM token streaming. However, under unstable network conditions, the LLM token streaming experience could suffer greatly from stalls since one packet loss could block the rendering of later tokens even if the packets containing them arrive on time. With a measurement study, we show that current applications suffer from increased stalls under unstable networks. For this emerging token streaming problem in LLM Chatbots that differs from previous multimedia and text applications, we propose a novel transmission scheme, called Eloquent, which puts newly generated tokens as well as currently unacknowledged tokens in the next outgoing packet. This ensures that each packet contains some new tokens and, in the meantime, is independently rendered when received, avoiding the aforementioned stalls caused by missing packets. Through simulation under various networks, we show Eloquent reduces stall ratio (proportion of token rendering wait time) by 71.0% compared to the retransmission method commonly used by real chatbot applications and by 31.6% compared to the baseline packet duplication scheme. By tailoring Eloquent to fit the token-by-token generation of LLM, we enable the Chatbots to respond like an eloquent speaker for users to better enjoy pervasive AI. △ Less

Submitted 16 June, 2024; v1 submitted 23 January, 2024; originally announced January 2024.

Comments: In SIGCOMM Workshop on Networks for AI Computing (NAIC '24)

Obtaining the pseudoinverse solution of singular range-symmetric linear systems with GMRES-type methods

Authors: Kui Du, Jia-Jun Fan, Fang Wang

Abstract: It is well known that for singular inconsistent range-symmetric linear systems, the generalized minimal residual (GMRES) method determines a least squares solution without breakdown. The reached least squares solution may be or not be the pseudoinverse solution. We show that a lift strategy can be used to obtain the pseudoinverse solution. In addition, we propose a new iterative method named RSMAR… ▽ More It is well known that for singular inconsistent range-symmetric linear systems, the generalized minimal residual (GMRES) method determines a least squares solution without breakdown. The reached least squares solution may be or not be the pseudoinverse solution. We show that a lift strategy can be used to obtain the pseudoinverse solution. In addition, we propose a new iterative method named RSMAR (minimum $\mathbf A$-residual) for range-symmetric linear systems $\mathbf A\mathbf x=\mathbf b$. At step $k$ RSMAR minimizes $\|\mathbf A\mathbf r_k\|$ in the $k$th Krylov subspace generated with $\{\mathbf A, \mathbf r_0\}$ rather than $\|\mathbf r_k\|$, where $\mathbf r_k$ is the $k$th residual vector and $\|\cdot\|$ denotes the Euclidean vector norm. We show that RSMAR and GMRES terminate with the same least squares solution when applied to range-symmetric linear systems. We provide two implementations for RSMAR. Our numerical experiments show that RSMAR is the most suitable method among GMRES-type methods for singular inconsistent range-symmetric linear systems. △ Less

Submitted 22 January, 2024; v1 submitted 22 January, 2024; originally announced January 2024.

Comments: 22 pages, 4 figures

MSC Class: 15A06; 15A09; 65F10; 65F25; 65F50

Towards Causal Relationship in Indefinite Data: Baseline Model and New Datasets

Authors: Hang Chen, Xinyu Yang, Keqing Du

Abstract: Integrating deep learning and causal discovery has encouraged us to spot that learning causal structures and representations in dialogue and video is full of challenges. We defined These data forms as "Indefinite Data", characterized by multi-structure data and multi-value representations. Unlike existing adaptable data forms, Indefinite Data still faces gaps in datasets and methods. To address th… ▽ More Integrating deep learning and causal discovery has encouraged us to spot that learning causal structures and representations in dialogue and video is full of challenges. We defined These data forms as "Indefinite Data", characterized by multi-structure data and multi-value representations. Unlike existing adaptable data forms, Indefinite Data still faces gaps in datasets and methods. To address the dataset gap, we release two high-quality datasets - Causalogue and Causaction, containing text dialogue samples and video action samples with causal annotations respectively. Moreover, the method gap arises from the coexistence of multi-structure data and multi-value representations, breaking the assumptions of all current methods and rendering them infeasible on Indefinite Data. To this end, we propose a probabilistic framework as a baseline, incorporating three designed highlights for this gap: 1) establishing Causation Condition of representations using the independence of noise terms under non-fixed causal structures, 2) treating causal strength as a latent variable and measuring the reconstruction loss in the correlation space, and 3) estimating the effects of latent confounders. These highpoints make the probabilistic model capable of overcoming challenges brought by the coexistence of multi-structure data and multi-value representations and pave the way for the extension of latent confounders. Comprehensive experiments have evaluated baseline results of causal structures, causal representations, and confounding disentanglement. △ Less

Submitted 16 January, 2024; originally announced January 2024.

Comments: If you are interested in the two new datasets, pls contact us by email

GPBiLQ and GPQMR: Two iterative methods for unsymmetric partitioned linear systems

Authors: Kui Du, Jia-Jun Fan, Fang Wang

Abstract: We introduce two iterative methods, GPBiLQ and GPQMR, for solving unsymmetric partitioned linear systems. The basic mechanism underlying GPBiLQ and GPQMR is a novel simultaneous tridiagonalization via biorthogonality that allows for short-recurrence iterative schemes. Similar to the biconjugate gradient method, it is possible to develop another method, GPBiCG, whose iterate (if it exists) can be o… ▽ More We introduce two iterative methods, GPBiLQ and GPQMR, for solving unsymmetric partitioned linear systems. The basic mechanism underlying GPBiLQ and GPQMR is a novel simultaneous tridiagonalization via biorthogonality that allows for short-recurrence iterative schemes. Similar to the biconjugate gradient method, it is possible to develop another method, GPBiCG, whose iterate (if it exists) can be obtained inexpensively from the GPBiLQ iterate. Whereas the iterate of GPBiCG may not exist, the iterates of GPBiLQ and GPQMR are always well defined as long as the biorthogonal tridiagonal reduction process does not break down. We discuss connections between the proposed methods and some existing methods, and give numerical experiments to illustrate the performance of the proposed methods. △ Less

Submitted 4 January, 2024; originally announced January 2024.

Comments: 22 pages, 4 figures

MSC Class: 15A06; 65F10; 65F25; 65F50

Chirality tuning and reversing with resonant phase-change metasurfaces

Authors: Xinbo Sha, Kang Du, Yixuan Zeng, Fangxing Lai, Jun Yin, Hanxu Zhang, Bo Song, Jiecai Han, Shumin Xiao, Yuri Kivshar, Qinghai Song

Abstract: Dynamic control of circular dichroism in photonic structures is critically important for compact spectrometers, stereoscopic displays, and information processing exploiting multiple degrees of freedom. Metasurfaces can help miniaturize chiral devices but only produce static and limited chiral responses. While external stimuli are able to tune resonances, their modulations are often weak, and rever… ▽ More Dynamic control of circular dichroism in photonic structures is critically important for compact spectrometers, stereoscopic displays, and information processing exploiting multiple degrees of freedom. Metasurfaces can help miniaturize chiral devices but only produce static and limited chiral responses. While external stimuli are able to tune resonances, their modulations are often weak, and reversing continuously the sign of circular dichroism is extremely challenging. Here, we demonstrate dynamically tunable chiral response of resonant metasurfaces supporting chiral bound states in the continuum combining them with phase-change materials. Phase transition between amorphous and crystalline phases allows to control chiral response and vary chirality rapidly from -0.947 to +0.958 backward and forward via chirality continuum. Our demonstrations underpin the rapid development of chiral photonics and its applications. △ Less

Submitted 2 January, 2024; originally announced January 2024.

Comments: 14 pages, 4 figures

Giant X-ray circular dichroism in a time-reversal invariant altermagnet

Authors: Jun Okamoto, Ru-Pan Wang, Yen-Yi Chu, Hung-Wei Shiu, Amol Singh, Hsiao-Yu Huang, Chung-Yu Mou, Sucitto Teh, Horng-Tay Jeng, Kai Du, Xianghan Xu, Sang-Wook Cheong, Chao-Hung Du, Chien-Te Chen, Atsushi Fujimori, Di-**g Huang

Abstract: X-ray circular dichroism, arising from the contrast in X-ray absorption between opposite photon helicities, serves as a spectroscopic tool to measure the magnetization of ferromagnetic materials and identify the handedness of chiral crystals. Antiferromagnets with crystallographic chirality typically lack X-ray magnetic circular dichroism because of time-reversal symmetry, yet exhibit weak X-ray n… ▽ More X-ray circular dichroism, arising from the contrast in X-ray absorption between opposite photon helicities, serves as a spectroscopic tool to measure the magnetization of ferromagnetic materials and identify the handedness of chiral crystals. Antiferromagnets with crystallographic chirality typically lack X-ray magnetic circular dichroism because of time-reversal symmetry, yet exhibit weak X-ray natural circular dichroism. Here, we report the observation of giant natural circular dichroism in the Ni $L_3$-edge X-ray absorption of Ni$_3$TeO$_6$, a polar and chiral antiferromagnet with effective time-reversal symmetry. To unravel this intriguing phenomenon, we propose a phenomenological model that classifies the movement of photons in a chiral crystal within the same symmetry class as that of a magnetic field. The coupling of X-ray polarization with the induced magnetization yields giant X-ray natural circular dichroism, revealing the altermagnetism of Ni$_3$TeO$_6$. Our findings provide evidence for the interplay between magnetism and crystal chirality in natural optical activity. Additionally, we establish the first example of a new class of magnetic materials exhibiting circular dichroism with time-reversal symmetry. △ Less

Submitted 23 February, 2024; v1 submitted 12 December, 2023; originally announced December 2023.

Comments: Accepted by Advanced Materials (2024.2.16) Revised title: Giant X-ray circular dichroism in a time-reversal invariant altermagnet Revised drafts: Main 14 pages, 4 figures, and SI 20 pages, 8 figures

Grammatical Gender's Influence on Distributional Semantics: A Causal Perspective

Authors: Karolina Stańczak, Kevin Du, Adina Williams, Isabelle Augenstein, Ryan Cotterell

Abstract: How much meaning influences gender assignment across languages is an active area of research in modern linguistics and cognitive science. We can view current approaches as aiming to determine where gender assignment falls on a spectrum, from being fully arbitrarily determined to being largely semantically determined. For the latter case, there is a formulation of the neo-Whorfian hypothesis, which… ▽ More How much meaning influences gender assignment across languages is an active area of research in modern linguistics and cognitive science. We can view current approaches as aiming to determine where gender assignment falls on a spectrum, from being fully arbitrarily determined to being largely semantically determined. For the latter case, there is a formulation of the neo-Whorfian hypothesis, which claims that even inanimate noun gender influences how people conceive of and talk about objects (using the choice of adjective used to modify inanimate nouns as a proxy for meaning). We offer a novel, causal graphical model that jointly represents the interactions between a noun's grammatical gender, its meaning, and adjective choice. In accordance with past results, we find a relationship between the gender of nouns and the adjectives which modify them. However, when we control for the meaning of the noun, we find that grammatical gender has a near-zero effect on adjective choice, thereby calling the neo-Whorfian hypothesis into question. △ Less

Submitted 30 November, 2023; originally announced November 2023.

CASR: Refining Action Segmentation via Marginalizing Frame-levle Causal Relationships

Authors: Keqing Du, Xinyu Yang, Hang Chen

Abstract: Integrating deep learning and causal discovery has increased the interpretability of Temporal Action Segmentation (TAS) tasks. However, frame-level causal relationships exist many complicated noises outside the segment-level, making it infeasible to directly express macro action semantics. Thus, we propose Causal Abstraction Segmentation Refiner (CASR), which can refine TAS results from various mo… ▽ More Integrating deep learning and causal discovery has increased the interpretability of Temporal Action Segmentation (TAS) tasks. However, frame-level causal relationships exist many complicated noises outside the segment-level, making it infeasible to directly express macro action semantics. Thus, we propose Causal Abstraction Segmentation Refiner (CASR), which can refine TAS results from various models by enhancing video causality in marginalizing frame-level casual relationships. Specifically, we define the equivalent frame-level casual model and segment-level causal model, so that the causal adjacency matrix constructed from marginalized frame-level causal relationships has the ability to represent the segmnet-level causal relationships. CASR works out by reducing the difference in the causal adjacency matrix between we constructed and pre-segmentation results of backbone models. In addition, we propose a novel evaluation metric Causal Edit Distance (CED) to evaluate the causal interpretability. Extensive experimental results on mainstream datasets indicate that CASR significantly surpasses existing various methods in action segmentation performance, as well as in causal explainability and generalization. △ Less

Submitted 26 January, 2024; v1 submitted 21 November, 2023; originally announced November 2023.

Comments: We found that the paper needs to be modified in the model and all experiments must be re-run, so we request to withdraw the current version

Hydrogen-induced tunable remanent polarization in a perovskite nickelate

Authors: Yifan Yuan, Michele Kotiuga, Tae Joon Park, Yuanyuan Ni, Arnob Saha, Hua Zhou, Jerzy T. Sadowski, Abdullah Al-Mahboob, Haoming Yu, Kai Du, Minning Zhu, Sunbin Deng, Ravindra S. Bisht, Xiao Lyu, Chung-Tse Michael Wu, Peide D. Ye, Abhronil Sengupta, Sang-Wook Cheong, Xiaoshan Xu, Karin M. Rabe, Shriram Ramanathan

Abstract: Materials with field-tunable polarization are of broad interest to condensed matter sciences and solid-state device technologies. Here, using hydrogen (H) donor do**, we modify the room temperature metallic phase of a perovskite nickelate NdNiO3 into an insulating phase with both metastable dipolar polarization and space-charge polarization. We then demonstrate transient negative differential ca… ▽ More Materials with field-tunable polarization are of broad interest to condensed matter sciences and solid-state device technologies. Here, using hydrogen (H) donor do**, we modify the room temperature metallic phase of a perovskite nickelate NdNiO3 into an insulating phase with both metastable dipolar polarization and space-charge polarization. We then demonstrate transient negative differential capacitance in thin film capacitors. The space-charge polarization caused by long-range movement and trap** of protons dominates when the electric field exceeds the threshold value. First-principles calculations suggest the polarization originates from the polar structure created by H do**. We find that polarization decays within ~1 second which is an interesting temporal regime for neuromorphic computing hardware design, and we implement the transient characteristics in a neural network to demonstrate unsupervised learning. These discoveries open new avenues for designing novel ferroelectric materials and electrets using light-ion do**. △ Less

Submitted 20 November, 2023; originally announced November 2023.

Comments: 13 pages, 5 figures

Self-interacting approximation to McKean-Vlasov long-time limit: a Markov chain Monte Carlo method

Authors: Kai Du, Zhenjie Ren, Florin Suciu, Songbo Wang

Abstract: For a certain class of McKean--Vlasov processes, we introduce proxy processes that substitute the mean-field interaction with self-interaction, employing a weighted occupation measure. Our study encompasses two key achievements. First, we demonstrate the ergodicity of the self-interacting dynamics, under broad conditions, by applying the reflection coupling method. Second, in scenarios where the d… ▽ More For a certain class of McKean--Vlasov processes, we introduce proxy processes that substitute the mean-field interaction with self-interaction, employing a weighted occupation measure. Our study encompasses two key achievements. First, we demonstrate the ergodicity of the self-interacting dynamics, under broad conditions, by applying the reflection coupling method. Second, in scenarios where the drifts are negative intrinsic gradients of convex mean-field potential functionals, we use entropy and functional inequalities to demonstrate that the stationary measures of the self-interacting processes approximate the invariant measures of the corresponding McKean--Vlasov processes. As an application, we show how to learn the optimal weights of a two-layer neural network by training a single neuron. △ Less

Submitted 14 January, 2024; v1 submitted 19 November, 2023; originally announced November 2023.

Comments: 42 pages, 1 figure; contains a minor correction

The \ion{H}{I}-rich Ultra-diffuse Galaxies follow the Extended Schmidt Law

Authors: Sai Zhai, Yong Shi, Zhi-Yu Zhang, Jun-Zhi Wang, Yu Gao, Qiusheng Gu, Tao Wang, Kaiyi Du, Xiaoling Yu, Xin Li

Abstract: The \ion{H}{I}-rich ultra-diffuse galaxies (HUDGs) offer a unique case for studies of star formation laws (SFLs) as they host low star formation efficiency (SFE) and low-metallicity environments where gas is predominantly atomic. We collect a sample of six HUDGs in the field and investigate their location in the extended Schmidt law(… ▽ More The \ion{H}{I}-rich ultra-diffuse galaxies (HUDGs) offer a unique case for studies of star formation laws (SFLs) as they host low star formation efficiency (SFE) and low-metallicity environments where gas is predominantly atomic. We collect a sample of six HUDGs in the field and investigate their location in the extended Schmidt law($Σ_{\text {SFR }} \propto \left(Σ_{\text{star}}^{0.5} Σ_{\text{gas}}\right)^{1.09}$). They are consistent with this relationship well (with deviations of only 1.1 sigma). Furthermore, we find that HUDGs follow the tight correlation between the hydrostatic pressure in the galaxy mid-plane and the quantity on the x-axis ($\rm log(Σ_{star}^{0.5}Σ_{gas})$) of the extended Schmidt law. This result indicates that these HUDGs can be self-regulated systems that reach the dynamical and thermal equilibrium. In this framework, the stellar gravity compresses the disk vertically and counteracts the gas pressure in the galaxy mid-plane to regulate the star formation as suggested by some theoretical models. △ Less

Submitted 13 November, 2023; originally announced November 2023.

Comments: 6 pages, 4 figures, accepted for publication in MNRAS

A Review and Roadmap of Deep Causal Model from Different Causal Structures and Representations

Authors: Hang Chen, Keqing Du, Chenguang Li, Xinyu Yang

Abstract: The fusion of causal models with deep learning introducing increasingly intricate data sets, such as the causal associations within images or between textual components, has surfaced as a focal research area. Nonetheless, the broadening of original causal concepts and theories to such complex, non-statistical data has been met with serious challenges. In response, our study proposes redefinitions… ▽ More The fusion of causal models with deep learning introducing increasingly intricate data sets, such as the causal associations within images or between textual components, has surfaced as a focal research area. Nonetheless, the broadening of original causal concepts and theories to such complex, non-statistical data has been met with serious challenges. In response, our study proposes redefinitions of causal data into three distinct categories from the standpoint of causal structure and representation: definite data, semi-definite data, and indefinite data. Definite data chiefly pertains to statistical data used in conventional causal scenarios, while semi-definite data refers to a spectrum of data formats germane to deep learning, including time-series, images, text, and others. Indefinite data is an emergent research sphere inferred from the progression of data forms by us. To comprehensively present these three data paradigms, we elaborate on their formal definitions, differences manifested in datasets, resolution pathways, and development of research. We summarize key tasks and achievements pertaining to definite and semi-definite data from myriad research undertakings, present a roadmap for indefinite data, beginning with its current research conundrums. Lastly, we classify and scrutinize the key datasets presently utilized within these three paradigms. △ Less

Submitted 1 November, 2023; originally announced November 2023.

Comments: under review

Empirical approximation to invariant measures of non-degenerate McKean-Vlasov dynamics

Authors: Wen**g Cao, Kai Du

Abstract: This paper studies the approximation of invariant measures of McKean-Vlasov dynamics with non-degenerate additive noise. While prior findings necessitated a strong monotonicity condition on the McKean-Vlasov process, we expand these results to encompass dissipative and weak interaction scenarios. Utilizing a reflection coupling technique, we prove that the empirical measures of the McKean-Vlasov p… ▽ More This paper studies the approximation of invariant measures of McKean-Vlasov dynamics with non-degenerate additive noise. While prior findings necessitated a strong monotonicity condition on the McKean-Vlasov process, we expand these results to encompass dissipative and weak interaction scenarios. Utilizing a reflection coupling technique, we prove that the empirical measures of the McKean-Vlasov process and its path-dependent counterpart can converge to the invariant measure in the Wasserstein metric. The Curie-Weiss mean-field lattice model serves as a numerical example to illustrate empirical approximation. △ Less

Submitted 23 January, 2024; v1 submitted 30 October, 2023; originally announced October 2023.

Comments: 21 pages, 1 figure; typos corrected, email address updated

MSC Class: 60B10; 37M25; 60F25; 60H10

SSL Framework for Causal Inconsistency between Structures and Representations

Authors: Hang Chen, Xinyu Yang, Keqing Du

Abstract: The cross-pollination of deep learning and causal discovery has catalyzed a burgeoning field of research seeking to elucidate causal relationships within non-statistical data forms like images, videos, and text. Such data, often being named `indefinite data', exhibit unique challenges-inconsistency between causal structure and representation, which are not common in conventional data forms. To tac… ▽ More The cross-pollination of deep learning and causal discovery has catalyzed a burgeoning field of research seeking to elucidate causal relationships within non-statistical data forms like images, videos, and text. Such data, often being named `indefinite data', exhibit unique challenges-inconsistency between causal structure and representation, which are not common in conventional data forms. To tackle this issue, we theoretically develop intervention strategies suitable for indefinite data and derive causal consistency condition (CCC). Moreover, we design a self-supervised learning (SSL) framework that considers interventions as `views' and CCC as a `philosophy' with two implement examples on Supervised Specialized Models (SSMs) and Large Language Models (LLMs), respectively. To evaluate pure inconsistency manifestations, we have prepared the first high-quality causal dialogue dataset-Causalogue. Evaluations are also performed on three other downstream tasks. Extensive experimentation has substantiated the efficacy of our methodology, illuminating how CCC could potentially play an influential role in various fields. △ Less

Submitted 28 October, 2023; originally announced October 2023.

CacheGen: KV Cache Compression and Streaming for Fast Language Model Serving

Authors: Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, Junchen Jiang

Abstract: As large language models (LLMs) take on complex tasks, their inputs are supplemented with longer contexts that incorporate domain knowledge or user-specific information. Yet using long contexts poses a challenge for responsive LLM systems, as nothing can be generated until the whole context is processed by the LLM. . CacheGen is a fast context-loading module for LLM systems. First, CacheGen uses… ▽ More As large language models (LLMs) take on complex tasks, their inputs are supplemented with longer contexts that incorporate domain knowledge or user-specific information. Yet using long contexts poses a challenge for responsive LLM systems, as nothing can be generated until the whole context is processed by the LLM. . CacheGen is a fast context-loading module for LLM systems. First, CacheGen uses a custom tensor encoder, which embraces KV cache's distributional properties, to encode a KV cache into more compact bitstream representations with negligible encoding/decoding overhead. This reduces the bandwidth demand to fetch the KV cache. Second, to maintain low context-loading delay and high generation quality, CacheGen adapts the streaming strategies to cope with changes in available bandwidth. When available bandwidth drops, CacheGen may raise the compression level for a part of the context or choose to recompute its KV cache on the fly. We test CacheGen on four popular LLMs of various sizes and four datasets (662 contexts in total). Compared to the recent systems that reuse the KV cache, CacheGen reduces the KV cache size by 3.5-4.3x and the total delay in fetching and processing contexts by 3.2-3.7x while having negligible impact on the LLM response quality in accuracy or perplexity. △ Less

Submitted 30 April, 2024; v1 submitted 11 October, 2023; originally announced October 2023.

Automatic and Efficient Customization of Neural Networks for ML Applications

Authors: Yuhan Liu, Chengcheng Wan, Kuntai Du, Henry Hoffmann, Junchen Jiang, Shan Lu, Michael Maire

Abstract: ML APIs have greatly relieved application developers of the burden to design and train their own neural network models -- classifying objects in an image can now be as simple as one line of Python code to call an API. However, these APIs offer the same pre-trained models regardless of how their output is used by different applications. This can be suboptimal as not all ML inference errors can caus… ▽ More ML APIs have greatly relieved application developers of the burden to design and train their own neural network models -- classifying objects in an image can now be as simple as one line of Python code to call an API. However, these APIs offer the same pre-trained models regardless of how their output is used by different applications. This can be suboptimal as not all ML inference errors can cause application failures, and the distinction between inference errors that can or cannot cause failures varies greatly across applications. To tackle this problem, we first study 77 real-world applications, which collectively use six ML APIs from two providers, to reveal common patterns of how ML API output affects applications' decision processes. Inspired by the findings, we propose ChameleonAPI, an optimization framework for ML APIs, which takes effect without changing the application source code. ChameleonAPI provides application developers with a parser that automatically analyzes the application to produce an abstract of its decision process, which is then used to devise an application-specific loss function that only penalizes API output errors critical to the application. ChameleonAPI uses the loss function to efficiently train a neural network model customized for each application and deploys it to serve API invocations from the respective application via existing interface. Compared to a baseline that selects the best-of-all commercial ML API, we show that ChameleonAPI reduces incorrect application decisions by 43%. △ Less

Submitted 7 October, 2023; originally announced October 2023.

Spin-Mediated Direct Photon Scattering by Plasmons in BiTeI

Authors: A. C. Lee, S. Sarkar, K. Du, H. -H. Kung, C. J. Won, K. Wang, S. -W. Cheong, S. Maiti, G. Blumberg

Abstract: We use polarization resolved Raman spectroscopy to demonstrate that for a 3D giant Rashba system the bulk plasmon collective mode can directly couple to the Raman response even in the long wavelength $\mathbf q \rightarrow 0$ limit. Although conventional theory predicts the plasmon spectral weight to be suppressed as the square of its quasi-momentum and thus negligibly weak in the Raman spectra, w… ▽ More We use polarization resolved Raman spectroscopy to demonstrate that for a 3D giant Rashba system the bulk plasmon collective mode can directly couple to the Raman response even in the long wavelength $\mathbf q \rightarrow 0$ limit. Although conventional theory predicts the plasmon spectral weight to be suppressed as the square of its quasi-momentum and thus negligibly weak in the Raman spectra, we observe a sharp in-gap plasmon mode in the Raman spectrum of BiTeI below the Rashba continuum. This coupling, in a polar system with spin-orbit coupling, occurs without assistance from phonons when the incoming photon excitation is resonant with Rashba-split intermediate states. We discuss the distinctive features of BiTeI's giant Rashba system band structure that enable the direct observation of plasmon in Raman scattering. △ Less

Submitted 18 February, 2024; v1 submitted 6 October, 2023; originally announced October 2023.

Comments: Editors' Suggestion

Journal ref: Phys. Rev. B 109, L041111 (2024)

OneAdapt: Fast Adaptation for Deep Learning Applications via Backpropagation

Authors: Kuntai Du, Yuhan Liu, Yitian Hao, Qizheng Zhang, Haodong Wang, Yuyang Huang, Ganesh Ananthanarayanan, Junchen Jiang

Abstract: Deep learning inference on streaming media data, such as object detection in video or LiDAR feeds and text extraction from audio waves, is now ubiquitous. To achieve high inference accuracy, these applications typically require significant network bandwidth to gather high-fidelity data and extensive GPU resources to run deep neural networks (DNNs). While the high demand for network bandwidth and G… ▽ More Deep learning inference on streaming media data, such as object detection in video or LiDAR feeds and text extraction from audio waves, is now ubiquitous. To achieve high inference accuracy, these applications typically require significant network bandwidth to gather high-fidelity data and extensive GPU resources to run deep neural networks (DNNs). While the high demand for network bandwidth and GPU resources could be substantially reduced by optimally adapting the configuration knobs, such as video resolution and frame rate, current adaptation techniques fail to meet three requirements simultaneously: adapt configurations (i) with minimum extra GPU or bandwidth overhead; (ii) to reach near-optimal decisions based on how the data affects the final DNN's accuracy, and (iii) do so for a range of configuration knobs. This paper presents OneAdapt, which meets these requirements by leveraging a gradient-ascent strategy to adapt configuration knobs. The key idea is to embrace DNNs' differentiability to quickly estimate the accuracy's gradient to each configuration knob, called AccGrad. Specifically, OneAdapt estimates AccGrad by multiplying two gradients: InputGrad (i.e. how each configuration knob affects the input to the DNN) and DNNGrad (i.e. how the DNN input affects the DNN inference output). We evaluate OneAdapt across five types of configurations, four analytic tasks, and five types of input data. Compared to state-of-the-art adaptation schemes, OneAdapt cuts bandwidth usage and GPU usage by 15-59% while maintaining comparable accuracy or improves accuracy by 1-5% while using equal or fewer resources. △ Less

Submitted 3 October, 2023; originally announced October 2023.

Comments: SoCC' 23

CHITNet: A Complementary to Harmonious Information Transfer Network for Infrared and Visible Image Fusion

Authors: Yafei Zhang, Keying Du, Huafeng Li, Zhengtao Yu, Yu Liu

Abstract: Current infrared and visible image fusion (IVIF) methods go to great lengths to excavate complementary features and design complex fusion strategies, which is extremely challenging. To this end, we rethink the IVIF outside the box, proposing a complementary to harmonious information transfer network (CHITNet). It reasonably transfers complementary information into harmonious one, which integrates… ▽ More Current infrared and visible image fusion (IVIF) methods go to great lengths to excavate complementary features and design complex fusion strategies, which is extremely challenging. To this end, we rethink the IVIF outside the box, proposing a complementary to harmonious information transfer network (CHITNet). It reasonably transfers complementary information into harmonious one, which integrates both the shared and complementary features from two modalities. Specifically, to skillfully sidestep aggregating complementary information in IVIF, we design a mutual information transfer (MIT) module to mutually represent features from two modalities, roughly transferring complementary information into harmonious one. Then, a harmonious information acquisition supervised by source image (HIASSI) module is devised to further ensure the complementary to harmonious information transfer after MIT. Meanwhile, we also propose a structure information preservation (SIP) module to guarantee that the edge structure information of the source images can be transferred to the fusion results. Moreover, a mutual promotion training paradigm (MPTP) with interaction loss is adopted to facilitate better collaboration among MIT, HIASSI and SIP. In this way, the proposed method is able to generate fused images with higher qualities. Extensive experimental results demonstrate the superiority of our CHITNet over state-of-the-art algorithms in terms of visual quality and quantitative evaluations. △ Less

Submitted 25 October, 2023; v1 submitted 12 September, 2023; originally announced September 2023.

PhotoVerse: Tuning-Free Image Customization with Text-to-Image Diffusion Models

Authors: Li Chen, Mengyi Zhao, Yiheng Liu, Mingxu Ding, Yangyang Song, Shizun Wang, Xu Wang, Hao Yang, **g Liu, Kang Du, Min Zheng

Abstract: Personalized text-to-image generation has emerged as a powerful and sought-after tool, empowering users to create customized images based on their specific concepts and prompts. However, existing approaches to personalization encounter multiple challenges, including long tuning times, large storage requirements, the necessity for multiple input images per identity, and limitations in preserving id… ▽ More Personalized text-to-image generation has emerged as a powerful and sought-after tool, empowering users to create customized images based on their specific concepts and prompts. However, existing approaches to personalization encounter multiple challenges, including long tuning times, large storage requirements, the necessity for multiple input images per identity, and limitations in preserving identity and editability. To address these obstacles, we present PhotoVerse, an innovative methodology that incorporates a dual-branch conditioning mechanism in both text and image domains, providing effective control over the image generation process. Furthermore, we introduce facial identity loss as a novel component to enhance the preservation of identity during training. Remarkably, our proposed PhotoVerse eliminates the need for test time tuning and relies solely on a single facial photo of the target identity, significantly reducing the resource cost associated with image generation. After a single training phase, our approach enables generating high-quality images within only a few seconds. Moreover, our method can produce diverse images that encompass various scenes and styles. The extensive evaluation demonstrates the superior performance of our approach, which achieves the dual objectives of preserving identity and facilitating editability. Project page: https://photoverse2d.github.io/ △ Less

Submitted 11 September, 2023; originally announced September 2023.

CodeApex: A Bilingual Programming Evaluation Benchmark for Large Language Models

Authors: Lingyue Fu, Huacan Chai, Shuang Luo, Kounianhua Du, Weiming Zhang, Longteng Fan, Jiayi Lei, Renting Rui, Jianghao Lin, Yuchen Fang, Yifan Liu, **gkuan Wang, Siyuan Qi, Kangning Zhang, Weinan Zhang, Yong Yu

Abstract: With the emergence of Large Language Models (LLMs), there has been a significant improvement in the programming capabilities of models, attracting growing attention from researchers. Evaluating the programming capabilities of LLMs is crucial as it reflects the multifaceted abilities of LLMs, and it has numerous downstream applications. In this paper, we propose CodeApex, a bilingual benchmark data… ▽ More With the emergence of Large Language Models (LLMs), there has been a significant improvement in the programming capabilities of models, attracting growing attention from researchers. Evaluating the programming capabilities of LLMs is crucial as it reflects the multifaceted abilities of LLMs, and it has numerous downstream applications. In this paper, we propose CodeApex, a bilingual benchmark dataset focusing on the programming comprehension, code generation, and code correction abilities of LLMs. Programming comprehension task tests LLMs on multiple-choice exam questions covering conceptual understanding, commonsense reasoning, and multi-hop reasoning. The code generation task evaluates LLMs through completing C++ functions based on provided descriptions and prototypes. The code correction task asks LLMs to fix real-world erroneous code segments with different error messages. We evaluate 12 widely used LLMs, including both general-purpose and specialized models. GPT-4 exhibits the best programming capabilities, achieving approximate accuracy of 69%, 54%, and 66% on the three tasks, respectively. Compared to human performance, there is still significant room for improvement in LLM programming. We hope that CodeApex can serve as a reference for evaluating the coding capabilities of LLMs, further promoting their development and growth. △ Less

Submitted 11 March, 2024; v1 submitted 5 September, 2023; originally announced September 2023.

Comments: 33pages

ReLLa: Retrieval-enhanced Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendation

Authors: Jianghao Lin, Rong Shan, Chenxu Zhu, Kounianhua Du, Bo Chen, Shigang Quan, Ruiming Tang, Yong Yu, Weinan Zhang

Abstract: With large language models (LLMs) achieving remarkable breakthroughs in natural language processing (NLP) domains, LLM-enhanced recommender systems have received much attention and have been actively explored currently. In this paper, we focus on adapting and empowering a pure large language model for zero-shot and few-shot recommendation tasks. First and foremost, we identify and formulate the li… ▽ More With large language models (LLMs) achieving remarkable breakthroughs in natural language processing (NLP) domains, LLM-enhanced recommender systems have received much attention and have been actively explored currently. In this paper, we focus on adapting and empowering a pure large language model for zero-shot and few-shot recommendation tasks. First and foremost, we identify and formulate the lifelong sequential behavior incomprehension problem for LLMs in recommendation domains, i.e., LLMs fail to extract useful information from a textual context of long user behavior sequence, even if the length of context is far from reaching the context limitation of LLMs. To address such an issue and improve the recommendation performance of LLMs, we propose a novel framework, namely Retrieval-enhanced Large Language models (ReLLa) for recommendation tasks in both zero-shot and few-shot settings. For zero-shot recommendation, we perform semantic user behavior retrieval (SUBR) to improve the data quality of testing samples, which greatly reduces the difficulty for LLMs to extract the essential knowledge from user behavior sequences. As for few-shot recommendation, we further design retrieval-enhanced instruction tuning (ReiT) by adopting SUBR as a data augmentation technique for training samples. Specifically, we develop a mixed training dataset consisting of both the original data samples and their retrieval-enhanced counterparts. We conduct extensive experiments on three real-world public datasets to demonstrate the superiority of ReLLa compared with existing baseline models, as well as its capability for lifelong sequential behavior comprehension. To be highlighted, with only less than 10% training samples, few-shot ReLLa can outperform traditional CTR models that are trained on the entire training set (e.g., DCNv2, DIN, SIM). The code is available \url{https://github.com/LaVieEnRose365/ReLLa}. △ Less

Submitted 26 June, 2024; v1 submitted 21 August, 2023; originally announced August 2023.

Comments: Accepted by WWW 2024. Full and More Readable Version

AvatarVerse: High-quality & Stable 3D Avatar Creation from Text and Pose

Authors: Huichao Zhang, Bowen Chen, Hao Yang, Liao Qu, Xu Wang, Li Chen, Chao Long, Feida Zhu, Kang Du, Min Zheng

Abstract: Creating expressive, diverse and high-quality 3D avatars from highly customized text descriptions and pose guidance is a challenging task, due to the intricacy of modeling and texturing in 3D that ensure details and various styles (realistic, fictional, etc). We present AvatarVerse, a stable pipeline for generating expressive high-quality 3D avatars from nothing but text descriptions and pose guid… ▽ More Creating expressive, diverse and high-quality 3D avatars from highly customized text descriptions and pose guidance is a challenging task, due to the intricacy of modeling and texturing in 3D that ensure details and various styles (realistic, fictional, etc). We present AvatarVerse, a stable pipeline for generating expressive high-quality 3D avatars from nothing but text descriptions and pose guidance. In specific, we introduce a 2D diffusion model conditioned on DensePose signal to establish 3D pose control of avatars through 2D images, which enhances view consistency from partially observed scenarios. It addresses the infamous Janus Problem and significantly stablizes the generation process. Moreover, we propose a progressive high-resolution 3D synthesis strategy, which obtains substantial improvement over the quality of the created 3D avatars. To this end, the proposed AvatarVerse pipeline achieves zero-shot 3D modeling of 3D avatars that are not only more expressive, but also in higher quality and fidelity than previous works. Rigorous qualitative evaluations and user studies showcase AvatarVerse's superiority in synthesizing high-fidelity 3D avatars, leading to a new standard in high-quality and stable 3D avatar creation. Our project page is: https://avatarverse3d.github.io △ Less

Submitted 7 August, 2023; originally announced August 2023.

Counterfactual Graph Transformer for Traffic Flow Prediction

Authors: Ying Yang, Kai Du, Xingyuan Dai, Jianwu Fang

Abstract: Traffic flow prediction (TFP) is a fundamental problem of the Intelligent Transportation System (ITS), as it models the latent spatial-temporal dependency of traffic flow for potential congestion prediction. Recent graph-based models with multiple kinds of attention mechanisms have achieved promising performance. However, existing methods for traffic flow prediction tend to inherit the bias patter… ▽ More Traffic flow prediction (TFP) is a fundamental problem of the Intelligent Transportation System (ITS), as it models the latent spatial-temporal dependency of traffic flow for potential congestion prediction. Recent graph-based models with multiple kinds of attention mechanisms have achieved promising performance. However, existing methods for traffic flow prediction tend to inherit the bias pattern from the dataset and lack interpretability. To this end, we propose a Counterfactual Graph Transformer (CGT) model with an instance-level explainer (e.g., finding the important subgraphs) specifically designed for TFP. We design a perturbation mask generator over input sensor features at the time dimension and the graph structure on the graph transformer module to obtain spatial and temporal counterfactual explanations. By searching the optimal perturbation masks on the input data feature and graph structures, we can obtain the concise and dominant data or graph edge links for the subsequent TFP task. After re-training the utilized graph transformer model after counterfactual perturbation, we can obtain improved and interpretable traffic flow prediction. Extensive results on three real-world public datasets show that CGT can produce reliable explanations and is promising for traffic flow prediction. △ Less

Submitted 1 August, 2023; originally announced August 2023.

Comments: accepted by ITSC 2023

On Krylov subspace methods for skew-symmetric and shifted skew-symmetric linear systems

Authors: Kui Du, Jia-Jun Fan, Xiao-Hui Sun, Fang Wang, Ya-Lan Zhang

Abstract: Krylov subspace methods for solving linear systems of equations involving skew-symmetric matrices have gained recent attention. Numerical equivalences among Krylov subspace methods for nonsingular skew-symmetric linear systems have been given in Greif et al. [SIAM J. Matrix Anal. Appl., 37 (2016), pp. 1071--1087]. In this work, we extend the results of Greif et al. to singular skew-symmetric linea… ▽ More Krylov subspace methods for solving linear systems of equations involving skew-symmetric matrices have gained recent attention. Numerical equivalences among Krylov subspace methods for nonsingular skew-symmetric linear systems have been given in Greif et al. [SIAM J. Matrix Anal. Appl., 37 (2016), pp. 1071--1087]. In this work, we extend the results of Greif et al. to singular skew-symmetric linear systems. In addition, we systematically study three Krylov subspace methods (called S$^3$CG, S$^3$MR, and S$^3$LQ) for solving shifted skew-symmetric linear systems. They all are based on Lanczos triangularization for skew-symmetric matrices, and correspond to CG, MINRES, and SYMMLQ for solving symmetric linear systems, respectively. To the best of our knowledge, this is the first work that studies S$^3$LQ. We give some new theoretical results on S$^3$CG, S$^3$MR, and S$^3$LQ. We also provide the relationship among the three methods and those based on Golub--Kahan bidiagonalization and Saunders--Simon--Yip tridiagonalization. Numerical examples are given to illustrate our theoretical findings. △ Less

Submitted 31 July, 2023; originally announced July 2023.

Comments: 23 pages, 3 figures

Light-induced electronic polarization in antiferromagnetic Cr2O3

Authors: Xinshu Zhang, Tyler Carbin, Adrian B. Culver, Kai Du, Kefeng Wang, Sang-Wook Cheong, Rahul Roy, Anshul Kogar

Abstract: In a solid, the electronic subsystem can exhibit incipient order with lower point group symmetry than the crystal lattice. External fields that couple to electronic order parameters have rarely been investigated, however, despite their potential importance to inducing exotic effects. Here, we show that when inversion symmetry is broken by the antiferromagnetic (AFM) order in Cr2O3, transmitting a… ▽ More In a solid, the electronic subsystem can exhibit incipient order with lower point group symmetry than the crystal lattice. External fields that couple to electronic order parameters have rarely been investigated, however, despite their potential importance to inducing exotic effects. Here, we show that when inversion symmetry is broken by the antiferromagnetic (AFM) order in Cr2O3, transmitting a linearly polarized light pulse through the crystal gives rise to an in-plane rotational symmetry breaking (from C3 to C1) via optical rectification. Using interferometric time-resolved second harmonic generation, we show that the ultrafast timescale of the symmetry reduction is indicative of a purely electronic response; the underlying spin and crystal structures remain unaffected. The symmetry-broken state exhibits a dipole moment, and its polar axis can be controlled with the incident light. Our results establish a coherent nonlinear optical protocol by which to break electronic symmetries and produce unconventional electronic effects in solids. △ Less

Submitted 9 January, 2024; v1 submitted 21 July, 2023; originally announced July 2023.