-
SOOD++: Leveraging Unlabeled Data to Boost Oriented Object Detection
Authors:
Dingkang Liang,
Wei Hua,
Chunsheng Shi,
Zhikang Zou,
Xiaoqing Ye,
Xiang Bai
Abstract:
Semi-supervised object detection (SSOD), leveraging unlabeled data to boost object detectors, has become a hot topic recently. However, existing SSOD approaches mainly focus on horizontal objects, leaving multi-oriented objects common in aerial images unexplored. At the same time, the annotation cost of multi-oriented objects is significantly higher than that of their horizontal counterparts. Ther…
▽ More
Semi-supervised object detection (SSOD), leveraging unlabeled data to boost object detectors, has become a hot topic recently. However, existing SSOD approaches mainly focus on horizontal objects, leaving multi-oriented objects common in aerial images unexplored. At the same time, the annotation cost of multi-oriented objects is significantly higher than that of their horizontal counterparts. Therefore, in this paper, we propose a simple yet effective Semi-supervised Oriented Object Detection method termed SOOD++. Specifically, we observe that objects from aerial images are usually arbitrary orientations, small scales, and aggregation, which inspires the following core designs: a Simple Instance-aware Dense Sampling (SIDS) strategy is used to generate comprehensive dense pseudo-labels; the Geometry-aware Adaptive Weighting (GAW) loss dynamically modulates the importance of each pair between pseudo-label and corresponding prediction by leveraging the intricate geometric information of aerial objects; we treat aerial images as global layouts and explicitly build the many-to-many relationship between the sets of pseudo-labels and predictions via the proposed Noise-driven Global Consistency (NGC). Extensive experiments conducted on various multi-oriented object datasets under various labeled settings demonstrate the effectiveness of our method. For example, on the DOTA-V1.5 benchmark, the proposed method outperforms previous state-of-the-art (SOTA) by a large margin (+2.92, +2.39, and +2.57 mAP under 10%, 20%, and 30% labeled data settings, respectively) with single-scale training and testing. More importantly, it still improves upon a strong supervised baseline with 70.66 mAP, trained using the full DOTA-V1.5 train-val set, by +1.82 mAP, resulting in a 72.48 mAP, pushing the new state-of-the-art. The code will be made available.
△ Less
Submitted 1 July, 2024;
originally announced July 2024.
-
MultiAgent Collaboration Attack: Investigating Adversarial Attacks in Large Language Model Collaborations via Debate
Authors:
Alfonso Amayuelas,
Xianjun Yang,
Antonis Antoniades,
Wenyue Hua,
Liangming Pan,
William Wang
Abstract:
Large Language Models (LLMs) have shown exceptional results on current benchmarks when working individually. The advancement in their capabilities, along with a reduction in parameter size and inference times, has facilitated the use of these models as agents, enabling interactions among multiple models to execute complex tasks. Such collaborations offer several advantages, including the use of sp…
▽ More
Large Language Models (LLMs) have shown exceptional results on current benchmarks when working individually. The advancement in their capabilities, along with a reduction in parameter size and inference times, has facilitated the use of these models as agents, enabling interactions among multiple models to execute complex tasks. Such collaborations offer several advantages, including the use of specialized models (e.g. coding), improved confidence through multiple computations, and enhanced divergent thinking, leading to more diverse outputs. Thus, the collaborative use of language models is expected to grow significantly in the coming years. In this work, we evaluate the behavior of a network of models collaborating through debate under the influence of an adversary. We introduce pertinent metrics to assess the adversary's effectiveness, focusing on system accuracy and model agreement. Our findings highlight the importance of a model's persuasive ability in influencing others. Additionally, we explore inference-time methods to generate more compelling arguments and evaluate the potential of prompt-based mitigation as a defensive strategy.
△ Less
Submitted 26 June, 2024; v1 submitted 20 June, 2024;
originally announced June 2024.
-
MoralBench: Moral Evaluation of LLMs
Authors:
Jianchao Ji,
Yutong Chen,
Mingyu **,
Wujiang Xu,
Wenyue Hua,
Yongfeng Zhang
Abstract:
In the rapidly evolving field of artificial intelligence, large language models (LLMs) have emerged as powerful tools for a myriad of applications, from natural language processing to decision-making support systems. However, as these models become increasingly integrated into societal frameworks, the imperative to ensure they operate within ethical and moral boundaries has never been more critica…
▽ More
In the rapidly evolving field of artificial intelligence, large language models (LLMs) have emerged as powerful tools for a myriad of applications, from natural language processing to decision-making support systems. However, as these models become increasingly integrated into societal frameworks, the imperative to ensure they operate within ethical and moral boundaries has never been more critical. This paper introduces a novel benchmark designed to measure and compare the moral reasoning capabilities of LLMs. We present the first comprehensive dataset specifically curated to probe the moral dimensions of LLM outputs, addressing a wide range of ethical dilemmas and scenarios reflective of real-world complexities.
The main contribution of this work lies in the development of benchmark datasets and metrics for assessing the moral identity of LLMs, which accounts for nuance, contextual sensitivity, and alignment with human ethical standards. Our methodology involves a multi-faceted approach, combining quantitative analysis with qualitative insights from ethics scholars to ensure a thorough evaluation of model performance. By applying our benchmark across several leading LLMs, we uncover significant variations in moral reasoning capabilities of different models. These findings highlight the importance of considering moral reasoning in the development and evaluation of LLMs, as well as the need for ongoing research to address the biases and limitations uncovered in our study. We publicly release the benchmark at https://drive.google.com/drive/u/0/folders/1k93YZJserYc2CkqP8d4B3M3sgd3kA8W7 and also open-source the code of the project at https://github.com/agiresearch/MoralBench.
△ Less
Submitted 6 June, 2024;
originally announced June 2024.
-
Disentangling Logic: The Role of Context in Large Language Model Reasoning Capabilities
Authors:
Wenyue Hua,
Kaijie Zhu,
Lingyao Li,
Lizhou Fan,
Shuhang Lin,
Mingyu **,
Haochen Xue,
Zelong Li,
**Dong Wang,
Yongfeng Zhang
Abstract:
This study intends to systematically disentangle pure logic reasoning and text understanding by investigating the contrast across abstract and contextualized logical problems from a comprehensive set of domains. We explore whether LLMs demonstrate genuine reasoning capabilities across various domains when the underlying logical structure remains constant. We focus on two main questions (1) Can abs…
▽ More
This study intends to systematically disentangle pure logic reasoning and text understanding by investigating the contrast across abstract and contextualized logical problems from a comprehensive set of domains. We explore whether LLMs demonstrate genuine reasoning capabilities across various domains when the underlying logical structure remains constant. We focus on two main questions (1) Can abstract logical problems alone accurately benchmark an LLM's reasoning ability in real-world scenarios, disentangled from contextual support in practical settings? (2) Does fine-tuning LLMs on abstract logic problem generalize to contextualized logic problems and vice versa? To investigate these questions, we focus on standard propositional logic, specifically propositional deductive and abductive logic reasoning. In particular, we construct instantiated datasets for deductive and abductive reasoning with 4 levels of difficulty, encompassing 12 distinct categories or domains based on the categorization of Wikipedia. Our experiments aim to provide insights into disentangling context in logical reasoning and the true reasoning capabilities of LLMs and their generalization potential. The code and dataset are available at: https://github.com/agiresearch/ContextHub.
△ Less
Submitted 4 June, 2024;
originally announced June 2024.
-
Entity Alignment with Noisy Annotations from Large Language Models
Authors:
Shengyuan Chen,
Qinggang Zhang,
Junnan Dong,
Wen Hua,
Qing Li,
Xiao Huang
Abstract:
Entity alignment (EA) aims to merge two knowledge graphs (KGs) by identifying equivalent entity pairs. While existing methods heavily rely on human-generated labels, it is prohibitively expensive to incorporate cross-domain experts for annotation in real-world scenarios. The advent of Large Language Models (LLMs) presents new avenues for automating EA with annotations, inspired by their comprehens…
▽ More
Entity alignment (EA) aims to merge two knowledge graphs (KGs) by identifying equivalent entity pairs. While existing methods heavily rely on human-generated labels, it is prohibitively expensive to incorporate cross-domain experts for annotation in real-world scenarios. The advent of Large Language Models (LLMs) presents new avenues for automating EA with annotations, inspired by their comprehensive capability to process semantic information. However, it is nontrivial to directly apply LLMs for EA since the annotation space in real-world KGs is large. LLMs could also generate noisy labels that may mislead the alignment. To this end, we propose a unified framework, LLM4EA, to effectively leverage LLMs for EA. Specifically, we design a novel active learning policy to significantly reduce the annotation space by prioritizing the most valuable entities based on the entire inter-KG and intra-KG structure. Moreover, we introduce an unsupervised label refiner to continuously enhance label accuracy through in-depth probabilistic reasoning. We iteratively optimize the policy based on the feedback from a base EA model. Extensive experiments demonstrate the advantages of LLM4EA on four benchmark datasets in terms of effectiveness, robustness, and efficiency. Codes are available via https://github.com/chensyCN/llm4ea_official.
△ Less
Submitted 28 May, 2024; v1 submitted 26 May, 2024;
originally announced May 2024.
-
A sco** review of using Large Language Models (LLMs) to investigate Electronic Health Records (EHRs)
Authors:
Lingyao Li,
Jiayan Zhou,
Zhenxiang Gao,
Wenyue Hua,
Lizhou Fan,
Huizi Yu,
Loni Hagen,
Yongfeng Zhang,
Themistocles L. Assimes,
Libby Hemphill,
Siyuan Ma
Abstract:
Electronic Health Records (EHRs) play an important role in the healthcare system. However, their complexity and vast volume pose significant challenges to data interpretation and analysis. Recent advancements in Artificial Intelligence (AI), particularly the development of Large Language Models (LLMs), open up new opportunities for researchers in this domain. Although prior studies have demonstrat…
▽ More
Electronic Health Records (EHRs) play an important role in the healthcare system. However, their complexity and vast volume pose significant challenges to data interpretation and analysis. Recent advancements in Artificial Intelligence (AI), particularly the development of Large Language Models (LLMs), open up new opportunities for researchers in this domain. Although prior studies have demonstrated their potential in language understanding and processing in the context of EHRs, a comprehensive sco** review is lacking. This study aims to bridge this research gap by conducting a sco** review based on 329 related papers collected from OpenAlex. We first performed a bibliometric analysis to examine paper trends, model applications, and collaboration networks. Next, we manually reviewed and categorized each paper into one of the seven identified topics: named entity recognition, information extraction, text similarity, text summarization, text classification, dialogue system, and diagnosis and prediction. For each topic, we discussed the unique capabilities of LLMs, such as their ability to understand context, capture semantic relations, and generate human-like text. Finally, we highlighted several implications for researchers from the perspectives of data resources, prompt engineering, fine-tuning, performance measures, and ethical concerns. In conclusion, this study provides valuable insights into the potential of LLMs to transform EHR research and discusses their applications and ethical considerations.
△ Less
Submitted 22 May, 2024; v1 submitted 5 May, 2024;
originally announced May 2024.
-
BattleAgent: Multi-modal Dynamic Emulation on Historical Battles to Complement Historical Analysis
Authors:
Shuhang Lin,
Wenyue Hua,
Lingyao Li,
Che-Jui Chang,
Lizhou Fan,
Jianchao Ji,
Hang Hua,
Mingyu **,
Jiebo Luo,
Yongfeng Zhang
Abstract:
This paper presents BattleAgent, an emulation system that combines the Large Vision-Language Model and Multi-agent System. This novel system aims to simulate complex dynamic interactions among multiple agents, as well as between agents and their environments, over a period of time. It emulates both the decision-making processes of leaders and the viewpoints of ordinary participants, such as soldie…
▽ More
This paper presents BattleAgent, an emulation system that combines the Large Vision-Language Model and Multi-agent System. This novel system aims to simulate complex dynamic interactions among multiple agents, as well as between agents and their environments, over a period of time. It emulates both the decision-making processes of leaders and the viewpoints of ordinary participants, such as soldiers. The emulation showcases the current capabilities of agents, featuring fine-grained multi-modal interactions between agents and landscapes. It develops customizable agent structures to meet specific situational requirements, for example, a variety of battle-related activities like scouting and trench digging. These components collaborate to recreate historical events in a lively and comprehensive manner while offering insights into the thoughts and feelings of individuals from diverse viewpoints. The technological foundations of BattleAgent establish detailed and immersive settings for historical battles, enabling individual agents to partake in, observe, and dynamically respond to evolving battle scenarios. This methodology holds the potential to substantially deepen our understanding of historical events, particularly through individual accounts. Such initiatives can also aid historical research, as conventional historical narratives often lack documentation and prioritize the perspectives of decision-makers, thereby overlooking the experiences of ordinary individuals. BattelAgent illustrates AI's potential to revitalize the human aspect in crucial social events, thereby fostering a more nuanced collective understanding and driving the progressive development of human society.
△ Less
Submitted 23 April, 2024;
originally announced April 2024.
-
Exploring Concept Depth: How Large Language Models Acquire Knowledge at Different Layers?
Authors:
Mingyu **,
Qinkai Yu,
**gyuan Huang,
Qingcheng Zeng,
Zhenting Wang,
Wenyue Hua,
Haiyan Zhao,
Kai Mei,
Yanda Meng,
Kaize Ding,
Fan Yang,
Mengnan Du,
Yongfeng Zhang
Abstract:
Large language models (LLMs) have shown remarkable performances across a wide range of tasks. However, the mechanisms by which these models encode tasks of varying complexities remain poorly understood. In this paper, we explore the hypothesis that LLMs process concepts of varying complexities in different layers, introducing the idea of "Concept Depth" to suggest that more complex concepts are ty…
▽ More
Large language models (LLMs) have shown remarkable performances across a wide range of tasks. However, the mechanisms by which these models encode tasks of varying complexities remain poorly understood. In this paper, we explore the hypothesis that LLMs process concepts of varying complexities in different layers, introducing the idea of "Concept Depth" to suggest that more complex concepts are typically acquired in deeper layers. Specifically, we categorize concepts based on their level of abstraction, defining them in the order of increasing complexity within factual, emotional, and inferential tasks. We conduct extensive probing experiments using layer-wise representations across various LLM families (Gemma, LLaMA, QWen) on various datasets spanning the three domains of tasks. Our findings reveal that models could efficiently conduct probing for simpler tasks in shallow layers, and more complex tasks typically necessitate deeper layers for accurate understanding. Additionally, we examine how external factors, such as adding noise to the input and quantizing the model weights, might affect layer-wise representations. Our findings suggest that these factors can impede the development of a conceptual understanding of LLMs until deeper layers are explored. We hope that our proposed concept and experimental insights will enhance the understanding of the mechanisms underlying LLMs. Our codes are available at https://github.com/Luckfort/CD.
△ Less
Submitted 30 April, 2024; v1 submitted 10 April, 2024;
originally announced April 2024.
-
Roadside Monocular 3D Detection via 2D Detection Prompting
Authors:
Yechi Ma,
Shuoquan Wei,
Churun Zhang,
Wei Hua,
Yanan Li,
Shu Kong
Abstract:
The problem of roadside monocular 3D detection requires detecting objects of interested classes in a 2D RGB frame and predicting their 3D information such as locations in bird's-eye-view (BEV). It has broad applications in traffic control, vehicle-vehicle communication, and vehicle-infrastructure cooperative perception. To approach this problem, we present a novel and simple method by prompting th…
▽ More
The problem of roadside monocular 3D detection requires detecting objects of interested classes in a 2D RGB frame and predicting their 3D information such as locations in bird's-eye-view (BEV). It has broad applications in traffic control, vehicle-vehicle communication, and vehicle-infrastructure cooperative perception. To approach this problem, we present a novel and simple method by prompting the 3D detector using 2D detections. Our method builds on a key insight that, compared with 3D detectors, a 2D detector is much easier to train and performs significantly better w.r.t detections on the 2D image plane. That said, one can exploit 2D detections of a well-trained 2D detector as prompts to a 3D detector, being trained in a way of inflating such 2D detections to 3D towards 3D detection. To construct better prompts using the 2D detector, we explore three techniques: (a) concatenating both 2D and 3D detectors' features, (b) attentively fusing 2D and 3D detectors' features, and (c) encoding predicted 2D boxes x, y, width, height, label and attentively fusing such with the 3D detector's features. Surprisingly, the third performs the best. Moreover, we present a yaw tuning tactic and a class-grou** strategy that merges classes based on their functionality; these techniques improve 3D detection performance further. Comprehensive ablation studies and extensive experiments demonstrate that our method resoundingly outperforms prior works, achieving the state-of-the-art on two large-scale roadside 3D detection benchmarks.
△ Less
Submitted 4 April, 2024; v1 submitted 1 April, 2024;
originally announced April 2024.
-
IDGenRec: LLM-RecSys Alignment with Textual ID Learning
Authors:
Juntao Tan,
Shuyuan Xu,
Wenyue Hua,
Yingqiang Ge,
Zelong Li,
Yongfeng Zhang
Abstract:
Generative recommendation based on Large Language Models (LLMs) have transformed the traditional ranking-based recommendation style into a text-to-text generation paradigm. However, in contrast to standard NLP tasks that inherently operate on human vocabulary, current research in generative recommendations struggles to effectively encode recommendation items within the text-to-text framework using…
▽ More
Generative recommendation based on Large Language Models (LLMs) have transformed the traditional ranking-based recommendation style into a text-to-text generation paradigm. However, in contrast to standard NLP tasks that inherently operate on human vocabulary, current research in generative recommendations struggles to effectively encode recommendation items within the text-to-text framework using concise yet meaningful ID representations. To better align LLMs with recommendation needs, we propose IDGen, representing each item as a unique, concise, semantically rich, platform-agnostic textual ID using human language tokens. This is achieved by training a textual ID generator alongside the LLM-based recommender, enabling seamless integration of personalized recommendations into natural language generation. Notably, as user history is expressed in natural language and decoupled from the original dataset, our approach suggests the potential for a foundational generative recommendation model. Experiments show that our framework consistently surpasses existing models in sequential recommendation under standard experimental setting. Then, we explore the possibility of training a foundation recommendation model with the proposed method on data collected from 19 different datasets and tested its recommendation performance on 6 unseen datasets across different platforms under a completely zero-shot setting. The results show that the zero-shot performance of the pre-trained foundation model is comparable to or even better than some traditional recommendation models based on supervised training, showing the potential of the IDGen paradigm serving as the foundation model for generative recommendation. Code and data are open-sourced at https://github.com/agiresearch/IDGenRec.
△ Less
Submitted 17 May, 2024; v1 submitted 27 March, 2024;
originally announced March 2024.
-
Large Language Models in Biomedical and Health Informatics: A Bibliometric Review
Authors:
Huizi Yu,
Lizhou Fan,
Lingyao Li,
Jiayan Zhou,
Zihui Ma,
Lu Xian,
Wenyue Hua,
Sijia He,
Mingyu **,
Yongfeng Zhang,
Ashvin Gandhi,
Xin Ma
Abstract:
Large Language Models (LLMs) have rapidly become important tools in Biomedical and Health Informatics (BHI), enabling new ways to analyze data, treat patients, and conduct research. This bibliometric review aims to provide a panoramic view of how LLMs have been used in BHI by examining research articles and collaboration networks from 2022 to 2023. It further explores how LLMs can improve Natural…
▽ More
Large Language Models (LLMs) have rapidly become important tools in Biomedical and Health Informatics (BHI), enabling new ways to analyze data, treat patients, and conduct research. This bibliometric review aims to provide a panoramic view of how LLMs have been used in BHI by examining research articles and collaboration networks from 2022 to 2023. It further explores how LLMs can improve Natural Language Processing (NLP) applications in various BHI areas like medical diagnosis, patient engagement, electronic health record management, and personalized medicine. To do this, our bibliometric review identifies key trends, maps out research networks, and highlights major developments in this fast-moving field. Lastly, it discusses the ethical concerns and practical challenges of using LLMs in BHI, such as data privacy and reliable medical recommendations. Looking ahead, we consider how LLMs could further transform biomedical research as well as healthcare delivery and patient outcomes. This bibliometric review serves as a resource for stakeholders in healthcare, including researchers, clinicians, and policymakers, to understand the current state and future potential of LLMs in BHI.
△ Less
Submitted 23 April, 2024; v1 submitted 24 March, 2024;
originally announced March 2024.
-
3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation
Authors:
Frank Zhang,
Yibo Zhang,
Quan Zheng,
Rui Ma,
Wei Hua,
Hujun Bao,
Weiwei Xu,
Changqing Zou
Abstract:
Text-driven 3D scene generation techniques have made rapid progress in recent years. Their success is mainly attributed to using existing generative models to iteratively perform image war** and inpainting to generate 3D scenes. However, these methods heavily rely on the outputs of existing models, leading to error accumulation in geometry and appearance that prevent the models from being used i…
▽ More
Text-driven 3D scene generation techniques have made rapid progress in recent years. Their success is mainly attributed to using existing generative models to iteratively perform image war** and inpainting to generate 3D scenes. However, these methods heavily rely on the outputs of existing models, leading to error accumulation in geometry and appearance that prevent the models from being used in various scenarios (e.g., outdoor and unreal scenarios). To address this limitation, we generatively refine the newly generated local views by querying and aggregating global 3D information, and then progressively generate the 3D scene. Specifically, we employ a tri-plane features-based NeRF as a unified representation of the 3D scene to constrain global 3D consistency, and propose a generative refinement network to synthesize new contents with higher quality by exploiting the natural image prior from 2D diffusion model as well as the global 3D information of the current scene. Our extensive experiments demonstrate that, in comparison to previous methods, our approach supports wide variety of scene generation and arbitrary camera trajectories with improved visual quality and 3D consistency.
△ Less
Submitted 14 March, 2024;
originally announced March 2024.
-
Effects of Structural Variations to X-ray Absorption Spectra of g-C$_3$N$_4$: Insights from DFT and TDDFT Simulations
Authors:
Jun-Rong Zhang,
Sheng-Yu Wang,
Minrui Wei,
Qiang Fu,
Weijie Hua
Abstract:
X-ray absorption spectroscopy (XAS) is widely employed for structure characterization of graphitic carbon nitride (g-C$_3$N$_4$) and its composites. Nevertheless, even for pure g-C$_3$N$_4$, discrepancies in energy and profile exist across different experiments, which can be attributed to variations in structures arising from diverse synthesis conditions and calibration procedures. Here, we conduc…
▽ More
X-ray absorption spectroscopy (XAS) is widely employed for structure characterization of graphitic carbon nitride (g-C$_3$N$_4$) and its composites. Nevertheless, even for pure g-C$_3$N$_4$, discrepancies in energy and profile exist across different experiments, which can be attributed to variations in structures arising from diverse synthesis conditions and calibration procedures. Here, we conducted a theoretical investigation on XAS of three representative g-C$_3$N$_4$ structures (planar, corrugated, and micro-corrugated) optimized with different strategies, to understand the structure-spectroscopy relation. Different methods were compared, including density functional theory (DFT) with the full (FCH) or equivalent (ECH) core-hole approximation, as well as the time-dependent DFT (TDDFT). FCH was responsible for getting accurate absolute absorption energy; while ECH and TDDFT aided in interpreting the spectra, through ECH-state canonical molecular orbitals (ECH-CMOs) and natural transition orbitals (NTOs), respectively. With each method, the spectra at the three structures show evident differences, which can be correlated to different individual experiments or in between. Our calculations explained the structural reason behind the spectral discrepancies among different experiments. Moreover, profiles predicted by these methods also displayed consistency, so their differences can be used as a reliable indicator of their accuracy. Both ECH-CMOs and NTO particle orbitals led to similar graphics, validating their applicability in interpreting the transitions. This work provides a comprehensive analysis of the structure-XAS relation for g-C$_3$N$_4$, provides concrete explanations for the spectral differences reported in various experiments, and offers insight for future structure dynamical and transient X-ray spectral analyses.
△ Less
Submitted 14 March, 2024;
originally announced March 2024.
-
Temperature and Tautomeric Effects in High-Resolution Oxygen 1s X-ray Photoelectron Spectroscopy of Purines and Pyrimidines
Authors:
Minrui Wei,
Junxiang Zuo,
Guangjun Tian,
Weijie Hua
Abstract:
Purines and pyrimidines, crucial building blocks in biological systems, have attracted significant interest across molecular physics, biochemistry, pharmacology, and chemistry. Extensive spectroscopies have been employed for characterization, while the temperature and potential tautomeric effects can complicate the interpretation of underlying physics and chemistry. Here, we conducted first-princi…
▽ More
Purines and pyrimidines, crucial building blocks in biological systems, have attracted significant interest across molecular physics, biochemistry, pharmacology, and chemistry. Extensive spectroscopies have been employed for characterization, while the temperature and potential tautomeric effects can complicate the interpretation of underlying physics and chemistry. Here, we conducted first-principles simulations to analyze the vibrationally-resolved O1s X-ray photoelectron spectra of 6 common biomolecules at different temperatures, comprising 3 purine (xanthine, caffeine, and hypoxanthine) and 3 pyrimidine (thymine, 5F-uracil, and uracil) derivatives, and the tautomeric effect of hypoxanthine at varying temperatures. Using both time-independent (TI) and time-dependent (TD) methods under the Franck-Condon approximation, we obtained theoretical spectra that exhibited excellent agreement with experiments. Our analysis of these systems, all featuring carbonyl oxygens, unveiled distinctive characteristics of oxygen in N-CO-N (O2) compared to that within a N-CO-C structure (O1), showcasing higher O1s binding energy and total vibrational reorganization energy. We observed small differences in the zero-point vibration energies between the core-ionized and ground states, indicating a weak Duschinsky rotation effect. We consistently found that O1s ionization resulted in elongation of the O*=C bond length. The TI method facilitated the assignment of experimental spectra to different atoms or tautomers, where the atom-specific vibronic profiles of all 6 molecules exhibited similarity, with the 0-2 transitions dominating. TD enabled a more comprehensive exploration of the temperature effect, and the tautomeric effect of hypoxanthine by incorporating the Boltzmann population ratios of tautomers. We observed significant temperature dependence in the vibronic features present in these spectra.
△ Less
Submitted 14 March, 2024;
originally announced March 2024.
-
NPHardEval4V: A Dynamic Reasoning Benchmark of Multimodal Large Language Models
Authors:
Lizhou Fan,
Wenyue Hua,
Xiang Li,
Kaijie Zhu,
Mingyu **,
Lingyao Li,
Haoyang Ling,
**kui Chi,
**dong Wang,
Xin Ma,
Yongfeng Zhang
Abstract:
Understanding the reasoning capabilities of Multimodal Large Language Models (MLLMs) is an important area of research. In this study, we introduce a dynamic benchmark, NPHardEval4V, aimed at addressing the existing gaps in evaluating the pure reasoning abilities of MLLMs. Our benchmark aims to provide a venue to disentangle the effect of various factors such as image recognition and instruction fo…
▽ More
Understanding the reasoning capabilities of Multimodal Large Language Models (MLLMs) is an important area of research. In this study, we introduce a dynamic benchmark, NPHardEval4V, aimed at addressing the existing gaps in evaluating the pure reasoning abilities of MLLMs. Our benchmark aims to provide a venue to disentangle the effect of various factors such as image recognition and instruction following, from the overall performance of the models, allowing us to focus solely on evaluating their reasoning abilities. It is built by converting textual description of questions from NPHardEval to image representations. Our findings reveal significant discrepancies in reasoning abilities across different models and highlight the relatively weak performance of MLLMs compared to LLMs in terms of reasoning. We also investigate the impact of different prompting styles, including visual, text, and combined visual and text prompts, on the reasoning abilities of MLLMs, demonstrating the different impacts of multimodal inputs in model performance. Unlike traditional benchmarks, which focus primarily on static evaluations, our benchmark will be updated monthly to prevent overfitting and ensure a more authentic and fine-grained evaluation of the models. We believe that this benchmark can aid in understanding and guide the further development of reasoning abilities in MLLMs. The benchmark dataset and code are available at https://github.com/lizhouf/NPHardEval4V
△ Less
Submitted 5 March, 2024; v1 submitted 4 March, 2024;
originally announced March 2024.
-
What if LLMs Have Different World Views: Simulating Alien Civilizations with LLM-based Agents
Authors:
Mingyu **,
Beichen Wang,
Zhaoqian Xue,
Suiyuan Zhu,
Wenyue Hua,
Hua Tang,
Kai Mei,
Mengnan Du,
Yongfeng Zhang
Abstract:
In this study, we introduce "CosmoAgent," an innovative artificial intelligence framework utilizing Large Language Models (LLMs) to simulate complex interactions between human and extraterrestrial civilizations, with a special emphasis on Stephen Hawking's cautionary advice about not sending radio signals haphazardly into the universe. The goal is to assess the feasibility of peaceful coexistence…
▽ More
In this study, we introduce "CosmoAgent," an innovative artificial intelligence framework utilizing Large Language Models (LLMs) to simulate complex interactions between human and extraterrestrial civilizations, with a special emphasis on Stephen Hawking's cautionary advice about not sending radio signals haphazardly into the universe. The goal is to assess the feasibility of peaceful coexistence while considering potential risks that could threaten well-intentioned civilizations. Employing mathematical models and state transition matrices, our approach quantitatively evaluates the development trajectories of civilizations, offering insights into future decision-making at critical points of growth and saturation. Furthermore, the paper acknowledges the vast diversity in potential living conditions across the universe, which could foster unique cosmologies, ethical codes, and worldviews among various civilizations. Recognizing the Earth-centric bias inherent in current LLM designs, we propose the novel concept of using LLMs with diverse ethical paradigms and simulating interactions between entities with distinct moral principles. This innovative research provides a new way to understand complex inter-civilizational dynamics, expanding our perspective while pioneering novel strategies for conflict resolution, crucial for preventing interstellar conflicts. We have also released the code and datasets to enable further academic investigation into this interesting area of research. The code is available at https://github.com/agiresearch/AlienAgent.
△ Less
Submitted 20 February, 2024; v1 submitted 20 February, 2024;
originally announced February 2024.
-
EmojiCrypt: Prompt Encryption for Secure Communication with Large Language Models
Authors:
Guo Lin,
Wenyue Hua,
Yongfeng Zhang
Abstract:
Cloud-based large language models (LLMs) such as ChatGPT have increasingly become integral to daily operations, serving as vital tools across various applications. While these models offer substantial benefits in terms of accessibility and functionality, they also introduce significant privacy concerns: the transmission and storage of user data in cloud infrastructures pose substantial risks of da…
▽ More
Cloud-based large language models (LLMs) such as ChatGPT have increasingly become integral to daily operations, serving as vital tools across various applications. While these models offer substantial benefits in terms of accessibility and functionality, they also introduce significant privacy concerns: the transmission and storage of user data in cloud infrastructures pose substantial risks of data breaches and unauthorized access to sensitive information; even if the transmission and storage of data is encrypted, the LLM service provider itself still knows the real contents of the data, preventing individuals or entities from confidently using such LLM services. To address these concerns, this paper proposes a simple yet effective mechanism EmojiCrypt to protect user privacy. It uses Emoji to encrypt the user inputs before sending them to LLM, effectively rendering them indecipherable to human or LLM's examination while retaining the original intent of the prompt, thus ensuring the model's performance remains unaffected. We conduct experiments on three tasks, personalized recommendation, sentiment analysis, and tabular data analysis. Experiment results reveal that EmojiCrypt can encrypt personal information within prompts in such a manner that not only prevents the discernment of sensitive data by humans or LLM itself, but also maintains or even improves the precision without further tuning, achieving comparable or even better task accuracy than directly prompting the LLM without prompt encryption. These results highlight the practicality of adopting encryption measures that safeguard user privacy without compromising the functional integrity and performance of LLMs. Code and dataset are available at https://github.com/agiresearch/EmojiCrypt.
△ Less
Submitted 12 February, 2024; v1 submitted 8 February, 2024;
originally announced February 2024.
-
TrustAgent: Towards Safe and Trustworthy LLM-based Agents through Agent Constitution
Authors:
Wenyue Hua,
Xianjun Yang,
Zelong Li,
Wei Cheng,
Yongfeng Zhang
Abstract:
The emergence of LLM-based agents has garnered considerable attention, yet their trustworthiness remains an under-explored area. As agents can directly interact with the physical environment, their reliability and safety is critical. This paper presents an Agent-Constitution-based agent framework, TrustAgent, an initial investigation into improving the safety dimension of trustworthiness in LLM-ba…
▽ More
The emergence of LLM-based agents has garnered considerable attention, yet their trustworthiness remains an under-explored area. As agents can directly interact with the physical environment, their reliability and safety is critical. This paper presents an Agent-Constitution-based agent framework, TrustAgent, an initial investigation into improving the safety dimension of trustworthiness in LLM-based agents. This framework consists of threefold strategies: pre-planning strategy which injects safety knowledge to the model prior to plan generation, in-planning strategy which bolsters safety during plan generation, and post-planning strategy which ensures safety by post-planning inspection. Through experimental analysis, we demonstrate how these approaches can effectively elevate an LLM agent's safety by identifying and preventing potential dangers. Furthermore, we explore the intricate relationships between safety and helpfulness, and between the model's reasoning ability and its efficacy as a safe agent. This paper underscores the imperative of integrating safety awareness and trustworthiness into the design and deployment of LLM-based agents, not only to enhance their performance but also to ensure their responsible integration into human-centric environments. Data and code are available at https://github.com/agiresearch/TrustAgent.
△ Less
Submitted 17 February, 2024; v1 submitted 2 February, 2024;
originally announced February 2024.
-
Formal-LLM: Integrating Formal Language and Natural Language for Controllable LLM-based Agents
Authors:
Zelong Li,
Wenyue Hua,
Hao Wang,
He Zhu,
Yongfeng Zhang
Abstract:
Recent advancements on Large Language Models (LLMs) enable AI Agents to automatically generate and execute multi-step plans to solve complex tasks. However, since LLM's content generation process is hardly controllable, current LLM-based agents frequently generate invalid or non-executable plans, which jeopardizes the performance of the generated plans and corrupts users' trust in LLM-based agents…
▽ More
Recent advancements on Large Language Models (LLMs) enable AI Agents to automatically generate and execute multi-step plans to solve complex tasks. However, since LLM's content generation process is hardly controllable, current LLM-based agents frequently generate invalid or non-executable plans, which jeopardizes the performance of the generated plans and corrupts users' trust in LLM-based agents. In response, this paper proposes a novel ``Formal-LLM'' framework for LLM-based agents by integrating the expressiveness of natural language and the precision of formal language. Specifically, the framework allows human users to express their requirements or constraints for the planning process as an automaton. A stack-based LLM plan generation process is then conducted under the supervision of the automaton to ensure that the generated plan satisfies the constraints, making the planning process controllable. We conduct experiments on both benchmark tasks and practical real-life tasks, and our framework achieves over 50% overall performance increase, which validates the feasibility and effectiveness of employing Formal-LLM to guide the plan generation of agents, preventing the agents from generating invalid and unsuccessful plans. Further, more controllable LLM-based agents can facilitate the broader utilization of LLM in application scenarios where high validity of planning is essential. The work is open-sourced at https://github.com/agiresearch/Formal-LLM.
△ Less
Submitted 18 June, 2024; v1 submitted 1 February, 2024;
originally announced February 2024.
-
Health-LLM: Personalized Retrieval-Augmented Disease Prediction System
Authors:
Mingyu **,
Qinkai Yu,
Dong Shu,
Chong Zhang,
Lizhou Fan,
Wenyue Hua,
Suiyuan Zhu,
Yanda Meng,
Zhenting Wang,
Mengnan Du,
Yongfeng Zhang
Abstract:
Recent advancements in artificial intelligence (AI), especially large language models (LLMs), have significantly advanced healthcare applications and demonstrated potentials in intelligent medical treatment. However, there are conspicuous challenges such as vast data volumes and inconsistent symptom characterization standards, preventing full integration of healthcare AI systems with individual pa…
▽ More
Recent advancements in artificial intelligence (AI), especially large language models (LLMs), have significantly advanced healthcare applications and demonstrated potentials in intelligent medical treatment. However, there are conspicuous challenges such as vast data volumes and inconsistent symptom characterization standards, preventing full integration of healthcare AI systems with individual patients' needs. To promote professional and personalized healthcare, we propose an innovative framework, Heath-LLM, which combines large-scale feature extraction and medical knowledge trade-off scoring. Compared to traditional health management applications, our system has three main advantages: (1) It integrates health reports and medical knowledge into a large model to ask relevant questions to large language model for disease prediction; (2) It leverages a retrieval augmented generation (RAG) mechanism to enhance feature extraction; (3) It incorporates a semi-automated feature updating framework that can merge and delete features to improve accuracy of disease prediction. We experiment on a large number of health reports to assess the effectiveness of Health-LLM system. The results indicate that the proposed system surpasses the existing ones and has the potential to significantly advance disease prediction and personalized health management. The code is available at https://github.com/jmyissb/HealthLLM.
△ Less
Submitted 19 March, 2024; v1 submitted 1 February, 2024;
originally announced February 2024.
-
PAP-REC: Personalized Automatic Prompt for Recommendation Language Model
Authors:
Zelong Li,
Jianchao Ji,
Yingqiang Ge,
Wenyue Hua,
Yongfeng Zhang
Abstract:
Recently emerged prompt-based Recommendation Language Models (RLM) can solve multiple recommendation tasks uniformly. The RLMs make full use of the inherited knowledge learned from the abundant pre-training data to solve the downstream recommendation tasks by prompts, without introducing additional parameters or network training. However, handcrafted prompts require significant expertise and human…
▽ More
Recently emerged prompt-based Recommendation Language Models (RLM) can solve multiple recommendation tasks uniformly. The RLMs make full use of the inherited knowledge learned from the abundant pre-training data to solve the downstream recommendation tasks by prompts, without introducing additional parameters or network training. However, handcrafted prompts require significant expertise and human effort since slightly rewriting prompts may cause massive performance changes. In this paper, we propose PAP-REC, a framework to generate the Personalized Automatic Prompt for RECommendation language models to mitigate the inefficiency and ineffectiveness problems derived from manually designed prompts. Specifically, personalized automatic prompts allow different users to have different prompt tokens for the same task, automatically generated using a gradient-based method. One challenge for personalized automatic prompt generation for recommendation language models is the extremely large search space, leading to a long convergence time. To effectively and efficiently address the problem, we develop surrogate metrics and leverage an alternative updating schedule for prompting recommendation language models. Experimental results show that our PAP-REC framework manages to generate personalized prompts, and the automatically generated prompts outperform manually constructed prompts and also outperform various baseline recommendation models. The source code of the work is available at https://github.com/rutgerswiselab/PAP-REC.
△ Less
Submitted 31 January, 2024;
originally announced February 2024.
-
Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks
Authors:
Wenyue Hua,
Jiang Guo,
Mingwen Dong,
Henghui Zhu,
Patrick Ng,
Zhiguo Wang
Abstract:
Current approaches of knowledge editing struggle to effectively propagate updates to interconnected facts. In this work, we delve into the barriers that hinder the appropriate propagation of updated knowledge within these models for accurate reasoning. To support our analysis, we introduce a novel reasoning-based benchmark -- ReCoE (Reasoning-based Counterfactual Editing dataset) -- which covers s…
▽ More
Current approaches of knowledge editing struggle to effectively propagate updates to interconnected facts. In this work, we delve into the barriers that hinder the appropriate propagation of updated knowledge within these models for accurate reasoning. To support our analysis, we introduce a novel reasoning-based benchmark -- ReCoE (Reasoning-based Counterfactual Editing dataset) -- which covers six common reasoning schemes in real world. We conduct a thorough analysis of existing knowledge editing techniques, including input augmentation, finetuning, and locate-and-edit. We found that all model editing methods show notably low performance on this dataset, especially in certain reasoning schemes. Our analysis over the chain-of-thought generation of edited models further uncover key reasons behind the inadequacy of existing knowledge editing methods from a reasoning standpoint, involving aspects on fact-wise editing, fact recall ability, and coherence in generation. We will make our benchmark publicly available.
△ Less
Submitted 30 January, 2024;
originally announced January 2024.
-
The Impact of Reasoning Step Length on Large Language Models
Authors:
Mingyu **,
Qinkai Yu,
Dong Shu,
Haiyan Zhao,
Wenyue Hua,
Yanda Meng,
Yongfeng Zhang,
Mengnan Du
Abstract:
Chain of Thought (CoT) is significant in improving the reasoning abilities of large language models (LLMs). However, the correlation between the effectiveness of CoT and the length of reasoning steps in prompts remains largely unknown. To shed light on this, we have conducted several empirical experiments to explore the relations. Specifically, we design experiments that expand and compress the ra…
▽ More
Chain of Thought (CoT) is significant in improving the reasoning abilities of large language models (LLMs). However, the correlation between the effectiveness of CoT and the length of reasoning steps in prompts remains largely unknown. To shed light on this, we have conducted several empirical experiments to explore the relations. Specifically, we design experiments that expand and compress the rationale reasoning steps within CoT demonstrations while kee** all other factors constant. We have the following key findings. First, the results indicate that lengthening the reasoning steps in prompts, even without adding new information into the prompt, considerably enhances LLMs' reasoning abilities across multiple datasets. Alternatively, shortening the reasoning steps, even while preserving the key information, significantly diminishes the reasoning abilities of models. This finding highlights the importance of the number of steps in CoT prompts and provides practical guidance to make better use of LLMs' potential in complex problem-solving scenarios. Second, we also investigated the relationship between the performance of CoT and the rationales used in demonstrations. Surprisingly, the result shows that even incorrect rationales can yield favorable outcomes if they maintain the requisite length of inference. Third, we observed that the advantages of increasing reasoning steps are task-dependent: simpler tasks require fewer steps, whereas complex tasks gain significantly from longer inference sequences. The code is available at https://github.com/MingyuJ666/The-Impact-of-Reasoning-Step-Length-on-Large-Language-Models
△ Less
Submitted 22 June, 2024; v1 submitted 9 January, 2024;
originally announced January 2024.
-
Improving the Robustness of Knowledge-Grounded Dialogue via Contrastive Learning
Authors:
Jiaan Wang,
Jianfeng Qu,
Kexin Wang,
Zhixu Li,
Wen Hua,
Ximing Li,
An Liu
Abstract:
Knowledge-grounded dialogue (KGD) learns to generate an informative response based on a given dialogue context and external knowledge (\emph{e.g.}, knowledge graphs; KGs). Recently, the emergence of large language models (LLMs) and pre-training techniques has brought great success to knowledge-grounded dialogue. However, when building KGD systems in real applications, there are various real-world…
▽ More
Knowledge-grounded dialogue (KGD) learns to generate an informative response based on a given dialogue context and external knowledge (\emph{e.g.}, knowledge graphs; KGs). Recently, the emergence of large language models (LLMs) and pre-training techniques has brought great success to knowledge-grounded dialogue. However, when building KGD systems in real applications, there are various real-world noises that are inevitable to face. For example, the dialogue context might involve perturbations such as misspellings and abbreviations. In addition, KGs typically suffer from incompletion and also might contain erroneous and outdated facts. Such real-world noises pose a challenge to the robustness of KGD systems and hinder their applications in the real world. In this paper, we propose an entity-based contrastive learning framework for improving the robustness of KGD. Specifically, we make use of the entity information in a KGD sample to create both its positive and negative samples which involve semantic-irrelevant and semantic-relevant perturbations, respectively. The contrastive learning framework ensures the KGD model is aware of these two types of perturbations, thus generating informative responses with the potentially noisy inputs in real applications. Experimental results on three benchmark datasets show that our method achieves new state-of-the-art performance in terms of automatic evaluation scores, verifying its effectiveness and potentiality. Furthermore, we show that our method can generate better responses than comparison models in both the noisy and the few-shot settings.
△ Less
Submitted 9 January, 2024;
originally announced January 2024.
-
NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes
Authors:
Lizhou Fan,
Wenyue Hua,
Lingyao Li,
Haoyang Ling,
Yongfeng Zhang
Abstract:
Complex reasoning ability is one of the most important features of current LLMs, which has also been leveraged to play an integral role in complex decision-making tasks. Therefore, the investigation into the reasoning capabilities of Large Language Models (LLMs) is critical: numerous benchmarks have been established to assess the reasoning abilities of LLMs. However, current benchmarks are inadequ…
▽ More
Complex reasoning ability is one of the most important features of current LLMs, which has also been leveraged to play an integral role in complex decision-making tasks. Therefore, the investigation into the reasoning capabilities of Large Language Models (LLMs) is critical: numerous benchmarks have been established to assess the reasoning abilities of LLMs. However, current benchmarks are inadequate in offering a rigorous evaluation of the full extent of reasoning abilities that LLMs are capable of achieving. They are also prone to the risk of overfitting, as these benchmarks, being publicly accessible and static, allow models to potentially tailor their responses to specific benchmark metrics, thereby inflating their performance. Addressing these limitations, our research introduces a new benchmark, named NPHardEval. This benchmark is designed to evaluate the reasoning abilities of LLMs across a broad spectrum of 900 algorithmic questions, extending up to the NP-Hard complexity class. These questions are meticulously chosen to represent a wide range of complexity class below the NP-hard complexity class, offering a rigorous measure of the reasoning ability of LLMs. Through this study, we shed light on the current state of reasoning in LLMs, providing an objective and rigorous perspective through the comparison of LLMs' performance across complex classes. Moreover, this benchmark is designed with a dynamic update mechanism, where the datapoints are refreshed on a monthly basis. Such regular updates play a crucial role in mitigating the risk of LLMs overfitting to the benchmark, promoting a more accurate and reliable assessment of their reasoning capabilities. The benchmark dataset and code of NPHardEval are available at https://github.com/casmlab/NPHardEval.
△ Less
Submitted 12 February, 2024; v1 submitted 22 December, 2023;
originally announced December 2023.
-
Long-Tailed 3D Detection via 2D Late Fusion
Authors:
Yechi Ma,
Neehar Peri,
Shuoquan Wei,
Wei Hua,
Deva Ramanan,
Yanan Li,
Shu Kong
Abstract:
Long-Tailed 3D Object Detection (LT3D) addresses the problem of accurately detecting objects from both common and rare classes. Contemporary multi-modal detectors achieve low AP on rare-classes (e.g., CMT only achieves 9.4 AP on stroller), presumably because training detectors end-to-end with significant class imbalance is challenging. To address this limitation, we delve into a simple late-fusion…
▽ More
Long-Tailed 3D Object Detection (LT3D) addresses the problem of accurately detecting objects from both common and rare classes. Contemporary multi-modal detectors achieve low AP on rare-classes (e.g., CMT only achieves 9.4 AP on stroller), presumably because training detectors end-to-end with significant class imbalance is challenging. To address this limitation, we delve into a simple late-fusion framework that ensembles independently trained uni-modal LiDAR and RGB detectors. Importantly, such a late-fusion framework allows us to leverage large-scale uni-modal datasets (with more examples for rare classes) to train better uni-modal RGB detectors, unlike prevailing multimodal detectors that require paired multi-modal training data. Notably, our approach significantly improves rare-class detection by 7.2% over prior work. Further, we examine three critical components of our simple late-fusion approach from first principles and investigate whether to train 2D or 3D RGB detectors, whether to match RGB and LiDAR detections in 3D or the projected 2D image plane for fusion, and how to fuse matched detections. Extensive experiments reveal that 2D RGB detectors achieve better recognition accuracy for rare classes than 3D RGB detectors and matching on the 2D image plane mitigates depth estimation errors. Our late-fusion approach achieves 51.4 mAP on the established nuScenes LT3D benchmark, improving over prior work by 5.9 mAP!
△ Less
Submitted 14 June, 2024; v1 submitted 18 December, 2023;
originally announced December 2023.
-
LLM as OS, Agents as Apps: Envisioning AIOS, Agents and the AIOS-Agent Ecosystem
Authors:
Yingqiang Ge,
Yujie Ren,
Wenyue Hua,
Shuyuan Xu,
Juntao Tan,
Yongfeng Zhang
Abstract:
This paper envisions a revolutionary AIOS-Agent ecosystem, where Large Language Model (LLM) serves as the (Artificial) Intelligent Operating System (IOS, or AIOS)--an operating system "with soul". Upon this foundation, a diverse range of LLM-based AI Agent Applications (Agents, or AAPs) are developed, enriching the AIOS-Agent ecosystem and signaling a paradigm shift from the traditional OS-APP eco…
▽ More
This paper envisions a revolutionary AIOS-Agent ecosystem, where Large Language Model (LLM) serves as the (Artificial) Intelligent Operating System (IOS, or AIOS)--an operating system "with soul". Upon this foundation, a diverse range of LLM-based AI Agent Applications (Agents, or AAPs) are developed, enriching the AIOS-Agent ecosystem and signaling a paradigm shift from the traditional OS-APP ecosystem. We envision that LLM's impact will not be limited to the AI application level, instead, it will in turn revolutionize the design and implementation of computer system, architecture, software, and programming language, featured by several main concepts: LLM as OS (system-level), Agents as Applications (application-level), Natural Language as Programming Interface (user-level), and Tools as Devices/Libraries (hardware/middleware-level). We begin by introducing the architecture of traditional OS. Then we formalize a conceptual framework for AIOS through "LLM as OS (LLMOS)", drawing analogies between AIOS and traditional OS: LLM is likened to OS kernel, context window to memory, external storage to file system, hardware tools to peripheral devices, software tools to programming libraries, and user prompts to user commands. Subsequently, we introduce the new AIOS-Agent Ecosystem, where users can easily program Agent Applications (AAPs) using natural language, democratizing the development of software, which is different from the traditional OS-APP ecosystem. Following this, we explore the diverse scope of Agent Applications. We delve into both single-agent and multi-agent systems, as well as human-agent interaction. Lastly, drawing on the insights from traditional OS-APP ecosystem, we propose a roadmap for the evolution of the AIOS-Agent ecosystem. This roadmap is designed to guide the future research and development, suggesting systematic progresses of AIOS and its Agent applications.
△ Less
Submitted 9 December, 2023; v1 submitted 6 December, 2023;
originally announced December 2023.
-
Beyond Isolation: Multi-Agent Synergy for Improving Knowledge Graph Construction
Authors:
Hongbin Ye,
Honghao Gui,
Aijia Zhang,
Tong Liu,
Wei Hua,
Weiqiang Jia
Abstract:
Knowledge graph construction (KGC) is a multifaceted undertaking involving the extraction of entities, relations, and events. Traditionally, large language models (LLMs) have been viewed as solitary task-solving agents in this complex landscape. However, this paper challenges this paradigm by introducing a novel framework, CooperKGC. Departing from the conventional approach, CooperKGC establishes…
▽ More
Knowledge graph construction (KGC) is a multifaceted undertaking involving the extraction of entities, relations, and events. Traditionally, large language models (LLMs) have been viewed as solitary task-solving agents in this complex landscape. However, this paper challenges this paradigm by introducing a novel framework, CooperKGC. Departing from the conventional approach, CooperKGC establishes a collaborative processing network, assembling a KGC collaboration team capable of concurrently addressing entity, relation, and event extraction tasks. Our experiments unequivocally demonstrate that fostering collaboration and information interaction among diverse agents within CooperKGC yields superior results compared to individual cognitive processes operating in isolation. Importantly, our findings reveal that the collaboration facilitated by CooperKGC enhances knowledge selection, correction, and aggregation capabilities across multiple rounds of interactions.
△ Less
Submitted 29 December, 2023; v1 submitted 5 December, 2023;
originally announced December 2023.
-
War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Wars
Authors:
Wenyue Hua,
Lizhou Fan,
Lingyao Li,
Kai Mei,
Jianchao Ji,
Yingqiang Ge,
Libby Hemphill,
Yongfeng Zhang
Abstract:
Can we avoid wars at the crossroads of history? This question has been pursued by individuals, scholars, policymakers, and organizations throughout human history. In this research, we attempt to answer the question based on the recent advances of Artificial Intelligence (AI) and Large Language Models (LLMs). We propose \textbf{WarAgent}, an LLM-powered multi-agent AI system, to simulate the partic…
▽ More
Can we avoid wars at the crossroads of history? This question has been pursued by individuals, scholars, policymakers, and organizations throughout human history. In this research, we attempt to answer the question based on the recent advances of Artificial Intelligence (AI) and Large Language Models (LLMs). We propose \textbf{WarAgent}, an LLM-powered multi-agent AI system, to simulate the participating countries, their decisions, and the consequences, in historical international conflicts, including the World War I (WWI), the World War II (WWII), and the Warring States Period (WSP) in Ancient China. By evaluating the simulation effectiveness, we examine the advancements and limitations of cutting-edge AI systems' abilities in studying complex collective human behaviors such as international conflicts under diverse settings. In these simulations, the emergent interactions among agents also offer a novel perspective for examining the triggers and conditions that lead to war. Our findings offer data-driven and AI-augmented insights that can redefine how we approach conflict resolution and peacekee** strategies. The implications stretch beyond historical analysis, offering a blueprint for using AI to understand human history and possibly prevent future international conflicts. Code and data are available at \url{https://github.com/agiresearch/WarAgent}.
△ Less
Submitted 30 January, 2024; v1 submitted 28 November, 2023;
originally announced November 2023.
-
Holistic Inverse Rendering of Complex Facade via Aerial 3D Scanning
Authors:
Zixuan Xie,
Rengan Xie,
Rong Li,
Kai Huang,
Pengju Qiao,
**gsen Zhu,
Xu Yin,
Qi Ye,
Wei Hua,
Yuchi Huo,
Hujun Bao
Abstract:
In this work, we use multi-view aerial images to reconstruct the geometry, lighting, and material of facades using neural signed distance fields (SDFs). Without the requirement of complex equipment, our method only takes simple RGB images captured by a drone as inputs to enable physically based and photorealistic novel-view rendering, relighting, and editing. However, a real-world facade usually h…
▽ More
In this work, we use multi-view aerial images to reconstruct the geometry, lighting, and material of facades using neural signed distance fields (SDFs). Without the requirement of complex equipment, our method only takes simple RGB images captured by a drone as inputs to enable physically based and photorealistic novel-view rendering, relighting, and editing. However, a real-world facade usually has complex appearances ranging from diffuse rocks with subtle details to large-area glass windows with specular reflections, making it hard to attend to everything. As a result, previous methods can preserve the geometry details but fail to reconstruct smooth glass windows or verse vise. In order to address this challenge, we introduce three spatial- and semantic-adaptive optimization strategies, including a semantic regularization approach based on zero-shot segmentation techniques to improve material consistency, a frequency-aware geometry regularization to balance surface smoothness and details in different surfaces, and a visibility probe-based scheme to enable efficient modeling of the local lighting in large-scale outdoor environments. In addition, we capture a real-world facade aerial 3D scanning image set and corresponding point clouds for training and benchmarking. The experiment demonstrates the superior quality of our method on facade holistic inverse rendering, novel view synthesis, and scene editing compared to state-of-the-art baselines.
△ Less
Submitted 8 April, 2024; v1 submitted 20 November, 2023;
originally announced November 2023.
-
Vibronic fine structure in the nitrogen 1s photoelectron spectra from Franck-Condon simulations. III. Rules for amine/imine N atoms in small N-heterocycles
Authors:
Minrui Wei,
Junxiang Zuo,
Guangjun Tian,
Weijie Hua
Abstract:
Vibronic coupling plays a crucial role in X-ray photoelectron spectra (XPS) of molecules. In a series of three papers, we present a comprehensive exploration of the N-heterocycles family, known for their diverse structures, to summarize the general rules of vibronic coupling in high-resolution vibrationally-resolved XPS spectra at the N1s edge. Building upon our previous studies on six-membered mo…
▽ More
Vibronic coupling plays a crucial role in X-ray photoelectron spectra (XPS) of molecules. In a series of three papers, we present a comprehensive exploration of the N-heterocycles family, known for their diverse structures, to summarize the general rules of vibronic coupling in high-resolution vibrationally-resolved XPS spectra at the N1s edge. Building upon our previous studies on six-membered monocyclic azines [Phys. Rev. A 106, 022811 (2022)] and fused bicyclic compounds indoles with five and six members [Phys. Rev. A 108, 022816 (2023)], in this study, we focus on investigating a series of 12 five-membered N-heterocycles using Franck-Condon simulations, incorporating Duschinsky rotation effects and density functional theory. Our calculations reveal distinct spectral characteristics of amine and imine within these 12 systems in binding energies, spectral characteristics, structural changes, vibrational coupling strengths, and effects of hydrogenation. Furthermore, we expand our analysis to encompass all 35 N-heterocycles discussed in the three papers and consolidate these findings into the general rules. we find that 1s ionization in amine nitrogen induces more substantial geometrical changes, resulting in larger vibronic coupling strength compared to imine nitrogens. The spectra of imine nitrogens exhibit two distinct characteristic peaks originating from the 0-0 and 0-1 transitions, whereas the spectra of amine nitrogens are characterized by a broad peak with numerous weak fingerprints due to significant mixing of various 0-$n$ transitions. We observe that amine (imine) nitrogens generally cause a negative (positive) change in zero-point vibrational energy. This study provides valuable insights into vibronic coupling in N-heterocycles, shedding light on the distinguishing features and behavior of amine and imine nitrogens in vibrationally-resolved XPS spectra.
△ Less
Submitted 10 November, 2023;
originally announced November 2023.
-
Global Impact and Balancing Act: Deciphering the Effect of Fluorination on B1s Binding Energies in Fluorinated $h$-BN Nanosheets
Authors:
Yang Xiao,
Jun-Rong Zhang,
Sheng-Yu Wang,
Weijie Hua
Abstract:
X-ray photoelectron spectroscopy (XPS) is an important characterization tool in the pursuit of controllable fluorination of two-dimensional hexagonal boron nitride ($h$-BN). However, there is a lack of clear spectral interpretation and seemingly conflicting measurements exist. To discern the structure-spectroscopy relation, we performed a comprehensive first-principles study on the boron 1s edge X…
▽ More
X-ray photoelectron spectroscopy (XPS) is an important characterization tool in the pursuit of controllable fluorination of two-dimensional hexagonal boron nitride ($h$-BN). However, there is a lack of clear spectral interpretation and seemingly conflicting measurements exist. To discern the structure-spectroscopy relation, we performed a comprehensive first-principles study on the boron 1s edge XPS of fluorinated $h$-BN (F-BN) nanosheets. By gradually introducing 1--6 fluorine atoms into different boron or nitrogen sites, we created various F-BN structures with do** ratios ranging from 1-6\%. Our calculations reveal that fluorines landed at boron or nitrogen sites exert competitive effects on the B1s binding energies (BEs), leading to red or blue shifts in different measurements. Our calculations affirmed the hypothesis that fluorination affects 1s BEs of all borons in the $Ï€$-conjugated system, undermining the transferability from $h$-BN to F-BN. Additionally, we observe that BE generally increases with higher fluorine concentration when both borons and nitrogens are non-exclusively fluorinated. These findings provide critical insights into how fluorination affects boron's 1s BEs, contributing to a better understanding of fluorination functionalization processes in $h$-BN and its potential applications in materials science.
△ Less
Submitted 28 January, 2024; v1 submitted 24 October, 2023;
originally announced October 2023.
-
M$^3$CS: Multi-Target Masked Point Modeling with Learnable Codebook and Siamese Decoders
Authors:
Qibo Qiu,
Honghui Yang,
Wenxiao Wang,
Shun Zhang,
Haiming Gao,
Haochao Ying,
Wei Hua,
Xiaofei He
Abstract:
Masked point modeling has become a promising scheme of self-supervised pre-training for point clouds. Existing methods reconstruct either the original points or related features as the objective of pre-training. However, considering the diversity of downstream tasks, it is necessary for the model to have both low- and high-level representation modeling capabilities to capture geometric details and…
▽ More
Masked point modeling has become a promising scheme of self-supervised pre-training for point clouds. Existing methods reconstruct either the original points or related features as the objective of pre-training. However, considering the diversity of downstream tasks, it is necessary for the model to have both low- and high-level representation modeling capabilities to capture geometric details and semantic contexts during pre-training. To this end, M$^3$CS is proposed to enable the model with the above abilities. Specifically, with masked point cloud as input, M$^3$CS introduces two decoders to predict masked representations and the original points simultaneously. While an extra decoder doubles parameters for the decoding process and may lead to overfitting, we propose siamese decoders to keep the amount of learnable parameters unchanged. Further, we propose an online codebook projecting continuous tokens into discrete ones before reconstructing masked points. In such way, we can enforce the decoder to take effect through the combinations of tokens rather than remembering each token. Comprehensive experiments show that M$^3$CS achieves superior performance at both classification and segmentation tasks, outperforming existing methods.
△ Less
Submitted 22 September, 2023;
originally announced September 2023.
-
Cognitive Mirage: A Review of Hallucinations in Large Language Models
Authors:
Hongbin Ye,
Tong Liu,
Aijia Zhang,
Wei Hua,
Weiqiang Jia
Abstract:
As large language models continue to develop in the field of AI, text generation systems are susceptible to a worrisome phenomenon known as hallucination. In this study, we summarize recent compelling insights into hallucinations in LLMs. We present a novel taxonomy of hallucinations from various text generation tasks, thus provide theoretical insights, detection methods and improvement approaches…
▽ More
As large language models continue to develop in the field of AI, text generation systems are susceptible to a worrisome phenomenon known as hallucination. In this study, we summarize recent compelling insights into hallucinations in LLMs. We present a novel taxonomy of hallucinations from various text generation tasks, thus provide theoretical insights, detection methods and improvement approaches. Based on this, future research directions are proposed. Our contribution are threefold: (1) We provide a detailed and complete taxonomy for hallucinations appearing in text generation tasks; (2) We provide theoretical analyses of hallucinations in LLMs and provide existing detection and improvement methods; (3) We propose several research directions that can be developed in the future. As hallucinations garner significant attention from the community, we will maintain updates on relevant research progress.
△ Less
Submitted 13 September, 2023;
originally announced September 2023.
-
Franck-Condon Simulation of Vibrationally-Resolved X-ray Spectra for Diatomic Systems: Validation of Harmonic Approximation and Density Functional Theory
Authors:
Lu Zhang,
Minrui Wei,
Guoyan Ge,
Weijie Hua
Abstract:
Under the Franck-Condon approximation, we systematically validated the performance of density functional theory (DFT) and the effects of anharmonicity in simulating C/N/O K-edge vibrationally-resolved X-ray spectra of common diatomic molecules. To get ``transparent'' validations, vibronic fine structures of only the lowest 1s excited or ionized state in the X-ray absorption (XAS) or photoelectron…
▽ More
Under the Franck-Condon approximation, we systematically validated the performance of density functional theory (DFT) and the effects of anharmonicity in simulating C/N/O K-edge vibrationally-resolved X-ray spectra of common diatomic molecules. To get ``transparent'' validations, vibronic fine structures of only the lowest 1s excited or ionized state in the X-ray absorption (XAS) or photoelectron (XPS) spectra were investigated. All 6 systems (N$_2$, N$_2^+$; NO, NO$^+$; CO, CO$^+$) were studied within the harmonic oscillator (HO) approximation using DFT with four functionals (BLYP, BP86, B3LYP, M06-2X) for 10 XAS and 4 XPS spectra, and excellent agreement between theoretical and experimental spectra was found in most systems, except O1s XAS of NO, CO, and NO$^+$. We analyzed and established a connection between their complex vibronic structures (many weak oscillating features within a broad peak) and the significant geometrical changes induced by the O1s hole. The three spectra were well reproduced with anharmonic (AH) calculations by using quantum wavepacket dynamics based on potential energy curves (PECs) generated by DFT methods or multiconfigurational levels, highlighting sensitivity to the anharmonic effect and the PEC quality. In other examples of XAS (CO$^+$, C1s and O1s; NO, N1s) corresponding to smaller structural changes, HO and AH approaches lead to similar fine structures, which are dominated by 0-0 and 0-1 transitions. This study highlights the use of DFT with selected functionals for such diatomic calculations due to its easy execution and generally reliable accuracy. Functional dependence in diatomic systems is generally more pronounced than in polyatomic ones. We found that BLYP, BP86, and B3LYP functionals consistently exhibited high accuracy in predicting spectral profiles, bond lengths, and vibrational frequencies, which slightly outperformed M06-2X.
△ Less
Submitted 20 November, 2023; v1 submitted 26 July, 2023;
originally announced July 2023.
-
Vibronic fine structure in the nitrogen 1s photoelectron spectra from Franck-Condon simulations II: Indoles
Authors:
Minrui Wei,
Lu Zhang,
Guangjun Tian,
Weijie Hua
Abstract:
The vibronic coupling effect in nitrogen 1s X-ray photoelectron spectra (XPS) was systematically studied for a family of 17 bicyclic indole molecules by combining Franck-Condon simulations (including the Duschinsky rotation effect) and density functional theory. The simulated vibrationally-resolved spectra of 4 molecules agree well with available experiments. Reliable predictions for this family f…
▽ More
The vibronic coupling effect in nitrogen 1s X-ray photoelectron spectra (XPS) was systematically studied for a family of 17 bicyclic indole molecules by combining Franck-Condon simulations (including the Duschinsky rotation effect) and density functional theory. The simulated vibrationally-resolved spectra of 4 molecules agree well with available experiments. Reliable predictions for this family further allowed us to summarize rules for spectral evolution in response to three types of common structural changes (side chain substitution, CH$\leftrightarrow$N replacement, and isomerization). Interestingly, vibronic properties of amine and imine nitrogen are clearly separated: they show negative and positive $Δ$ZPE (zero-point vibration energy of the core-ionized with respect to the ground state), respectively, indicating flatter and steeper PESs induced by the N 1s ionization; amine N's show stronger mode mixing effects than imine N's; the 1s ionizations on two types of nitrogens led to distinct changes in local bond lengths and angles. The rules are useful for a basic understanding of vibronic coupling in this family, and the precise spectra are useful for future reference and data mining studies.
△ Less
Submitted 4 July, 2023;
originally announced July 2023.
-
GenRec: Large Language Model for Generative Recommendation
Authors:
Jianchao Ji,
Zelong Li,
Shuyuan Xu,
Wenyue Hua,
Yingqiang Ge,
Juntao Tan,
Yongfeng Zhang
Abstract:
In recent years, large language models (LLM) have emerged as powerful tools for diverse natural language processing tasks. However, their potential for recommender systems under the generative recommendation paradigm remains relatively unexplored. This paper presents an innovative approach to recommendation systems using large language models (LLMs) based on text data. In this paper, we present a…
▽ More
In recent years, large language models (LLM) have emerged as powerful tools for diverse natural language processing tasks. However, their potential for recommender systems under the generative recommendation paradigm remains relatively unexplored. This paper presents an innovative approach to recommendation systems using large language models (LLMs) based on text data. In this paper, we present a novel LLM for generative recommendation (GenRec) that utilized the expressive power of LLM to directly generate the target item to recommend, rather than calculating ranking score for each candidate item one by one as in traditional discriminative recommendation. GenRec uses LLM's understanding ability to interpret context, learn user preferences, and generate relevant recommendation. Our proposed approach leverages the vast knowledge encoded in large language models to accomplish recommendation tasks. We first we formulate specialized prompts to enhance the ability of LLM to comprehend recommendation tasks. Subsequently, we use these prompts to fine-tune the LLaMA backbone LLM on a dataset of user-item interactions, represented by textual data, to capture user preferences and item characteristics. Our research underscores the potential of LLM-based generative recommendation in revolutionizing the domain of recommendation systems and offers a foundational framework for future explorations in this field. We conduct extensive experiments on benchmark datasets, and the experiments shows that our GenRec has significant better results on large dataset.
△ Less
Submitted 4 July, 2023; v1 submitted 1 July, 2023;
originally announced July 2023.
-
Iterated Piecewise Affine (IPA) Approximation for Language Modeling
Authors:
Davood Shamsi,
Wen-yu Hua,
Brian Williams
Abstract:
In this work, we demonstrate the application of a first-order Taylor expansion to approximate a generic function $F: R^{n \times m} \to R^{n \times m}$ and utilize it in language modeling. To enhance the basic Taylor expansion, we introduce iteration and piecewise modeling, leading us to name the algorithm the Iterative Piecewise Affine (IPA) approximation. The final algorithm exhibits interesting…
▽ More
In this work, we demonstrate the application of a first-order Taylor expansion to approximate a generic function $F: R^{n \times m} \to R^{n \times m}$ and utilize it in language modeling. To enhance the basic Taylor expansion, we introduce iteration and piecewise modeling, leading us to name the algorithm the Iterative Piecewise Affine (IPA) approximation. The final algorithm exhibits interesting resemblances to the Transformers decoder architecture. By comparing parameter arrangements in IPA and Transformers, we observe a strikingly similar performance, with IPA outperforming Transformers by 1.5\% in the next token prediction task with cross-entropy loss for smaller sequence lengths.
△ Less
Submitted 1 November, 2023; v1 submitted 21 June, 2023;
originally announced June 2023.
-
OpenP5: An Open-Source Platform for Develo**, Training, and Evaluating LLM-based Recommender Systems
Authors:
Shuyuan Xu,
Wenyue Hua,
Yongfeng Zhang
Abstract:
In recent years, the integration of Large Language Models (LLMs) into recommender systems has garnered interest among both practitioners and researchers. Despite this interest, the field is still emerging, and the lack of open-source R&D platforms may impede the exploration of LLM-based recommendations. This paper introduces OpenP5, an open-source platform designed as a resource to facilitate the…
▽ More
In recent years, the integration of Large Language Models (LLMs) into recommender systems has garnered interest among both practitioners and researchers. Despite this interest, the field is still emerging, and the lack of open-source R&D platforms may impede the exploration of LLM-based recommendations. This paper introduces OpenP5, an open-source platform designed as a resource to facilitate the development, training, and evaluation of LLM-based generative recommender systems for research purposes. The platform is implemented using encoder-decoder LLMs (e.g., T5) and decoder-only LLMs (e.g., Llama-2) across 10 widely recognized public datasets, catering to two fundamental recommendation tasks: sequential and straightforward recommendations. Recognizing the crucial role of item IDs in LLM-based recommendations, we have also incorporated three item indexing methods within the OpenP5 platform: random indexing, sequential indexing and collaborative indexing. Built on the Transformers library, the platform facilitates easy customization of LLM-based recommendations for users. OpenP5 boasts a range of features including extensible data processing, task-centric optimization, comprehensive datasets and checkpoints, efficient acceleration, and standardized evaluations, making it a valuable tool for the implementation and evaluation of LLM-based recommender systems. The open-source code and pre-trained checkpoints for the OpenP5 library are publicly available at https://github.com/agiresearch/OpenP5.
△ Less
Submitted 10 April, 2024; v1 submitted 19 June, 2023;
originally announced June 2023.
-
ICDAR 2023 Competition on Structured Text Extraction from Visually-Rich Document Images
Authors:
Wenwen Yu,
Chengquan Zhang,
Haoyu Cao,
Wei Hua,
Bohan Li,
Huang Chen,
Mingyu Liu,
Mingrui Chen,
Jianfeng Kuang,
Mengjun Cheng,
Yuning Du,
Shikun Feng,
Xiaoguang Hu,
Pengyuan Lyu,
Kun Yao,
Yuechen Yu,
Yuliang Liu,
Wanxiang Che,
Errui Ding,
Cheng-Lin Liu,
Jiebo Luo,
Shuicheng Yan,
Min Zhang,
Dimosthenis Karatzas,
Xing Sun
, et al. (2 additional authors not shown)
Abstract:
Structured text extraction is one of the most valuable and challenging application directions in the field of Document AI. However, the scenarios of past benchmarks are limited, and the corresponding evaluation protocols usually focus on the submodules of the structured text extraction scheme. In order to eliminate these problems, we organized the ICDAR 2023 competition on Structured text extracti…
▽ More
Structured text extraction is one of the most valuable and challenging application directions in the field of Document AI. However, the scenarios of past benchmarks are limited, and the corresponding evaluation protocols usually focus on the submodules of the structured text extraction scheme. In order to eliminate these problems, we organized the ICDAR 2023 competition on Structured text extraction from Visually-Rich Document images (SVRD). We set up two tracks for SVRD including Track 1: HUST-CELL and Track 2: Baidu-FEST, where HUST-CELL aims to evaluate the end-to-end performance of Complex Entity Linking and Labeling, and Baidu-FEST focuses on evaluating the performance and generalization of Zero-shot / Few-shot Structured Text extraction from an end-to-end perspective. Compared to the current document benchmarks, our two tracks of competition benchmark enriches the scenarios greatly and contains more than 50 types of visually-rich document images (mainly from the actual enterprise applications). The competition opened on 30th December, 2022 and closed on 24th March, 2023. There are 35 participants and 91 valid submissions received for Track 1, and 15 participants and 26 valid submissions received for Track 2. In this report we will presents the motivation, competition datasets, task definition, evaluation protocol, and submission summaries. According to the performance of the submissions, we believe there is still a large gap on the expected information extraction performance for complex and zero-shot scenarios. It is hoped that this competition will attract many researchers in the field of CV and NLP, and bring some new thoughts to the field of Document AI.
△ Less
Submitted 5 June, 2023;
originally announced June 2023.
-
Information Flow Control in Machine Learning through Modular Model Architecture
Authors:
Trishita Tiwari,
Suchin Gururangan,
Chuan Guo,
Weizhe Hua,
Sanjay Kariyappa,
Udit Gupta,
Wenjie Xiong,
Kiwan Maeng,
Hsien-Hsin S. Lee,
G. Edward Suh
Abstract:
In today's machine learning (ML) models, any part of the training data can affect the model output. This lack of control for information flow from training data to model output is a major obstacle in training models on sensitive data when access control only allows individual users to access a subset of data. To enable secure machine learning for access-controlled data, we propose the notion of in…
▽ More
In today's machine learning (ML) models, any part of the training data can affect the model output. This lack of control for information flow from training data to model output is a major obstacle in training models on sensitive data when access control only allows individual users to access a subset of data. To enable secure machine learning for access-controlled data, we propose the notion of information flow control for machine learning, and develop an extension to the Transformer language model architecture that strictly adheres to the IFC definition we propose. Our architecture controls information flow by limiting the influence of training data from each security domain to a single expert module, and only enables a subset of experts at inference time based on the access control policy.The evaluation using large text and code datasets show that our proposed parametric IFC architecture has minimal (1.9%) performance overhead and can significantly improve model accuracy (by 38% for the text dataset, and between 44%--62% for the code datasets) by enabling training on access-controlled data.
△ Less
Submitted 2 July, 2024; v1 submitted 5 June, 2023;
originally announced June 2023.
-
SelFLoc: Selective Feature Fusion for Large-scale Point Cloud-based Place Recognition
Authors:
Qibo Qiu,
Haiming Gao,
Wenxiao Wang,
Zhiyi Su,
Tian Xie,
Wei Hua,
Xiaofei He
Abstract:
Point cloud-based place recognition is crucial for mobile robots and autonomous vehicles, especially when the global positioning sensor is not accessible. LiDAR points are scattered on the surface of objects and buildings, which have strong shape priors along different axes. To enhance message passing along particular axes, Stacked Asymmetric Convolution Block (SACB) is designed, which is one of t…
▽ More
Point cloud-based place recognition is crucial for mobile robots and autonomous vehicles, especially when the global positioning sensor is not accessible. LiDAR points are scattered on the surface of objects and buildings, which have strong shape priors along different axes. To enhance message passing along particular axes, Stacked Asymmetric Convolution Block (SACB) is designed, which is one of the main contributions in this paper. Comprehensive experiments demonstrate that asymmetric convolution and its corresponding strategies employed by SACB can contribute to the more effective representation of point cloud feature. On this basis, Selective Feature Fusion Block (SFFB), which is formed by stacking point- and channel-wise gating layers in a predefined sequence, is proposed to selectively boost salient local features in certain key regions, as well as to align the features before fusion phase. SACBs and SFFBs are combined to construct a robust and accurate architecture for point cloud-based place recognition, which is termed SelFLoc. Comparative experimental results show that SelFLoc achieves the state-of-the-art (SOTA) performance on the Oxford and other three in-house benchmarks with an improvement of 1.6 absolute percentages on mean average recall@1.
△ Less
Submitted 5 June, 2023; v1 submitted 1 June, 2023;
originally announced June 2023.
-
UP5: Unbiased Foundation Model for Fairness-aware Recommendation
Authors:
Wenyue Hua,
Yingqiang Ge,
Shuyuan Xu,
Jianchao Ji,
Yongfeng Zhang
Abstract:
Recent advances in Foundation Models such as Large Language Models (LLMs) have propelled them to the forefront of Recommender Systems (RS). Despite their utility, there is a growing concern that LLMs might inadvertently perpetuate societal stereotypes, resulting in unfair recommendations. Since fairness is critical for RS as many users take it for decision-making and demand fulfillment, this paper…
▽ More
Recent advances in Foundation Models such as Large Language Models (LLMs) have propelled them to the forefront of Recommender Systems (RS). Despite their utility, there is a growing concern that LLMs might inadvertently perpetuate societal stereotypes, resulting in unfair recommendations. Since fairness is critical for RS as many users take it for decision-making and demand fulfillment, this paper focuses on user-side fairness for LLM-based recommendation where the users may require a recommender system to be fair on specific sensitive features such as gender or age. In this paper, we dive into the extent of unfairness exhibited by LLM-based recommender models based on both T5 and LLaMA backbones, and discuss appropriate methods for promoting equitable treatment of users in LLM-based recommendation models. We introduce a novel Counterfactually-Fair-Prompt (CFP) method towards Unbiased Foundation mOdels (UFO) for fairness-aware LLM-based recommendation. Experiments are conducted on two real-world datasets, MovieLens-1M and Insurance, and compared with both matching-based and sequential-based fairness-aware recommendation models. Results show that CFP achieves better recommendation performance with a high level of fairness. Data and code are open-sourced at https://github.com/agiresearch/UP5.
△ Less
Submitted 29 May, 2024; v1 submitted 20 May, 2023;
originally announced May 2023.
-
A Hybrid 3D Eddy Detection Technique Based on Sea Surface Height and Velocity Field
Authors:
Wei** Hua,
Karen Bemis,
Dujuan Kang,
Sedat Ozer,
Deborah Silver
Abstract:
Eddy detection is a critical task for ocean scientists to understand and analyze ocean circulation. In this paper, we introduce a hybrid eddy detection approach that combines sea surface height (SSH) and velocity fields with geometric criteria defining eddy behavior. Our approach searches for SSH minima and maxima, which oceanographers expect to find at the center of eddies. Geometric criteria are…
▽ More
Eddy detection is a critical task for ocean scientists to understand and analyze ocean circulation. In this paper, we introduce a hybrid eddy detection approach that combines sea surface height (SSH) and velocity fields with geometric criteria defining eddy behavior. Our approach searches for SSH minima and maxima, which oceanographers expect to find at the center of eddies. Geometric criteria are used to verify expected velocity field properties, such as net rotation and symmetry, by tracing velocity components along a circular path surrounding each eddy center. Progressive searches outward and into deeper layers yield each eddy's 3D region of influence. Isolation of each eddy structure from the dataset, using it's cylindrical footprint, facilitates visualization of internal eddy structures using horizontal velocity, vertical velocity, temperature and salinity. A quantitative comparison of Okubo-Weiss vorticity (OW) thresholding, the standard winding angle, and this new SSH-velocity hybrid methods of eddy detection as applied to the Red Sea dataset suggests that detection results are highly dependent on the choices of method, thresholds, and criteria. Our new SSH-velocity hybrid detection approach has the advantages of providing eddy structures with verified rotation properties, 3D visualization of the internal structure of physical properties, and rapid efficient estimations of eddy footprints without calculating streamlines. Our approach combines visualization of internal structure and tracking overall movement to support the study of the transport mechanisms key to understanding the interaction of nutrient distribution and ocean circulation. Our method is applied to three different datasets to showcase the generality of its application.
△ Less
Submitted 31 October, 2023; v1 submitted 14 May, 2023;
originally announced May 2023.
-
Visual Information Extraction in the Wild: Practical Dataset and End-to-end Solution
Authors:
Jianfeng Kuang,
Wei Hua,
Dingkang Liang,
Mingkun Yang,
Deqiang Jiang,
Bo Ren,
Xiang Bai
Abstract:
Visual information extraction (VIE), which aims to simultaneously perform OCR and information extraction in a unified framework, has drawn increasing attention due to its essential role in various applications like understanding receipts, goods, and traffic signs. However, as existing benchmark datasets for VIE mainly consist of document images without the adequate diversity of layout structures,…
▽ More
Visual information extraction (VIE), which aims to simultaneously perform OCR and information extraction in a unified framework, has drawn increasing attention due to its essential role in various applications like understanding receipts, goods, and traffic signs. However, as existing benchmark datasets for VIE mainly consist of document images without the adequate diversity of layout structures, background disturbs, and entity categories, they cannot fully reveal the challenges of real-world applications. In this paper, we propose a large-scale dataset consisting of camera images for VIE, which contains not only the larger variance of layout, backgrounds, and fonts but also much more types of entities. Besides, we propose a novel framework for end-to-end VIE that combines the stages of OCR and information extraction in an end-to-end learning fashion. Different from the previous end-to-end approaches that directly adopt OCR features as the input of an information extraction module, we propose to use contrastive learning to narrow the semantic gap caused by the difference between the tasks of OCR and information extraction. We evaluate the existing end-to-end methods for VIE on the proposed dataset and observe that the performance of these methods has a distinguishable drop from SROIE (a widely used English dataset) to our proposed dataset due to the larger variance of layout and entities. These results demonstrate our dataset is more practical for promoting advanced VIE algorithms. In addition, experiments demonstrate that the proposed VIE method consistently achieves the obvious performance gains on the proposed and SROIE datasets.
△ Less
Submitted 14 June, 2023; v1 submitted 12 May, 2023;
originally announced May 2023.
-
How to Index Item IDs for Recommendation Foundation Models
Authors:
Wenyue Hua,
Shuyuan Xu,
Yingqiang Ge,
Yongfeng Zhang
Abstract:
Recommendation foundation model utilizes large language models (LLM) for recommendation by converting recommendation tasks into natural language tasks. It enables generative recommendation which directly generates the item(s) to recommend rather than calculating a ranking score for each and every candidate item as in traditional recommendation models, simplifying the recommendation pipeline from m…
▽ More
Recommendation foundation model utilizes large language models (LLM) for recommendation by converting recommendation tasks into natural language tasks. It enables generative recommendation which directly generates the item(s) to recommend rather than calculating a ranking score for each and every candidate item as in traditional recommendation models, simplifying the recommendation pipeline from multi-stage filtering to single-stage filtering. To avoid generating excessively long text and hallucinated recommendations when deciding which item(s) to recommend, creating LLM-compatible item IDs to uniquely identify each item is essential for recommendation foundation models. In this study, we systematically examine the item ID creation and indexing problem for recommendation foundation models, using P5 as an example of the backbone LLM. To emphasize the importance of item indexing, we first discuss the issues of several trivial item indexing methods, such as random indexing, title indexing, and independent indexing. We then propose four simple yet effective solutions, including sequential indexing, collaborative indexing, semantic (content-based) indexing, and hybrid indexing. Our study highlights the significant influence of item indexing methods on the performance of LLM-based recommendation, and our results on real-world datasets validate the effectiveness of our proposed solutions. The research also demonstrates how recent advances on language modeling and traditional IR principles such as indexing can help each other for better learning and inference. Source code and data are available at https://github.com/Wenyueh/LLM-RecSys-ID.
△ Less
Submitted 25 September, 2023; v1 submitted 11 May, 2023;
originally announced May 2023.
-
LACoS-BLOOM: Low-rank Adaptation with Contrastive objective on 8 bits Siamese-BLOOM
Authors:
Wen-Yu Hua,
Brian Williams,
Davood Shamsi
Abstract:
Text embeddings are useful features for several NLP applications, such as sentence similarity, text clustering, and semantic search. In this paper, we present a Low-rank Adaptation with a Contrastive objective on top of 8-bit Siamese-BLOOM, a multilingual large language model optimized to produce semantically meaningful word embeddings. The innovation is threefold. First, we cast BLOOM weights to…
▽ More
Text embeddings are useful features for several NLP applications, such as sentence similarity, text clustering, and semantic search. In this paper, we present a Low-rank Adaptation with a Contrastive objective on top of 8-bit Siamese-BLOOM, a multilingual large language model optimized to produce semantically meaningful word embeddings. The innovation is threefold. First, we cast BLOOM weights to 8-bit values. Second, we fine-tune BLOOM with a scalable adapter (LoRA) and 8-bit Adam optimizer for sentence similarity classification. Third, we apply a Siamese architecture on BLOOM model with a contrastive objective to ease the multi-lingual labeled data scarcity. The experiment results show the quality of learned embeddings from LACoS-BLOOM is proportional to the number of model parameters and the amount of unlabeled training data. With the parameter efficient fine-tuning design, we are able to run BLOOM 7.1 billion parameters end-to-end on a single GPU machine with 32GB memory. Compared to previous solution Sentence-BERT, we achieve significant improvement on both English and multi-lingual STS tasks.
△ Less
Submitted 10 May, 2023;
originally announced May 2023.
-
Singularity swap** method for nearly singular integrals based on trapezoidal rule
Authors:
Gang Bao,
Wenmao Hua,
Jun Lai,
**rui Zhang
Abstract:
Accurate evaluation of nearly singular integrals plays an important role in many boundary integral equation based numerical methods. In this paper, we propose a variant of singularity swap** method to accurately evaluate the layer potentials for arbitrarily close targets. Our method is based on the global trapezoidal rule and trigonometric interpolation, resulting in an explicit quadrature formu…
▽ More
Accurate evaluation of nearly singular integrals plays an important role in many boundary integral equation based numerical methods. In this paper, we propose a variant of singularity swap** method to accurately evaluate the layer potentials for arbitrarily close targets. Our method is based on the global trapezoidal rule and trigonometric interpolation, resulting in an explicit quadrature formula. The method achieves spectral accuracy for nearly singular integrals on closed analytic curves. In order to extract the singularity from the complexified distance function, an efficient root finding method is proposed based on contour integration. Through the change of variables, we also extend the quadrature method to integrals on the piecewise analytic curves. Numerical examples for Laplace's and Helmholtz equations show that high order accuracy can be achieved for arbitrarily close field evaluation.
△ Less
Submitted 9 May, 2023;
originally announced May 2023.
-
SOOD: Towards Semi-Supervised Oriented Object Detection
Authors:
Wei Hua,
Dingkang Liang,
**gyu Li,
Xiaolong Liu,
Zhikang Zou,
Xiaoqing Ye,
Xiang Bai
Abstract:
Semi-Supervised Object Detection (SSOD), aiming to explore unlabeled data for boosting object detectors, has become an active task in recent years. However, existing SSOD approaches mainly focus on horizontal objects, leaving multi-oriented objects that are common in aerial images unexplored. This paper proposes a novel Semi-supervised Oriented Object Detection model, termed SOOD, built upon the m…
▽ More
Semi-Supervised Object Detection (SSOD), aiming to explore unlabeled data for boosting object detectors, has become an active task in recent years. However, existing SSOD approaches mainly focus on horizontal objects, leaving multi-oriented objects that are common in aerial images unexplored. This paper proposes a novel Semi-supervised Oriented Object Detection model, termed SOOD, built upon the mainstream pseudo-labeling framework. Towards oriented objects in aerial scenes, we design two loss functions to provide better supervision. Focusing on the orientations of objects, the first loss regularizes the consistency between each pseudo-label-prediction pair (includes a prediction and its corresponding pseudo label) with adaptive weights based on their orientation gap. Focusing on the layout of an image, the second loss regularizes the similarity and explicitly builds the many-to-many relation between the sets of pseudo-labels and predictions. Such a global consistency constraint can further boost semi-supervised learning. Our experiments show that when trained with the two proposed losses, SOOD surpasses the state-of-the-art SSOD methods under various settings on the DOTA-v1.5 benchmark. The code will be available at https://github.com/HamPerdredes/SOOD.
△ Less
Submitted 10 April, 2023;
originally announced April 2023.
-
OpenAGI: When LLM Meets Domain Experts
Authors:
Yingqiang Ge,
Wenyue Hua,
Kai Mei,
Jianchao Ji,
Juntao Tan,
Shuyuan Xu,
Zelong Li,
Yongfeng Zhang
Abstract:
Human Intelligence (HI) excels at combining basic skills to solve complex tasks. This capability is vital for Artificial Intelligence (AI) and should be embedded in comprehensive AI Agents, enabling them to harness expert models for complex task-solving towards Artificial General Intelligence (AGI). Large Language Models (LLMs) show promising learning and reasoning abilities, and can effectively u…
▽ More
Human Intelligence (HI) excels at combining basic skills to solve complex tasks. This capability is vital for Artificial Intelligence (AI) and should be embedded in comprehensive AI Agents, enabling them to harness expert models for complex task-solving towards Artificial General Intelligence (AGI). Large Language Models (LLMs) show promising learning and reasoning abilities, and can effectively use external models, tools, plugins, or APIs to tackle complex problems. In this work, we introduce OpenAGI, an open-source AGI research and development platform designed for solving multi-step, real-world tasks. Specifically, OpenAGI uses a dual strategy, integrating standard benchmark tasks for benchmarking and evaluation, and open-ended tasks including more expandable models, tools, plugins, or APIs for creative problem-solving. Tasks are presented as natural language queries to the LLM, which then selects and executes appropriate models. We also propose a Reinforcement Learning from Task Feedback (RLTF) mechanism that uses task results to improve the LLM's task-solving ability, which creates a self-improving AI feedback loop. While we acknowledge that AGI is a broad and multifaceted research challenge with no singularly defined solution path, the integration of LLMs with domain-specific expert models, inspired by mirroring the blend of general and specialized intelligence in humans, offers a promising approach towards AGI. We are open-sourcing the OpenAGI project's code, dataset, benchmarks, evaluation methods, and the UI demo to foster community involvement in AGI advancement: https://github.com/agiresearch/OpenAGI.
△ Less
Submitted 3 November, 2023; v1 submitted 9 April, 2023;
originally announced April 2023.