Showing 1–46 of 46 results for author: Tu, D

Concise and Precise Context Compression for Tool-Using Language Models

Authors: Yang Xu, Yunlong Feng, Honglin Mu, Yutai Hou, Yitong Li, Xinghao Wang, Wanjun Zhong, Zhongyang Li, Dandan Tu, Qingfu Zhu, Min Zhang, Wanxiang Che

Abstract: Through reading the documentation in the context, tool-using language models can dynamically extend their capability using external tools. The cost is that we have to input lengthy documentation every time the model needs to use the tool, occupying the input window as well as slowing down the decoding process. Given the progress in general-purpose compression, soft context compression is a suita… ▽ More Through reading the documentation in the context, tool-using language models can dynamically extend their capability using external tools. The cost is that we have to input lengthy documentation every time the model needs to use the tool, occupying the input window as well as slowing down the decoding process. Given the progress in general-purpose compression, soft context compression is a suitable approach to alleviate the problem. However, when compressing tool documentation, existing methods suffer from the weaknesses of key information loss (specifically, tool/parameter name errors) and difficulty in adjusting the length of compressed sequences based on documentation lengths. To address these problems, we propose two strategies for compressing tool documentation into concise and precise summary sequences for tool-using language models. 1) Selective compression strategy mitigates key information loss by deliberately retaining key information as raw text tokens. 2) Block compression strategy involves dividing tool documentation into short chunks and then employing a fixed-length compression model to achieve variable-length compression. This strategy facilitates the flexible adjustment of the compression ratio. Results on API-Bank and APIBench show that our approach reaches a performance comparable to the upper-bound baseline under up to 16x compression ratio. △ Less

Submitted 2 July, 2024; originally announced July 2024.

Shear-enhanced Liquid Crystal Spinning of Conjugated Polymer Fibers

Authors: Hao Jiang, Chi-yuan Yang, Deyu Tu, Zhu Chen, Wei Huang, Liang-wen Feng, Hengda Sun, Hongzhi Wang, Simone Fabiano, Meifang Zhu, Gang Wang

Abstract: Conjugated polymer fibers can be used to manufacture various soft fibrous optoelectronic devices, significantly advancing wearable devices and smart textiles. Recently, conjugated polymer-based fibrous electronic devices have been widely used in energy conversion, electrochemical sensing, and human-machine interaction. However, the insufficient mechanical properties of conjugated polymer fibers, t… ▽ More Conjugated polymer fibers can be used to manufacture various soft fibrous optoelectronic devices, significantly advancing wearable devices and smart textiles. Recently, conjugated polymer-based fibrous electronic devices have been widely used in energy conversion, electrochemical sensing, and human-machine interaction. However, the insufficient mechanical properties of conjugated polymer fibers, the difficulty in solution processing semiconductors with rigid main chains, and the challenges in large-scale continuous production have limited their further development in the wearable field. We regulated the pi - pi stacking interactions in conjugated polymer molecules below their critical liquid crystal concentration by applying fluid shear stress. We implemented secondary orientation, leading to the continuous fabrication of anisotropic semiconductor fibers. This strategy enables conjugated polymers with rigid backbones to synergistically enhance the mechanical and semiconductor properties of fibers through liquid crystal spinning. Furthermore, conjugated polymer fibers, exhibiting excellent electrochemical performance and high mechanical strength (600 MPa) that essentially meet the requirements for industrialized preparation, maintain stability under extreme temperatures, radiation, and chemical reagents. Lastly, we have demonstrated logic circuits using semiconductor fiber organic electrochemical transistors, showcasing its application potential in the field of wearable fabric-style logic processing. These findings confirm the importance of the liquid crystalline state and solution control in optimizing the performance of conjugated polymer fibers, thus paving the way for develo** a new generation of soft fiber semiconductor devices. △ Less

Submitted 6 March, 2024; v1 submitted 5 March, 2024; originally announced March 2024.

Organic electrochemical neurons and synapses with ion mediated spiking

Authors: H. Padinhare, C. Yang, D. Tu, J. Gerasimov, A. M. M. Dar, A. A. Moreira, M. Massetti, R. Kroon, D. Bliman, R. Olsson, E. Stavrinidou, M. Berggren, S. Fabiano

Abstract: Future brain-machine interfaces, prosthetics, and intelligent soft robotics will require integrating artificial neuromorphic devices with biological systems. Due to their poor biocompatibility, circuit complexity, low energy efficiency, and operating principles fundamentally different from the ion signal modulation of biology, traditional Silicon-based neuromorphic implementations have limited bio… ▽ More Future brain-machine interfaces, prosthetics, and intelligent soft robotics will require integrating artificial neuromorphic devices with biological systems. Due to their poor biocompatibility, circuit complexity, low energy efficiency, and operating principles fundamentally different from the ion signal modulation of biology, traditional Silicon-based neuromorphic implementations have limited bio-integration potential. Here, we report the first organic electrochemical neurons (OECNs) with ion-modulated spiking, based on allprinted complementary organic electrochemical transistors. We demonstrate facile biointegration of OECNs with Venus Flytrap (Dionaea muscipula) to induce lobe closure upon input stimuli. The OECNs can also be integrated with all-printed organic electrochemical synapses (OECSs), exhibiting short-term plasticity with paired-pulse facilitation and longterm plasticity with retention >1000 s, facilitating Hebbian learning. These soft and flexible OECNs operate below 0.6 V and respond to multiple stimuli, defining a new vista for localized artificial neuronal systems possible to integrate with bio-signaling systems of plants, invertebrates, and vertebrates. △ Less

Submitted 18 January, 2024; originally announced March 2024.

Object Detection in Thermal Images Using Deep Learning for Unmanned Aerial Vehicles

Authors: Minh Dang Tu, Kieu Trang Le, Manh Duong Phung

Abstract: This work presents a neural network model capable of recognizing small and tiny objects in thermal images collected by unmanned aerial vehicles. Our model consists of three parts, the backbone, the neck, and the prediction head. The backbone is developed based on the structure of YOLOv5 combined with the use of a transformer encoder at the end. The neck includes a BI-FPN block combined with the us… ▽ More This work presents a neural network model capable of recognizing small and tiny objects in thermal images collected by unmanned aerial vehicles. Our model consists of three parts, the backbone, the neck, and the prediction head. The backbone is developed based on the structure of YOLOv5 combined with the use of a transformer encoder at the end. The neck includes a BI-FPN block combined with the use of a sliding window and a transformer to increase the information fed into the prediction head. The prediction head carries out the detection by evaluating feature maps with the Sigmoid function. The use of transformers with attention and sliding windows increases recognition accuracy while kee** the model at a reasonable number of parameters and computation requirements for embedded systems. Experiments conducted on public dataset VEDAI and our collected datasets show that our model has a higher accuracy than state-of-the-art methods such as ResNet, Faster RCNN, ComNet, ViT, YOLOv5, SMPNet, and DPNetV3. Experiments on the embedded computer Jetson AGX show that our model achieves a real-time computation speed with a stability rate of over 90%. △ Less

Submitted 13 February, 2024; originally announced February 2024.

Comments: Published in: 2024 IEEE/SICE International Symposium on System Integration (SII)

Intrinsic orbital fourfold anisotropic magnetoresistance in Dirac materials

Authors: Daifeng Tu, Can Wang, Jianhui Zhou

Abstract: Fourfold anisotropic magnetoresistance (AMR) have been widely observed in quantum materials, but the underlying mechanisms remain poorly understood. Here we find, in a variety of three-dimensional Dirac materials that can be unifiedly described by the massive Dirac equation, the intrinsic orbital magnetic moment of electrons vary synchronously with the magnetic field and give rise to a π periodic… ▽ More Fourfold anisotropic magnetoresistance (AMR) have been widely observed in quantum materials, but the underlying mechanisms remain poorly understood. Here we find, in a variety of three-dimensional Dirac materials that can be unifiedly described by the massive Dirac equation, the intrinsic orbital magnetic moment of electrons vary synchronously with the magnetic field and give rise to a π periodic correction to its velocity, further leading to unusual fourfold AMR, dubbed intrinsic orbital fourfold AMR. Our theory not only explains the observation of fourfold AMR in bismuth but also uncovers the nature of the dominant fourfold AMR in thin films of antiferromagnetic topological insulator MnBi2Te4, which arises from the near cancellation of the twofold AMR from the surface states and bulk states due to distinct spin-momentum lockings. Our work provides a new mechanism for creation and manipulation of intrinsic fourfold AMR in both conventional conductors and various topological insulators. △ Less

Submitted 2 February, 2024; originally announced February 2024.

Comments: 6 pages and 2 figures. Comments are welcome

Synergistic Effect of Multi-Walled Carbon Nanotubes and Ladder-Type Conjugated Polymers on the Performance of N-Type Organic Electrochemical Transistors

Authors: S. Zhang, M. Massetti, T. P. Ruoko, D. Tu, C. Y. Yang, X. Liu, Z. Wu, Y. Lee, R. Kroon, P. Persson, H. Y. Woo, M. Berggren, C. Müller, M. Fahlman, S. Fabiano

Abstract: Organic electrochemical transistors (OECTs) have the potential to revolutionize the field of organic bioelectronics. To date, most of the reported OECTs include p-type (semi-)conducting polymers as the channel material, while n-type OECTs are yet at an early stage of development, with the best performing electron-transporting materials still suffering from low transconductance, low electron mobili… ▽ More Organic electrochemical transistors (OECTs) have the potential to revolutionize the field of organic bioelectronics. To date, most of the reported OECTs include p-type (semi-)conducting polymers as the channel material, while n-type OECTs are yet at an early stage of development, with the best performing electron-transporting materials still suffering from low transconductance, low electron mobility, and slow response time. Here, the high electrical conductivity of multi-walled carbon nanotubes (MWCNTs) and the large volumetric capacitance of the ladder-type π-conjugated redox polymer poly(benzimidazobenzophenanthroline) (BBL) are leveraged to develop n-type OECTs with record-high performance. It is demonstrated that the use of MWCNTs enhances the electron mobility by more than one order of magnitude, yielding fast transistor transient response (down to 15 ms) and high uC* (electron mobility x volumetric capacitance) of about 1 F/cmVs. This enables the development of complementary inverters with a voltage gain of > 16 and a large worst-case noise margin at a supply voltage of < 0.6 V, while consuming less than 1 uW of power. △ Less

Submitted 18 January, 2024; originally announced January 2024.

Predicting the activity of chemical compounds based on machine learning approaches

Authors: Do Hoang Tu, Tran Van Lang, Pham Cong Xuyen, Le Mau Long

Abstract: Exploring methods and techniques of machine learning (ML) to address specific challenges in various fields is essential. In this work, we tackle a problem in the domain of Cheminformatics; that is, providing a suitable solution to aid in predicting the activity of a chemical compound to the best extent possible. To address the problem at hand, this study conducts experiments on 100 different combi… ▽ More Exploring methods and techniques of machine learning (ML) to address specific challenges in various fields is essential. In this work, we tackle a problem in the domain of Cheminformatics; that is, providing a suitable solution to aid in predicting the activity of a chemical compound to the best extent possible. To address the problem at hand, this study conducts experiments on 100 different combinations of existing techniques. These solutions are then selected based on a set of criteria that includes the G-means, F1-score, and AUC metrics. The results have been tested on a dataset of about 10,000 chemical compounds from PubChem that have been classified according to their activity △ Less

Submitted 10 September, 2023; originally announced January 2024.

Gemini: A Family of Highly Capable Multimodal Models

Authors: Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hennigan, Benjamin Lee , et al. (1325 additional authors not shown)

Abstract: This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use-cases. Evaluation on a broad range of benchmarks shows that our most-capable Gemini Ultr… ▽ More This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use-cases. Evaluation on a broad range of benchmarks shows that our most-capable Gemini Ultra model advances the state of the art in 30 of 32 of these benchmarks - notably being the first model to achieve human-expert performance on the well-studied exam benchmark MMLU, and improving the state of the art in every one of the 20 multimodal benchmarks we examined. We believe that the new capabilities of the Gemini family in cross-modal reasoning and language understanding will enable a wide variety of use cases. We discuss our approach toward post-training and deploying Gemini models responsibly to users through services including Gemini, Gemini Advanced, Google AI Studio, and Cloud Vertex AI. △ Less

Submitted 17 June, 2024; v1 submitted 18 December, 2023; originally announced December 2023.

Surface skyrmions and dual topological Hall effect in antiferromagnetic topological insulator EuCd$_2$As$_2$

Authors: Min Wu, R. Yang, Xiangde Zhu, Yixiong Ren, Ang Qian, Yongjie Xie, Changming Yue, Yong Nie, Xiang Yuan, Ning Wang, Daifeng Tu, Ding Li, Yuyan Han, Zhaosheng Wang, Yaomin Dai, Guolin Zheng, Jianhui Zhou, Wei Ning, Xianggang Qiu, Mingliang Tian

Abstract: In this work, we synthesized single crystal of EuCd$_2$As$_2$, which exhibits A-type antiferromagnetic (AFM) order with in-plane spin orientation below $T_N$ = 9.5~K.Optical spectroscopy and transport measurements suggest its topological insulator (TI) nature with an insulating gap around 0.1eV. Remarkably, a dual topological Hall resistivity that exhibits same magnitude but opposite signs in the… ▽ More In this work, we synthesized single crystal of EuCd$_2$As$_2$, which exhibits A-type antiferromagnetic (AFM) order with in-plane spin orientation below $T_N$ = 9.5~K.Optical spectroscopy and transport measurements suggest its topological insulator (TI) nature with an insulating gap around 0.1eV. Remarkably, a dual topological Hall resistivity that exhibits same magnitude but opposite signs in the positive to negative and negative to positive magnetic field hysteresis branches emerges below 20~K. With magnetic force microscopy (MFM) images and numerical simulations, we attribute the dual topological Hall effect to the Néel-type skyrmions stabilized by the interactions between topological surface states and magnetism, and the sign reversal in different hysteresis branches indicates potential coexistence of skyrmions and antiskyrmions. Our work uncovers a unique two-dimensional (2D) magnetism on the surface of intrinsic AFM TI, providing a promising platform for novel topological quantum states and AFM spintronic applications. △ Less

Submitted 27 November, 2023; originally announced November 2023.

Comments: 7 pages, 3 figures

DocStormer: Revitalizing Multi-Degraded Colored Document Images to Pristine PDF

Authors: Chaowei Liu, Jichun Li, Yihua Teng, Chaoqun Wang, Nuo Xu, Jihao Wu, Dandan Tu

Abstract: For capturing colored document images, e.g. posters and magazines, it is common that multiple degradations such as shadows, wrinkles, etc., are simultaneously introduced due to external factors. Restoring multi-degraded colored document images is a great challenge, yet overlooked, as most existing algorithms focus on enhancing color-ignored document images via binarization. Thus, we propose DocSto… ▽ More For capturing colored document images, e.g. posters and magazines, it is common that multiple degradations such as shadows, wrinkles, etc., are simultaneously introduced due to external factors. Restoring multi-degraded colored document images is a great challenge, yet overlooked, as most existing algorithms focus on enhancing color-ignored document images via binarization. Thus, we propose DocStormer, a novel algorithm designed to restore multi-degraded colored documents to their potential pristine PDF. The contributions are: firstly, we propose a "Perceive-then-Restore" paradigm with a reinforced transformer block, which more effectively encodes and utilizes the distribution of degradations. Secondly, we are the first to utilize GAN and pristine PDF magazine images to narrow the distribution gap between the enhanced results and PDF images, in pursuit of less degradation and better visual quality. Thirdly, we propose a non-parametric strategy, PFILI, which enables a smaller training scale and larger testing resolutions with acceptable detail trade-off, while saving memory and inference time. Fourthly, we are the first to propose a novel Multi-Degraded Colored Document image Enhancing dataset, named MD-CDE, for both training and evaluation. Experimental results show that the DocStormer exhibits superior performance, capable of revitalizing multi-degraded colored documents into their potential pristine digital versions, which fills the current academic gap from the perspective of method, data, and task. △ Less

Submitted 27 October, 2023; originally announced October 2023.

Joint Gaze-Location and Gaze-Object Detection

Authors: Danyang Tu, Wei Shen, Wei Sun, Xiongkuo Min, Guangtao Zhai

Abstract: This paper proposes an efficient and effective method for joint gaze location detection (GL-D) and gaze object detection (GO-D), \emph{i.e.}, gaze following detection. Current approaches frame GL-D and GO-D as two separate tasks, employing a multi-stage framework where human head crops must first be detected and then be fed into a subsequent GL-D sub-network, which is further followed by an additi… ▽ More This paper proposes an efficient and effective method for joint gaze location detection (GL-D) and gaze object detection (GO-D), \emph{i.e.}, gaze following detection. Current approaches frame GL-D and GO-D as two separate tasks, employing a multi-stage framework where human head crops must first be detected and then be fed into a subsequent GL-D sub-network, which is further followed by an additional object detector for GO-D. In contrast, we reframe the gaze following detection task as detecting human head locations and their gaze followings simultaneously, aiming at jointly detect human gaze location and gaze object in a unified and single-stage pipeline. To this end, we propose GTR, short for \underline{G}aze following detection \underline{TR}ansformer, streamlining the gaze following detection pipeline by eliminating all additional components, leading to the first unified paradigm that unites GL-D and GO-D in a fully end-to-end manner. GTR enables an iterative interaction between holistic semantics and human head features through a hierarchical structure, inferring the relations of salient objects and human gaze from the global image context and resulting in an impressive accuracy. Concretely, GTR achieves a 12.1 mAP gain ($\mathbf{25.1}\%$) on GazeFollowing and a 18.2 mAP gain ($\mathbf{43.3\%}$) on VideoAttentionTarget for GL-D, as well as a 19 mAP improvement ($\mathbf{45.2\%}$) on GOO-Real for GO-D. Meanwhile, unlike existing systems detecting gaze following sequentially due to the need for a human head as input, GTR has the flexibility to comprehend any number of people's gaze followings simultaneously, resulting in high efficiency. Specifically, GTR introduces over a $\times 9$ improvement in FPS and the relative gap becomes more pronounced as the human number grows. △ Less

Submitted 26 August, 2023; originally announced August 2023.

Comments: Technical Report. arXiv admin note: text overlap with arXiv:2203.10433

Agglomerative Transformer for Human-Object Interaction Detection

Authors: Danyang Tu, Wei Sun, Guangtao Zhai, Wei Shen

Abstract: We propose an agglomerative Transformer (AGER) that enables Transformer-based human-object interaction (HOI) detectors to flexibly exploit extra instance-level cues in a single-stage and end-to-end manner for the first time. AGER acquires instance tokens by dynamically clustering patch tokens and aligning cluster centers to instances with textual guidance, thus enjoying two benefits: 1) Integralit… ▽ More We propose an agglomerative Transformer (AGER) that enables Transformer-based human-object interaction (HOI) detectors to flexibly exploit extra instance-level cues in a single-stage and end-to-end manner for the first time. AGER acquires instance tokens by dynamically clustering patch tokens and aligning cluster centers to instances with textual guidance, thus enjoying two benefits: 1) Integrality: each instance token is encouraged to contain all discriminative feature regions of an instance, which demonstrates a significant improvement in the extraction of different instance-level cues and subsequently leads to a new state-of-the-art performance of HOI detection with 36.75 mAP on HICO-Det. 2) Efficiency: the dynamical clustering mechanism allows AGER to generate instance tokens jointly with the feature learning of the Transformer encoder, eliminating the need of an additional object detector or instance decoder in prior methods, thus allowing the extraction of desirable extra cues for HOI detection in a single-stage and end-to-end pipeline. Concretely, AGER reduces GFLOPs by 8.5% and improves FPS by 36%, even compared to a vanilla DETR-like pipeline without extra cue extraction. △ Less

Submitted 16 August, 2023; originally announced August 2023.

Comments: Accepted by ICCV'23

Generating Synergistic Formulaic Alpha Collections via Reinforcement Learning

Authors: Shuo Yu, Hongyan Xue, Xiang Ao, Feiyang Pan, Jia He, Dandan Tu, Qing He

Abstract: In the field of quantitative trading, it is common practice to transform raw historical stock data into indicative signals for the market trend. Such signals are called alpha factors. Alphas in formula forms are more interpretable and thus favored by practitioners concerned with risk. In practice, a set of formulaic alphas is often used together for better modeling precision, so we need to find sy… ▽ More In the field of quantitative trading, it is common practice to transform raw historical stock data into indicative signals for the market trend. Such signals are called alpha factors. Alphas in formula forms are more interpretable and thus favored by practitioners concerned with risk. In practice, a set of formulaic alphas is often used together for better modeling precision, so we need to find synergistic formulaic alpha sets that work well together. However, most traditional alpha generators mine alphas one by one separately, overlooking the fact that the alphas would be combined later. In this paper, we propose a new alpha-mining framework that prioritizes mining a synergistic set of alphas, i.e., it directly uses the performance of the downstream combination model to optimize the alpha generator. Our framework also leverages the strong exploratory capabilities of reinforcement learning~(RL) to better explore the vast search space of formulaic alphas. The contribution to the combination models' performance is assigned to be the return used in the RL process, driving the alpha generator to find better alphas that improve upon the current set. Experimental evaluations on real-world stock market data demonstrate both the effectiveness and the efficiency of our framework for stock trend forecasting. The investment simulation results show that our framework is able to achieve higher returns compared to previous approaches. △ Less

Submitted 25 May, 2023; originally announced June 2023.

Comments: Accepted by KDD '23, ADS track

Auto-Validate by-History: Auto-Program Data Quality Constraints to Validate Recurring Data Pipelines

Authors: Dezhan Tu, Yeye He, Weiwei Cui, Song Ge, Haidong Zhang, Han Shi, Dongmei Zhang, Surajit Chaudhuri

Abstract: Data pipelines are widely employed in modern enterprises to power a variety of Machine-Learning (ML) and Business-Intelligence (BI) applications. Crucially, these pipelines are \emph{recurring} (e.g., daily or hourly) in production settings to keep data updated so that ML models can be re-trained regularly, and BI dashboards refreshed frequently. However, data quality (DQ) issues can often creep i… ▽ More Data pipelines are widely employed in modern enterprises to power a variety of Machine-Learning (ML) and Business-Intelligence (BI) applications. Crucially, these pipelines are \emph{recurring} (e.g., daily or hourly) in production settings to keep data updated so that ML models can be re-trained regularly, and BI dashboards refreshed frequently. However, data quality (DQ) issues can often creep into recurring pipelines because of upstream schema and data drift over time. As modern enterprises operate thousands of recurring pipelines, today data engineers have to spend substantial efforts to \emph{manually} monitor and resolve DQ issues, as part of their DataOps and MLOps practices. Given the high human cost of managing large-scale pipeline operations, it is imperative that we can \emph{automate} as much as possible. In this work, we propose Auto-Validate-by-History (AVH) that can automatically detect DQ issues in recurring pipelines, leveraging rich statistics from historical executions. We formalize this as an optimization problem, and develop constant-factor approximation algorithms with provable precision guarantees. Extensive evaluations using 2000 production data pipelines at Microsoft demonstrate the effectiveness and efficiency of AVH. △ Less

Submitted 4 June, 2023; originally announced June 2023.

Comments: full version of a paper accepted to KDD 2023

Masked Autoencoders as Image Processors

Authors: Huiyu Duan, Wei Shen, Xiongkuo Min, Danyang Tu, Long Teng, Jia Wang, Guangtao Zhai

Abstract: Transformers have shown significant effectiveness for various vision tasks including both high-level vision and low-level vision. Recently, masked autoencoders (MAE) for feature pre-training have further unleashed the potential of Transformers, leading to state-of-the-art performances on various high-level vision tasks. However, the significance of MAE pre-training on low-level vision tasks has no… ▽ More Transformers have shown significant effectiveness for various vision tasks including both high-level vision and low-level vision. Recently, masked autoencoders (MAE) for feature pre-training have further unleashed the potential of Transformers, leading to state-of-the-art performances on various high-level vision tasks. However, the significance of MAE pre-training on low-level vision tasks has not been sufficiently explored. In this paper, we show that masked autoencoders are also scalable self-supervised learners for image processing tasks. We first present an efficient Transformer model considering both channel attention and shifted-window-based self-attention termed CSformer. Then we develop an effective MAE architecture for image processing (MAEIP) tasks. Extensive experimental results show that with the help of MAEIP pre-training, our proposed CSformer achieves state-of-the-art performance on various image processing tasks, including Gaussian denoising, real image denoising, single-image motion deblurring, defocus deblurring, and image deraining. △ Less

Submitted 30 March, 2023; originally announced March 2023.

MD-VQA: Multi-Dimensional Quality Assessment for UGC Live Videos

Authors: Zicheng Zhang, Wei Wu, Wei Sun, Dangyang Tu, Wei Lu, Xiongkuo Min, Ying Chen, Guangtao Zhai

Abstract: User-generated content (UGC) live videos are often bothered by various distortions during capture procedures and thus exhibit diverse visual qualities. Such source videos are further compressed and transcoded by media server providers before being distributed to end-users. Because of the flourishing of UGC live videos, effective video quality assessment (VQA) tools are needed to monitor and percep… ▽ More User-generated content (UGC) live videos are often bothered by various distortions during capture procedures and thus exhibit diverse visual qualities. Such source videos are further compressed and transcoded by media server providers before being distributed to end-users. Because of the flourishing of UGC live videos, effective video quality assessment (VQA) tools are needed to monitor and perceptually optimize live streaming videos in the distributing process. In this paper, we address \textbf{UGC Live VQA} problems by constructing a first-of-a-kind subjective UGC Live VQA database and develo** an effective evaluation tool. Concretely, 418 source UGC videos are collected in real live streaming scenarios and 3,762 compressed ones at different bit rates are generated for the subsequent subjective VQA experiments. Based on the built database, we develop a \underline{M}ulti-\underline{D}imensional \underline{VQA} (\textbf{MD-VQA}) evaluator to measure the visual quality of UGC live videos from semantic, distortion, and motion aspects respectively. Extensive experimental results show that MD-VQA achieves state-of-the-art performance on both our UGC Live VQA database and existing compressed UGC VQA databases. △ Less

Submitted 19 April, 2023; v1 submitted 27 March, 2023; originally announced March 2023.

Comments: Accepted to CVPR2023

Oral-3Dv2: 3D Oral Reconstruction from Panoramic X-Ray Imaging with Implicit Neural Representation

Authors: Weinan Song, Haoxin Zheng, Dezhan Tu, Chengwen Liang, Lei He

Abstract: 3D reconstruction of medical imaging from 2D images has become an increasingly interesting topic with the development of deep learning models in recent years. Previous studies in 3D reconstruction from limited X-ray images mainly rely on learning from paired 2D and 3D images, where the reconstruction quality relies on the scale and variation of collected data. This has brought significant challeng… ▽ More 3D reconstruction of medical imaging from 2D images has become an increasingly interesting topic with the development of deep learning models in recent years. Previous studies in 3D reconstruction from limited X-ray images mainly rely on learning from paired 2D and 3D images, where the reconstruction quality relies on the scale and variation of collected data. This has brought significant challenges in the collection of training data, as only a tiny fraction of patients take two types of radiation examinations in the same period. Although simulation from higher-dimension images could solve this problem, the variance between real and simulated data could bring great uncertainty at the same time. In oral reconstruction, the situation becomes more challenging as only a single panoramic X-ray image is available, where models need to infer the curved shape by prior individual knowledge. To overcome these limitations, we propose Oral-3Dv2 to solve this cross-dimension translation problem in dental healthcare by learning solely on projection information, i.e., the projection image and trajectory of the X-ray tube. Our model learns to represent the 3D oral structure in an implicit way by map** 2D coordinates into density values of voxels in the 3D space. To improve efficiency and effectiveness, we utilize a multi-head model that predicts a bunch of voxel values in 3D space simultaneously from a 2D coordinate in the axial plane and the dynamic sampling strategy to refine details of the density distribution in the reconstruction result. Extensive experiments in simulated and real data show that our model significantly outperforms existing state-of-the-art models without learning from paired images or prior individual knowledge. To the best of our knowledge, this is the first work of a non-adversarial-learning-based model in 3D radiology reconstruction from a single panoramic X-ray image. △ Less

Submitted 3 September, 2023; v1 submitted 21 March, 2023; originally announced March 2023.

Style Miner: Find Significant and Stable Explanatory Factors in Time Series with Constrained Reinforcement Learning

Authors: Dapeng Li, Feiyang Pan, Jia He, Zhiwei Xu, Dandan Tu, Guoliang Fan

Abstract: In high-dimensional time-series analysis, it is essential to have a set of key factors (namely, the style factors) that explain the change of the observed variable. For example, volatility modeling in finance relies on a set of risk factors, and climate change studies in climatology rely on a set of causal factors. The ideal low-dimensional style factors should balance significance (with high expl… ▽ More In high-dimensional time-series analysis, it is essential to have a set of key factors (namely, the style factors) that explain the change of the observed variable. For example, volatility modeling in finance relies on a set of risk factors, and climate change studies in climatology rely on a set of causal factors. The ideal low-dimensional style factors should balance significance (with high explanatory power) and stability (consistent, no significant fluctuations). However, previous supervised and unsupervised feature extraction methods can hardly address the tradeoff. In this paper, we propose Style Miner, a reinforcement learning method to generate style factors. We first formulate the problem as a Constrained Markov Decision Process with explanatory power as the return and stability as the constraint. Then, we design fine-grained immediate rewards and costs and use a Lagrangian heuristic to balance them adaptively. Experiments on real-world financial data sets show that Style Miner outperforms existing learning-based methods by a large margin and achieves a relatively 10% gain in R-squared explanatory power compared to the industry-renowned factors proposed by human experts. △ Less

Submitted 21 March, 2023; originally announced March 2023.

Comments: 9 pages, 6 figures

Stable Attribute Group Editing for Reliable Few-shot Image Generation

Authors: Guanqi Ding, Xinzhe Han, Shuhui Wang, Xin **, Dandan Tu, Qingming Huang

Abstract: Few-shot image generation aims to generate data of an unseen category based on only a few samples. Apart from basic content generation, a bunch of downstream applications hopefully benefit from this task, such as low-data detection and few-shot classification. To achieve this goal, the generated images should guarantee category retention for classification beyond the visual quality and diversity.… ▽ More Few-shot image generation aims to generate data of an unseen category based on only a few samples. Apart from basic content generation, a bunch of downstream applications hopefully benefit from this task, such as low-data detection and few-shot classification. To achieve this goal, the generated images should guarantee category retention for classification beyond the visual quality and diversity. In our preliminary work, we present an ``editing-based'' framework Attribute Group Editing (AGE) for reliable few-shot image generation, which largely improves the generation performance. Nevertheless, AGE's performance on downstream classification is not as satisfactory as expected. This paper investigates the class inconsistency problem and proposes Stable Attribute Group Editing (SAGE) for more stable class-relevant image generation. SAGE takes use of all given few-shot images and estimates a class center embedding based on the category-relevant attribute dictionary. Meanwhile, according to the projection weights on the category-relevant attribute dictionary, we can select category-irrelevant attributes from the similar seen categories. Consequently, SAGE injects the whole distribution of the novel class into StyleGAN's latent space, thus largely remains the category retention and stability of the generated images. Going one step further, we find that class inconsistency is a common problem in GAN-generated images for downstream classification. Even though the generated images look photo-realistic and requires no category-relevant editing, they are usually of limited help for downstream classification. We systematically discuss this issue from both the generative model and classification model perspectives, and propose to boost the downstream classification performance of SAGE by enhancing the pixel and frequency components. △ Less

Submitted 31 January, 2023; originally announced February 2023.

A biologically interfaced evolvable organic pattern classifier

Authors: Jennifer Gerasimov, Deyu Tu, Vivek Hitaishi, Padinhare Cholakkal Harikesh, Chi-Yuan Yang, Tobias Abrahamsson, Meysam Rad, Mary J. Donahue, Malin Silverå Ejneby, Magnus Berggren, Robert Forchheimer, Simone Fabiano

Abstract: Future brain-computer interfaces will require local and highly individualized signal processing of fully integrated electronic circuits within the nervous system and other living tissue. New devices will need to be developed that can receive data from a sensor array, process data into meaningful information, and translate that information into a format that living systems can interpret. Here, we r… ▽ More Future brain-computer interfaces will require local and highly individualized signal processing of fully integrated electronic circuits within the nervous system and other living tissue. New devices will need to be developed that can receive data from a sensor array, process data into meaningful information, and translate that information into a format that living systems can interpret. Here, we report the first example of interfacing a hardware-based pattern classifier with a biological nerve. The classifier implements the Widrow-Hoff learning algorithm on an array of evolvable organic electrochemical transistors (EOECTs). The EOECTs' channel conductance is modulated in situ by electropolymerizing the semiconductor material within the channel, allowing for low voltage operation, high reproducibility, and an improvement in state retention of two orders of magnitude over state-of-the-art OECT devices. The organic classifier is interfaced with a biological nerve using an organic electrochemical spiking neuron to translate the classifier's output to a simulated action potential. The latter is then used to stimulate muscle contraction selectively based on the input pattern, thus paving the way for the development of closed-loop therapeutic systems. △ Less

Submitted 29 November, 2022; originally announced November 2022.

Stable ion-tunable antiambipolarity in mixed ion-electron conducting polymers enables biorealistic artificial neurons

Authors: Padinhare Cholakkal Harikesh, Chi-Yuan Yang, Han-Yan Wu, Silan Zhang, Jun-Da Huang, Magnus Berggren, Deyu Tu, Simone Fabiano

Abstract: Bio-integrated neuromorphic systems promise for new protocols to record and regulate the signaling of biological systems. Making such artificial neural circuits successful requires minimal circuit complexity and ion-based operating mechanisms similar to that of biology. However, simple leaky integrate-and-fire model neurons, commonly realized in either silicon or organic semiconductor neuromorphic… ▽ More Bio-integrated neuromorphic systems promise for new protocols to record and regulate the signaling of biological systems. Making such artificial neural circuits successful requires minimal circuit complexity and ion-based operating mechanisms similar to that of biology. However, simple leaky integrate-and-fire model neurons, commonly realized in either silicon or organic semiconductor neuromorphic systems, can emulate only a few neural features. More functional neuron models, based on traditional complex Si-based complementary-metal-oxide-semiconductor (CMOS) or negative differential resistance (NDR) device circuits, are complicated to fabricate, not biocompatible, and lack ion- and chemical-based modulation features. Here we report a biorealistic conductance-based organic electrochemical neuron (c-OECN) using a mixed ion-electron conducting ladder-type polymer with reliable ion-tunable antiambipolarity. The latter is used to emulate the activation/inactivation of Na channels and delayed activation of K channels of biological neurons. These c-OECNs can then spike at bioplausible frequencies nearing 100 Hz, emulate most critical biological neural features, demonstrate stochastic spiking, and enable neurotransmitter and Ca2+-based spiking modulation. These combined features are impossible to achieve using previous technologies. △ Less

Submitted 19 October, 2022; originally announced October 2022.

Fully 3D-Printed Organic Electrochemical Transistors

Authors: Matteo Massetti, Silan Zhang, Harikesh Padinare, Bernhard Burtscher, Chiara Diacci, Daniel T. Simon, Xianjie Liu, Mats Fahlman, Deyu Tu, Magnus Berggren, Simone Fabiano

Abstract: Organic electrochemical transistors (OECTs) are currently being investigated for various applications, ranging from sensors to logics and neuromorphic hardware. The fabrication process must be compatible with flexible and scalable digital techniques to address this wide spectrum of applications. Here, we report a direct-write additive process to fabricate fully 3D printed OECTs. We developed 3D pr… ▽ More Organic electrochemical transistors (OECTs) are currently being investigated for various applications, ranging from sensors to logics and neuromorphic hardware. The fabrication process must be compatible with flexible and scalable digital techniques to address this wide spectrum of applications. Here, we report a direct-write additive process to fabricate fully 3D printed OECTs. We developed 3D printable conducting, semiconducting, insulating, and electrolyte inks to achieve this. The 3D-printed OECTs, operating in the depletion mode, can be fabricated on thin and flexible substrates, yielding high mechanical and environmental stability. We also developed a 3D printable nanocellulose formulation for the OECT substrate, demonstrating one of the first examples of fully 3D printed electronic devices. Good dopamine biosensing capabilities (limit of detection down to 6 uM without metal gate electrodes) and long-term (~1 hour) synapses response underscore that the present OECT manufacturing strategy is suitable for diverse applications requiring rapid design change and digitally enabled direct-write techniques. △ Less

Submitted 14 September, 2022; originally announced September 2022.

Consistent Covariance estimation for stratum imbalances under minimization method for covariate-adaptive randomization

Authors: Zixuan Zhao, Yanglei Song, Wenyu Jiang, Dongsheng Tu

Abstract: Pocock and Simon's minimization method is a popular approach for covariate-adaptive randomization in clinical trials. Valid statistical inference with data collected under the minimization method requires the knowledge of the limiting covariance matrix of within-stratum imbalances, whose existence is only recently established. In this work, we propose a bootstrap-based estimator for this limit and… ▽ More Pocock and Simon's minimization method is a popular approach for covariate-adaptive randomization in clinical trials. Valid statistical inference with data collected under the minimization method requires the knowledge of the limiting covariance matrix of within-stratum imbalances, whose existence is only recently established. In this work, we propose a bootstrap-based estimator for this limit and establish its consistency, in particular, by Le Cam's third lemma. As an application, we consider in simulation studies adjustments to existing robust tests for treatment effects with survival data by the proposed estimator. It shows that the adjusted tests achieve a size close to the nominal level, and unlike other designs, the robust tests without adjustment may have an asymptotic size inflation issue under the minimization method. △ Less

Submitted 26 December, 2023; v1 submitted 26 September, 2022; originally announced September 2022.

Comments: 29 pages, peer reviewed version, will appear in Scandinavian Journal of Statistics

In-plane anomalous Hall effect in PT-symmetric antiferromagnetic materials

Authors: ** Cao, Wei Jiang, Xiao-** Li, Daifeng Tu, Jiadong Zhou, Jianhui Zhou, Yugui Yao

Abstract: Anomalous Hall effect (AHE), a protocol of various low-power dissipation quantum phenomena and a fundamental precursor of intriguing topological phases of matter, is usually observed in ferromagnetic materials with orthogonal configuration between the electric field, magnetization and the Hall current. Here, based on the symmetry analysis, we find an unconventional AHE induced by the in-plane magn… ▽ More Anomalous Hall effect (AHE), a protocol of various low-power dissipation quantum phenomena and a fundamental precursor of intriguing topological phases of matter, is usually observed in ferromagnetic materials with orthogonal configuration between the electric field, magnetization and the Hall current. Here, based on the symmetry analysis, we find an unconventional AHE induced by the in-plane magnetic field (IPAHE) via spin-canting effect in $\mathcal{PT}$ symmetric antiferromagnetic (AFM) systems, featuring a linear dependence of magnetic field and 2$π$ angle periodicity with a comparable magnitude as conventional AHE. We demonstrate the key findings in the known AFM Dirac semimetal CuMnAs and a new kind of AFM heterodimensional VS$_2$-VS superlattice with a nodal-line Fermi surface and also briefly discuss the experimental detection. Our work provides an efficient pathway to search and/or design realistic materials for novel IPAHE that could greatly facilitate their application in AFM spintronic devices. △ Less

Submitted 30 August, 2022; originally announced August 2022.

Comments: 6 pages, 4 figures, 1 table

Photonic sampled and quantized analog-to-digital converters on thin-film lithium niobate platform

Authors: Donghe Tu, Xingrui Huang, Yang Liu, Zhiguo Yu, Zhiyong Li

Abstract: In this paper, an on-chip photonic sampled and quantized analog-to-digital converter (ADC) on thin-film lithium niobate platform is experimentally demonstrated. Using two phase modulators as a sampler and a 5$\times$5 multimode interference (MMI) coupler as a quantizer, an 1 GHz sinusoidal analog input signal was successfully converted to a digitized output with a 20 GSample/s sampling rate. To ev… ▽ More In this paper, an on-chip photonic sampled and quantized analog-to-digital converter (ADC) on thin-film lithium niobate platform is experimentally demonstrated. Using two phase modulators as a sampler and a 5$\times$5 multimode interference (MMI) coupler as a quantizer, an 1 GHz sinusoidal analog input signal was successfully converted to a digitized output with a 20 GSample/s sampling rate. To evaluate the system performance, the quantization curves together with the transfer function of the ADC were measured. The experimental effective number of bits (ENOB) was 3.17 bit. The demonstrated device is capable of operating at a high frequency up to 70 GHz, making it a promising solution for on-chip ultra-high speed analog-to-digital conversion. △ Less

Submitted 28 July, 2022; originally announced July 2022.

Video-based Human-Object Interaction Detection from Tubelet Tokens

Authors: Danyang Tu, Wei Sun, Xiongkuo Min, Guangtao Zhai, Wei Shen

Abstract: We present a novel vision Transformer, named TUTOR, which is able to learn tubelet tokens, served as highly-abstracted spatiotemporal representations, for video-based human-object interaction (V-HOI) detection. The tubelet tokens structurize videos by agglomerating and linking semantically-related patch tokens along spatial and temporal domains, which enjoy two benefits: 1) Compactness: each tubel… ▽ More We present a novel vision Transformer, named TUTOR, which is able to learn tubelet tokens, served as highly-abstracted spatiotemporal representations, for video-based human-object interaction (V-HOI) detection. The tubelet tokens structurize videos by agglomerating and linking semantically-related patch tokens along spatial and temporal domains, which enjoy two benefits: 1) Compactness: each tubelet token is learned by a selective attention mechanism to reduce redundant spatial dependencies from others; 2) Expressiveness: each tubelet token is enabled to align with a semantic instance, i.e., an object or a human, across frames, thanks to agglomeration and linking. The effectiveness and efficiency of TUTOR are verified by extensive experiments. Results shows our method outperforms existing works by large margins, with a relative mAP gain of $16.14\%$ on VidHOI and a 2 points gain on CAD-120 as well as a $4 \times$ speedup. △ Less

Submitted 4 June, 2022; originally announced June 2022.

Saliency in Augmented Reality

Authors: Huiyu Duan, Wei Shen, Xiongkuo Min, Danyang Tu, **g Li, Guangtao Zhai

Abstract: With the rapid development of multimedia technology, Augmented Reality (AR) has become a promising next-generation mobile platform. The primary theory underlying AR is human visual confusion, which allows users to perceive the real-world scenes and augmented contents (virtual-world scenes) simultaneously by superimposing them together. To achieve good Quality of Experience (QoE), it is important t… ▽ More With the rapid development of multimedia technology, Augmented Reality (AR) has become a promising next-generation mobile platform. The primary theory underlying AR is human visual confusion, which allows users to perceive the real-world scenes and augmented contents (virtual-world scenes) simultaneously by superimposing them together. To achieve good Quality of Experience (QoE), it is important to understand the interaction between two scenarios, and harmoniously display AR contents. However, studies on how this superimposition will influence the human visual attention are lacking. Therefore, in this paper, we mainly analyze the interaction effect between background (BG) scenes and AR contents, and study the saliency prediction problem in AR. Specifically, we first construct a Saliency in AR Dataset (SARD), which contains 450 BG images, 450 AR images, as well as 1350 superimposed images generated by superimposing BG and AR images in pair with three mixing levels. A large-scale eye-tracking experiment among 60 subjects is conducted to collect eye movement data. To better predict the saliency in AR, we propose a vector quantized saliency prediction method and generalize it for AR saliency prediction. For comparison, three benchmark methods are proposed and evaluated together with our proposed method on our SARD. Experimental results demonstrate the superiority of our proposed method on both of the common saliency prediction problem and the AR saliency prediction problem over benchmark methods. Our dataset and code are available at: https://github.com/DuanHuiyu/ARSaliency. △ Less

Submitted 12 July, 2022; v1 submitted 18 April, 2022; originally announced April 2022.

Unsupervised Coherent Video Cartoonization with Perceptual Motion Consistency

Authors: Zhenhuan Liu, Liang Li, Huajie Jiang, Xin **, Dandan Tu, Shuhui Wang, Zheng-Jun Zha

Abstract: In recent years, creative content generations like style transfer and neural photo editing have attracted more and more attention. Among these, cartoonization of real-world scenes has promising applications in entertainment and industry. Different from image translations focusing on improving the style effect of generated images, video cartoonization has additional requirements on the temporal con… ▽ More In recent years, creative content generations like style transfer and neural photo editing have attracted more and more attention. Among these, cartoonization of real-world scenes has promising applications in entertainment and industry. Different from image translations focusing on improving the style effect of generated images, video cartoonization has additional requirements on the temporal consistency. In this paper, we propose a spatially-adaptive semantic alignment framework with perceptual motion consistency for coherent video cartoonization in an unsupervised manner. The semantic alignment module is designed to restore deformation of semantic structure caused by spatial information lost in the encoder-decoder architecture. Furthermore, we devise the spatio-temporal correlative map as a style-independent, global-aware regularization on the perceptual motion consistency. Deriving from similarity measurement of high-level features in photo and cartoon frames, it captures global semantic information beyond raw pixel-value in optical flow. Besides, the similarity measurement disentangles temporal relationships from domain-specific style properties, which helps regularize the temporal consistency without hurting style effects of cartoon images. Qualitative and quantitative experiments demonstrate our method is able to generate highly stylistic and temporal consistent cartoon videos. △ Less

Submitted 2 April, 2022; originally announced April 2022.

Intrinsic Bias Identification on Medical Image Datasets

Authors: Shijie Zhang, Lanjun Wang, Lian Ding, An-an Liu, Senhua Zhu, Dandan Tu

Abstract: Machine learning based medical image analysis highly depends on datasets. Biases in the dataset can be learned by the model and degrade the generalizability of the applications. There are studies on debiased models. However, scientists and practitioners are difficult to identify implicit biases in the datasets, which causes lack of reliable unbias test datasets to valid models. To tackle this issu… ▽ More Machine learning based medical image analysis highly depends on datasets. Biases in the dataset can be learned by the model and degrade the generalizability of the applications. There are studies on debiased models. However, scientists and practitioners are difficult to identify implicit biases in the datasets, which causes lack of reliable unbias test datasets to valid models. To tackle this issue, we first define the data intrinsic bias attribute, and then propose a novel bias identification framework for medical image datasets. The framework contains two major components, KlotskiNet and Bias Discriminant Direction Analysis(bdda), where KlostkiNet is to build the map** which makes backgrounds to distinguish positive and negative samples and bdda provides a theoretical solution on determining bias attributes. Experimental results on three datasets show the effectiveness of the bias attributes discovered by the framework. △ Less

Submitted 29 March, 2022; v1 submitted 24 March, 2022; originally announced March 2022.

Comments: 19pages, 12 figures

Iwin: Human-Object Interaction Detection via Transformer with Irregular Windows

Authors: Danyang Tu, Xiongkuo Min, Huiyu Duan, Guodong Guo, Guangtao Zhai, Wei Shen

Abstract: This paper presents a new vision Transformer, named Iwin Transformer, which is specifically designed for human-object interaction (HOI) detection, a detailed scene understanding task involving a sequential process of human/object detection and interaction recognition. Iwin Transformer is a hierarchical Transformer which progressively performs token representation learning and token agglomeration w… ▽ More This paper presents a new vision Transformer, named Iwin Transformer, which is specifically designed for human-object interaction (HOI) detection, a detailed scene understanding task involving a sequential process of human/object detection and interaction recognition. Iwin Transformer is a hierarchical Transformer which progressively performs token representation learning and token agglomeration within irregular windows. The irregular windows, achieved by augmenting regular grid locations with learned offsets, 1) eliminate redundancy in token representation learning, which leads to efficient human/object detection, and 2) enable the agglomerated tokens to align with humans/objects with different shapes, which facilitates the acquisition of highly-abstracted visual semantics for interaction recognition. The effectiveness and efficiency of Iwin Transformer are verified on the two standard HOI detection benchmark datasets, HICO-DET and V-COCO. Results show our method outperforms existing Transformers-based methods by large margins (3.7 mAP gain on HICO-DET and 2.0 mAP gain on V-COCO) with fewer training epochs ($0.5 \times$). △ Less

Submitted 19 October, 2022; v1 submitted 20 March, 2022; originally announced March 2022.

Comments: Accepted to ECCV 2022

End-to-End Human-Gaze-Target Detection with Transformers

Authors: Danyang Tu, Xiongkuo Min, Huiyu Duan, Guodong Guo, Guangtao Zhai, Wei Shen

Abstract: In this paper, we propose an effective and efficient method for Human-Gaze-Target (HGT) detection, i.e., gaze following. Current approaches decouple the HGT detection task into separate branches of salient object detection and human gaze prediction, employing a two-stage framework where human head locations must first be detected and then be fed into the next gaze target prediction sub-network. In… ▽ More In this paper, we propose an effective and efficient method for Human-Gaze-Target (HGT) detection, i.e., gaze following. Current approaches decouple the HGT detection task into separate branches of salient object detection and human gaze prediction, employing a two-stage framework where human head locations must first be detected and then be fed into the next gaze target prediction sub-network. In contrast, we redefine the HGT detection task as detecting human head locations and their gaze targets, simultaneously. By this way, our method, named Human-Gaze-Target detection TRansformer or HGTTR, streamlines the HGT detection pipeline by eliminating all other additional components. HGTTR reasons about the relations of salient objects and human gaze from the global image context. Moreover, unlike existing two-stage methods that require human head locations as input and can predict only one human's gaze target at a time, HGTTR can directly predict the locations of all people and their gaze targets at one time in an end-to-end manner. The effectiveness and robustness of our proposed method are verified with extensive experiments on the two standard benchmark datasets, GazeFollowing and VideoAttentionTarget. Without bells and whistles, HGTTR outperforms existing state-of-the-art methods by large margins (6.4 mAP gain on GazeFollowing and 10.3 mAP gain on VideoAttentionTarget) with a much simpler architecture. △ Less

Submitted 23 March, 2022; v1 submitted 19 March, 2022; originally announced March 2022.

Comments: Accepted to CVPR 2022

Attribute Group Editing for Reliable Few-shot Image Generation

Authors: Guanqi Ding, Xinzhe Han, Shuhui Wang, Shuzhe Wu, Xin **, Dandan Tu, Qingming Huang

Abstract: Few-shot image generation is a challenging task even using the state-of-the-art Generative Adversarial Networks (GANs). Due to the unstable GAN training process and the limited training data, the generated images are often of low quality and low diversity. In this work, we propose a new editing-based method, i.e., Attribute Group Editing (AGE), for few-shot image generation. The basic assumption i… ▽ More Few-shot image generation is a challenging task even using the state-of-the-art Generative Adversarial Networks (GANs). Due to the unstable GAN training process and the limited training data, the generated images are often of low quality and low diversity. In this work, we propose a new editing-based method, i.e., Attribute Group Editing (AGE), for few-shot image generation. The basic assumption is that any image is a collection of attributes and the editing direction for a specific attribute is shared across all categories. AGE examines the internal representation learned in GANs and identifies semantically meaningful directions. Specifically, the class embedding, i.e., the mean vector of the latent codes from a specific category, is used to represent the category-relevant attributes, and the category-irrelevant attributes are learned globally by Sparse Dictionary Learning on the difference between the sample embedding and the class embedding. Given a GAN well trained on seen categories, diverse images of unseen categories can be synthesized through editing category-irrelevant attributes while kee** category-relevant attributes unchanged. Without re-training the GAN, AGE is capable of not only producing more realistic and diverse images for downstream visual applications with limited data but achieving controllable image editing with interpretable category-irrelevant directions. △ Less

Submitted 16 March, 2022; originally announced March 2022.

Comments: CVPR2022

ClueGraphSum: Let Key Clues Guide the Cross-Lingual Abstractive Summarization

Authors: Shuyu Jiang, Dengbiao Tu, Xingshu Chen, Rui Tang, Wenxian Wang, Haizhou Wang

Abstract: Cross-Lingual Summarization (CLS) is the task to generate a summary in one language for an article in a different language. Previous studies on CLS mainly take pipeline methods or train the end-to-end model using the translated parallel data. However, the quality of generated cross-lingual summaries needs more further efforts to improve, and the model performance has never been evaluated on the ha… ▽ More Cross-Lingual Summarization (CLS) is the task to generate a summary in one language for an article in a different language. Previous studies on CLS mainly take pipeline methods or train the end-to-end model using the translated parallel data. However, the quality of generated cross-lingual summaries needs more further efforts to improve, and the model performance has never been evaluated on the hand-written CLS dataset. Therefore, we first propose a clue-guided cross-lingual abstractive summarization method to improve the quality of cross-lingual summaries, and then construct a novel hand-written CLS dataset for evaluation. Specifically, we extract keywords, named entities, etc. of the input article as key clues for summarization and then design a clue-guided algorithm to transform an article into a graph with less noisy sentences. One Graph encoder is built to learn sentence semantics and article structures and one Clue encoder is built to encode and translate key clues, ensuring the information of important parts are reserved in the generated summary. These two encoders are connected by one decoder to directly learn cross-lingual semantics. Experimental results show that our method has stronger robustness for longer inputs and substantially improves the performance over the strong baseline, achieving an improvement of 8.55 ROUGE-1 (English-to-Chinese summarization) and 2.13 MoverScore (Chinese-to-English summarization) scores over the existing SOTA. △ Less

Submitted 9 March, 2022; v1 submitted 5 March, 2022; originally announced March 2022.

Comments: 12 pages,4 figures

Advancing COVID-19 Diagnosis with Privacy-Preserving Collaboration in Artificial Intelligence

Authors: Xiang Bai, Hanchen Wang, Liya Ma, Yongchao Xu, Jiefeng Gan, Ziwei Fan, Fan Yang, Ke Ma, Jiehua Yang, Song Bai, Chang Shu, Xinyu Zou, Renhao Huang, Changzheng Zhang, Xiaowu Liu, Dandan Tu, Chuou Xu, Wenqing Zhang, Xi Wang, Anguo Chen, Yu Zeng, Dehua Yang, Ming-Wei Wang, Nagaraj Holalkere, Neil J. Halin , et al. (21 additional authors not shown)

Abstract: Artificial intelligence (AI) provides a promising substitution for streamlining COVID-19 diagnoses. However, concerns surrounding security and trustworthiness impede the collection of large-scale representative medical data, posing a considerable challenge for training a well-generalised model in clinical practices. To address this, we launch the Unified CT-COVID AI Diagnostic Initiative (UCADI),… ▽ More Artificial intelligence (AI) provides a promising substitution for streamlining COVID-19 diagnoses. However, concerns surrounding security and trustworthiness impede the collection of large-scale representative medical data, posing a considerable challenge for training a well-generalised model in clinical practices. To address this, we launch the Unified CT-COVID AI Diagnostic Initiative (UCADI), where the AI model can be distributedly trained and independently executed at each host institution under a federated learning framework (FL) without data sharing. Here we show that our FL model outperformed all the local models by a large yield (test sensitivity /specificity in China: 0.973/0.951, in the UK: 0.730/0.942), achieving comparable performance with a panel of professional radiologists. We further evaluated the model on the hold-out (collected from another two hospitals leaving out the FL) and heterogeneous (acquired with contrast materials) data, provided visual explanations for decisions made by the model, and analysed the trade-offs between the model performance and the communication costs in the federated training process. Our study is based on 9,573 chest computed tomography scans (CTs) from 3,336 patients collected from 23 hospitals located in China and the UK. Collectively, our work advanced the prospects of utilising federated learning for privacy-preserving AI in digital health. △ Less

Submitted 17 November, 2021; originally announced November 2021.

Comments: Nature Machine Intelligence

DuCN: Dual-children Network for Medical Diagnosis and Similar Case Recommendation towards COVID-19

Authors: Chengtao Peng, Yunfei Long, Senhua Zhu, Dandan Tu, Bin Li

Abstract: Early detection of the coronavirus disease 2019 (COVID-19) helps to treat patients timely and increase the cure rate, thus further suppressing the spread of the disease. In this study, we propose a novel deep learning based detection and similar case recommendation network to help control the epidemic. Our proposed network contains two stages: the first one is a lung region segmentation step and i… ▽ More Early detection of the coronavirus disease 2019 (COVID-19) helps to treat patients timely and increase the cure rate, thus further suppressing the spread of the disease. In this study, we propose a novel deep learning based detection and similar case recommendation network to help control the epidemic. Our proposed network contains two stages: the first one is a lung region segmentation step and is used to exclude irrelevant factors, and the second is a detection and recommendation stage. Under this framework, in the second stage, we develop a dual-children network (DuCN) based on a pre-trained ResNet-18 to simultaneously realize the disease diagnosis and similar case recommendation. Besides, we employ triplet loss and intrapulmonary distance maps to assist the detection, which helps incorporate tiny differences between two images and is conducive to improving the diagnostic accuracy. For each confirmed COVID-19 case, we give similar cases to provide radiologists with diagnosis and treatment references. We conduct experiments on a large publicly available dataset (CC-CCII) and compare the proposed model with state-of-the-art COVID-19 detection methods. The results show that our proposed model achieves a promising clinical performance. △ Less

Submitted 3 August, 2021; originally announced August 2021.

Low-power/high-gain flexible complementary circuits based on printed organic electrochemical transistors

Authors: Chi-Yuan Yang, Deyu Tu, Tero-Petri Ruoko, Jennifer Y. Gerasimov, Han-Yan Wu, P. C. Harikesh, Renee Kroon, Christian Müller, Magnus Berggren, Simone Fabiano

Abstract: The ability to accurately extract low-amplitude voltage signals is crucial in several fields, ranging from single-use diagnostics and medical technology to robotics and the Internet of Things. The organic electrochemical transistor, which features large transconductance values at low operation voltages, is ideal for monitoring small signals. Its large transconductance translates small gate voltage… ▽ More The ability to accurately extract low-amplitude voltage signals is crucial in several fields, ranging from single-use diagnostics and medical technology to robotics and the Internet of Things. The organic electrochemical transistor, which features large transconductance values at low operation voltages, is ideal for monitoring small signals. Its large transconductance translates small gate voltage variations into significant changes in the drain current. However, a current-to-voltage conversion is further needed to allow proper data acquisition and signal processing. Low power consumption, high amplification, and manufacturability on flexible and low-cost carriers are also crucial and highly anticipated for targeted applications. Here, we report low-power and high-gain flexible circuits based on printed complementary organic electrochemical transistors (OECTs). We leverage the low threshold voltage of both p-type and n-type enhancement-mode OECTs to develop complementary voltage amplifiers that can sense voltages as low as 100 $μ$V, with gains of 30.4 dB and at a power consumption < 2.7 $μ$W (single-stage amplifier). At the optimal operating conditions, the voltage gain normalized to power consumption reaches 169 dB/$μ$W, which is > 50 times larger than state-of-the-art OECT-based amplifiers. In a two-stage configuration, the complementary voltage amplifiers reach a DC voltage gain of 193 V/V, which is the highest among emerging CMOS-like technologies operating at supply voltages below 1 volt. Our findings demonstrate that flexible complementary circuits based on printed OECTs define a power-efficient platform for sensing and amplifying low-amplitude voltage signals in several emerging beyond-silicon applications. △ Less

Submitted 14 June, 2021; originally announced June 2021.

Blind Quality Assessment for in-the-Wild Images via Hierarchical Feature Fusion and Iterative Mixed Database Training

Authors: Wei Sun, Xiongkuo Min, Danyang Tu, Guangtao Zhai, Siwei Ma

Abstract: Image quality assessment (IQA) is very important for both end-users and service providers since a high-quality image can significantly improve the user's quality of experience (QoE) and also benefit lots of computer vision algorithms. Most existing blind image quality assessment (BIQA) models were developed for synthetically distorted images, however, they perform poorly on in-the-wild images, whi… ▽ More Image quality assessment (IQA) is very important for both end-users and service providers since a high-quality image can significantly improve the user's quality of experience (QoE) and also benefit lots of computer vision algorithms. Most existing blind image quality assessment (BIQA) models were developed for synthetically distorted images, however, they perform poorly on in-the-wild images, which are widely existed in various practical applications. In this paper, we propose a novel BIQA model for in-the-wild images by addressing two critical problems in this field: how to learn better quality-aware feature representation, and how to solve the problem of insufficient training samples in terms of their content and distortion diversity. Considering that perceptual visual quality is affected by both low-level visual features (e.g. distortions) and high-level semantic information (e.g. content), we first propose a staircase structure to hierarchically integrate the features from intermediate layers into the final feature representation, which enables the model to make full use of visual information from low-level to high-level. Then an iterative mixed database training (IMDT) strategy is proposed to train the BIQA model on multiple databases simultaneously, so the model can benefit from the increase in both training samples and image content and distortion diversity and can learn a more general feature representation. Experimental results show that the proposed model outperforms other state-of-the-art BIQA models on six in-the-wild IQA databases by a large margin. Moreover, the proposed model shows an excellent performance in the cross-database evaluation experiments, which further demonstrates that the learned feature representation is robust to images with diverse distortions and content. The code is available at https://github.com/sunwei925/StairIQA. △ Less

Submitted 27 April, 2023; v1 submitted 30 May, 2021; originally announced May 2021.

Comments: Accepted by IEEE Journal of Selected Topics in Signal Processing

OralViewer: 3D Demonstration of Dental Surgeries for Patient Education with Oral Cavity Reconstruction from a 2D Panoramic X-ray

Authors: Yuan Liang, Liang Qiu, Tiancheng Lu, Zhujun Fang, Dezhan Tu, Jiawei Yang, Tiandong Zhao, Yiting Shao, Kun Wang, Xiang 'Anthony' Chen, Lei He

Abstract: Patient's understanding on forthcoming dental surgeries is required by patient-centered care and helps reduce fear and anxiety. Due to the gap of expertise between patients and dentists, conventional techniques of patient education are usually not effective for explaining surgical steps. In this paper, we present \textit{OralViewer} -- the first interactive application that enables dentist's demon… ▽ More Patient's understanding on forthcoming dental surgeries is required by patient-centered care and helps reduce fear and anxiety. Due to the gap of expertise between patients and dentists, conventional techniques of patient education are usually not effective for explaining surgical steps. In this paper, we present \textit{OralViewer} -- the first interactive application that enables dentist's demonstration of dental surgeries in 3D to promote patients' understanding. \textit{OralViewer} takes a single 2D panoramic dental X-ray to reconstruct patient-specific 3D teeth structures, which are then assembled with registered gum and jaw bone models for complete oral cavity modeling. During the demonstration, \textit{OralViewer} enables dentists to show surgery steps with virtual dental instruments that can animate effects on a 3D model in real-time. A technical evaluation shows our deep learning based model achieves a mean Intersection over Union (IoU) of 0.771 for 3D teeth reconstruction. A patient study with 12 participants shows \textit{OralViewer} can improve patients' understanding of surgeries. An expert study with 3 board-certified dentists further verifies the clinical validity of our system. △ Less

Submitted 31 December, 2020; originally announced January 2021.

Trust the Model When It Is Confident: Masked Model-based Actor-Critic

Authors: Feiyang Pan, Jia He, Dandan Tu, Qing He

Abstract: It is a popular belief that model-based Reinforcement Learning (RL) is more sample efficient than model-free RL, but in practice, it is not always true due to overweighed model errors. In complex and noisy settings, model-based RL tends to have trouble using the model if it does not know when to trust the model. In this work, we find that better model usage can make a huge difference. We show th… ▽ More It is a popular belief that model-based Reinforcement Learning (RL) is more sample efficient than model-free RL, but in practice, it is not always true due to overweighed model errors. In complex and noisy settings, model-based RL tends to have trouble using the model if it does not know when to trust the model. In this work, we find that better model usage can make a huge difference. We show theoretically that if the use of model-generated data is restricted to state-action pairs where the model error is small, the performance gap between model and real rollouts can be reduced. It motivates us to use model rollouts only when the model is confident about its predictions. We propose Masked Model-based Actor-Critic (M2AC), a novel policy optimization algorithm that maximizes a model-based lower-bound of the true value function. M2AC implements a masking mechanism based on the model's uncertainty to decide whether its prediction should be used or not. Consequently, the new algorithm tends to give robust policy improvements. Experiments on continuous control benchmarks demonstrate that M2AC has strong performance even when using long model rollouts in very noisy environments, and it significantly outperforms previous state-of-the-art methods. △ Less

Submitted 9 October, 2020; originally announced October 2020.

Comments: NeurIPS 2020

Learning Directional Feature Maps for Cardiac MRI Segmentation

Authors: Feng Cheng, Cheng Chen, Yukang Wang, Heshui Shi, Yukun Cao, Dandan Tu, Changzheng Zhang, Yongchao Xu

Abstract: Cardiac MRI segmentation plays a crucial role in clinical diagnosis for evaluating personalized cardiac performance parameters. Due to the indistinct boundaries and heterogeneous intensity distributions in the cardiac MRI, most existing methods still suffer from two aspects of challenges: inter-class indistinction and intra-class inconsistency. To tackle these two problems, we propose a novel meth… ▽ More Cardiac MRI segmentation plays a crucial role in clinical diagnosis for evaluating personalized cardiac performance parameters. Due to the indistinct boundaries and heterogeneous intensity distributions in the cardiac MRI, most existing methods still suffer from two aspects of challenges: inter-class indistinction and intra-class inconsistency. To tackle these two problems, we propose a novel method to exploit the directional feature maps, which can simultaneously strengthen the differences between classes and the similarities within classes. Specifically, we perform cardiac segmentation and learn a direction field pointing away from the nearest cardiac tissue boundary to each pixel via a direction field (DF) module. Based on the learned direction field, we then propose a feature rectification and fusion (FRF) module to improve the original segmentation features, and obtain the final segmentation. The proposed modules are simple yet effective and can be flexibly added to any existing segmentation network without excessively increasing time and space complexity. We evaluate the proposed method on the 2017 MICCAI Automated Cardiac Diagnosis Challenge (ACDC) dataset and a large-scale self-collected dataset, showing good segmentation performance and robust generalization ability of the proposed method. △ Less

Submitted 22 July, 2020; originally announced July 2020.

Comments: Accepted by MICCAI2020

Bringing Stories Alive: Generating Interactive Fiction Worlds

Authors: Prithviraj Ammanabrolu, Wesley Cheung, Dan Tu, William Broniec, Mark O. Riedl

Abstract: World building forms the foundation of any task that requires narrative intelligence. In this work, we focus on procedurally generating interactive fiction worlds---text-based worlds that players "see" and "talk to" using natural language. Generating these worlds requires referencing everyday and thematic commonsense priors in addition to being semantically consistent, interesting, and coherent th… ▽ More World building forms the foundation of any task that requires narrative intelligence. In this work, we focus on procedurally generating interactive fiction worlds---text-based worlds that players "see" and "talk to" using natural language. Generating these worlds requires referencing everyday and thematic commonsense priors in addition to being semantically consistent, interesting, and coherent throughout. Using existing story plots as inspiration, we present a method that first extracts a partial knowledge graph encoding basic information regarding world structure such as locations and objects. This knowledge graph is then automatically completed utilizing thematic knowledge and used to guide a neural language generation model that fleshes out the rest of the world. We perform human participant-based evaluations, testing our neural model's ability to extract and fill-in a knowledge graph and to generate language conditioned on it against rule-based and human-made baselines. Our code is available at https://github.com/rajammanabrolu/WorldGeneration. △ Less

Submitted 27 January, 2020; originally announced January 2020.

Symmetry of extending properties in nonsingular Utumi rings

Authors: Thuat Do, Hai Dinh Hoang, Truong Dinh Tu

Abstract: This paper presents the right-left symmetry of the CS and max-min CS conditions on nonsingular rings, and generalization to nonsingular modules. We prove that a ring is right nonsingular right CS and left Utumi if and only if it is left nonsingular left CS and right Utumi. A nonsingular Utumi ring is right max (resp. right min, right max-min) CS if and only if it is left min (resp. left max, left… ▽ More This paper presents the right-left symmetry of the CS and max-min CS conditions on nonsingular rings, and generalization to nonsingular modules. We prove that a ring is right nonsingular right CS and left Utumi if and only if it is left nonsingular left CS and right Utumi. A nonsingular Utumi ring is right max (resp. right min, right max-min) CS if and only if it is left min (resp. left max, left max-min) CS. In addition, a semiprime nonsingular ring is right max-min CS with finite right uniform dimension if and only if it is left max-min CS with finite left uniform dimension. △ Less

Submitted 6 December, 2019; originally announced December 2019.

MSC Class: 16D70; 16S50

DADI: Dynamic Discovery of Fair Information with Adversarial Reinforcement Learning

Authors: Michiel A. Bakker, Duy Patrick Tu, Humberto Riverón Valdés, Krishna P. Gummadi, Kush R. Varshney, Adrian Weller, Alex Pentland

Abstract: We introduce a framework for dynamic adversarial discovery of information (DADI), motivated by a scenario where information (a feature set) is used by third parties with unknown objectives. We train a reinforcement learning agent to sequentially acquire a subset of the information while balancing accuracy and fairness of predictors downstream. Based on the set of already acquired features, the age… ▽ More We introduce a framework for dynamic adversarial discovery of information (DADI), motivated by a scenario where information (a feature set) is used by third parties with unknown objectives. We train a reinforcement learning agent to sequentially acquire a subset of the information while balancing accuracy and fairness of predictors downstream. Based on the set of already acquired features, the agent decides dynamically to either collect more information from the set of available features or to stop and predict using the information that is currently available. Building on previous work exploring adversarial representation learning, we attain group fairness (demographic parity) by rewarding the agent with the adversary's loss, computed over the final feature set. Importantly, however, the framework provides a more general starting point for fair or private dynamic information discovery. Finally, we demonstrate empirically, using two real-world datasets, that we can trade-off fairness and predictive performance △ Less

Submitted 30 October, 2019; originally announced October 2019.

Comments: Accepted at NeurIPS 2019 HCML Workshop

Blended Conditional Gradients: the unconditioning of conditional gradients

Authors: Gábor Braun, Sebastian Pokutta, Dan Tu, Stephen Wright

Abstract: We present a blended conditional gradient approach for minimizing a smooth convex function over a polytope P, combining the Frank--Wolfe algorithm (also called conditional gradient) with gradient-based steps, different from away steps and pairwise steps, but still achieving linear convergence for strongly convex functions, along with good practical performance. Our approach retains all favorable p… ▽ More We present a blended conditional gradient approach for minimizing a smooth convex function over a polytope P, combining the Frank--Wolfe algorithm (also called conditional gradient) with gradient-based steps, different from away steps and pairwise steps, but still achieving linear convergence for strongly convex functions, along with good practical performance. Our approach retains all favorable properties of conditional gradient algorithms, notably avoidance of projections onto P and maintenance of iterates as sparse convex combinations of a limited number of extreme points of P. The algorithm is lazy, making use of inexpensive inexact solutions of the linear programming subproblem that characterizes the conditional gradient approach. It decreases measures of optimality (primal and dual gaps) rapidly, both in the number of iterations and in wall-clock time, outperforming even the lazy conditional gradient algorithms of [arXiv:1410.8816]. We also present a streamlined version of the algorithm for the probability simplex. △ Less

Submitted 31 May, 2019; v1 submitted 18 May, 2018; originally announced May 2018.

Comments: 33 pages + 12 figures

MSC Class: 68Q32; 90C52

Face Detection Using Improved Faster RCNN

Authors: Changzheng Zhang, Xiang Xu, Dandan Tu

Abstract: Faster RCNN has achieved great success for generic object detection including PASCAL object detection and MS COCO object detection. In this report, we propose a detailed designed Faster RCNN method named FDNet1.0 for face detection. Several techniques were employed including multi-scale training, multi-scale testing, light-designed RCNN, some tricks for inference and a vote-based ensemble method.… ▽ More Faster RCNN has achieved great success for generic object detection including PASCAL object detection and MS COCO object detection. In this report, we propose a detailed designed Faster RCNN method named FDNet1.0 for face detection. Several techniques were employed including multi-scale training, multi-scale testing, light-designed RCNN, some tricks for inference and a vote-based ensemble method. Our method achieves two 1th places and one 2nd place in three tasks over WIDER FACE validation dataset (easy set, medium set, hard set). △ Less

Submitted 6 February, 2018; originally announced February 2018.

The improved Gaussian approximation Calculation of Bogoliubov Mode in One Dimensional Bosonic Gas

Authors: Qiong Li, Daoguang Tu, Ding** Li

Abstract: In this paper, we study the homogeneous one-dimensional bosonic gas interacting via a repulsive contact potential by using the improved Gaussian approximation. We obtain the gapless excitation spectrum of Bogoliubov mode. Our result is in good agreement with the exact numerical calculation based on the Bethe ansatz. We speculate that the improved Gaussian approximation could be a quantitatively go… ▽ More In this paper, we study the homogeneous one-dimensional bosonic gas interacting via a repulsive contact potential by using the improved Gaussian approximation. We obtain the gapless excitation spectrum of Bogoliubov mode. Our result is in good agreement with the exact numerical calculation based on the Bethe ansatz. We speculate that the improved Gaussian approximation could be a quantitatively good approximation for higher dimensional systems. △ Less

Submitted 23 February, 2012; originally announced February 2012.

Comments: 16 pages, 2 figures

Journal ref: Phys. Rev. A 88, 053604, 2013