Showing 1–50 of 58 results for author: Shen, J

Interference Cancellation Based Neural Receiver for Superimposed Pilot in Multi-Layer Transmission

Authors: Han Xiao, Wenqiang Tian, Shi **, Wendong Liu, Jia Shen, Zhihua Shi, Zhi Zhang

Abstract: In this paper, an interference cancellation based neural receiver for superimposed pilot (SIP) in multi-layer transmission is proposed, where the data and pilot are non-orthogonally superimposed in the same time-frequency resource. Specifically, to deal with the intra-layer and inter-layer interference of SIP under multi-layer transmission, the interference cancellation with superimposed symbol ai… ▽ More In this paper, an interference cancellation based neural receiver for superimposed pilot (SIP) in multi-layer transmission is proposed, where the data and pilot are non-orthogonally superimposed in the same time-frequency resource. Specifically, to deal with the intra-layer and inter-layer interference of SIP under multi-layer transmission, the interference cancellation with superimposed symbol aided channel estimation is leveraged in the neural receiver, accompanied by the pre-design of pilot code-division orthogonal mechanism at transmitter. In addition, to address the complexity issue for inter-vendor collaboration and the generalization problem in practical deployments, respectively, this paper also provides a fixed SIP (F-SIP) design based on constant pilot power ratio and scalable mechanisms for different modulation and coding schemes (MCSs) and transmission layers. Simulation results demonstrate the superiority of the proposed schemes on the performance of block error rate and throughput compared with existing counterparts. △ Less

Submitted 27 June, 2024; originally announced June 2024.

Large Language Models for Explainable Decisions in Dynamic Digital Twins

Authors: Nan Zhang, Christian Vergara-Marcillo, Georgios Diamantopoulos, **gran Shen, Nikos Tziritas, Rami Bahsoon, Georgios Theodoropoulos

Abstract: Dynamic data-driven Digital Twins (DDTs) can enable informed decision-making and provide an optimisation platform for the underlying system. By leveraging principles of Dynamic Data-Driven Applications Systems (DDDAS), DDTs can formulate computational modalities for feedback loops, model updates and decision-making, including autonomous ones. However, understanding autonomous decision-making often… ▽ More Dynamic data-driven Digital Twins (DDTs) can enable informed decision-making and provide an optimisation platform for the underlying system. By leveraging principles of Dynamic Data-Driven Applications Systems (DDDAS), DDTs can formulate computational modalities for feedback loops, model updates and decision-making, including autonomous ones. However, understanding autonomous decision-making often requires technical and domain-specific knowledge. This paper explores using large language models (LLMs) to provide an explainability platform for DDTs, generating natural language explanations of the system's decision-making by leveraging domain-specific knowledge bases. A case study from smart agriculture is presented. △ Less

Submitted 23 May, 2024; originally announced May 2024.

Comments: 8 pages, 3 figures, under review

A Novel Context driven Critical Integrative Levels (CIL) Approach: Advancing Human-Centric and Integrative Lighting Asset Management in Public Libraries with Practical Thresholds

Authors: **g Lin, Nina Mylly, Per Olof Hedekvist, **gchun Shen

Abstract: This paper proposes the context driven Critical Integrative Levels (CIL), a novel approach to lighting asset management in public libraries that aligns with the transformative vision of human-centric and integrative lighting. This approach encompasses not only the visual aspects of lighting performance but also prioritizes the physiological and psychological well-being of library users. Incorporat… ▽ More This paper proposes the context driven Critical Integrative Levels (CIL), a novel approach to lighting asset management in public libraries that aligns with the transformative vision of human-centric and integrative lighting. This approach encompasses not only the visual aspects of lighting performance but also prioritizes the physiological and psychological well-being of library users. Incorporating a newly defined metric, Mean Time of Exposure (MTOE), the approach quantifies user-light interaction, enabling tailored lighting strategies that respond to diverse activities and needs in library spaces. Case studies demonstrate how the CIL matrix can be practically applied, offering significant improvements over conventional methods by focusing on optimized user experiences from both visual impacts and non-visual effects. △ Less

Submitted 26 April, 2024; originally announced April 2024.

Realtime Person Identification via Gait Analysis

Authors: Shanmuga Venkatachalam, Harideep Nair, Prabhu Vellaisamy, Yongqi Zhou, Ziad Youssfi, John Paul Shen

Abstract: Each person has a unique gait, i.e., walking style, that can be used as a biometric for personal identification. Recent works have demonstrated effective gait recognition using deep neural networks, however most of these works predominantly focus on classification accuracy rather than model efficiency. In order to perform gait recognition using wearable devices on the edge, it is imperative to dev… ▽ More Each person has a unique gait, i.e., walking style, that can be used as a biometric for personal identification. Recent works have demonstrated effective gait recognition using deep neural networks, however most of these works predominantly focus on classification accuracy rather than model efficiency. In order to perform gait recognition using wearable devices on the edge, it is imperative to develop highly efficient low-power models that can be deployed on to small form-factor devices such as microcontrollers. In this paper, we propose a small CNN model with 4 layers that is very amenable for edge AI deployment and realtime gait recognition. This model was trained on a public gait dataset with 20 classes augmented with data collected by the authors, aggregating to 24 classes in total. Our model achieves 96.7% accuracy and consumes only 5KB RAM with an inferencing time of 70 ms and 125mW power, while running continuous inference on Arduino Nano 33 BLE Sense. We successfully demonstrated realtime identification of the authors with the model running on Arduino, thus underscoring the efficacy and providing a proof of feasiblity for deployment in practical systems in near future. △ Less

Submitted 2 April, 2024; originally announced April 2024.

Model-free Resilient Controller Design based on Incentive Feedback Stackelberg Game and Q-learning

Authors: Jiajun Shen, Fengjun Li, Morteza Hashemi, Huazhen Fang

Abstract: In the swift evolution of Cyber-Physical Systems (CPSs) within intelligent environments, especially in the industrial domain shaped by Industry 4.0, the surge in development brings forth unprecedented security challenges. This paper explores the intricate security issues of Industrial CPSs (ICPSs), with a specific focus on the unique threats presented by intelligent attackers capable of directly c… ▽ More In the swift evolution of Cyber-Physical Systems (CPSs) within intelligent environments, especially in the industrial domain shaped by Industry 4.0, the surge in development brings forth unprecedented security challenges. This paper explores the intricate security issues of Industrial CPSs (ICPSs), with a specific focus on the unique threats presented by intelligent attackers capable of directly compromising the controller, thereby posing a direct risk to physical security. Within the framework of hierarchical control and incentive feedback Stackelberg game, we design a resilient leading controller (leader) that is adaptive to a compromised following controller (follower) such that the compromised follower acts cooperatively with the leader, aligning its strategies with the leader's objective to achieve a team-optimal solution. First, we provide sufficient conditions for the existence of an incentive Stackelberg solution when system dynamics are known. Then, we propose a Q-learning-based Approximate Dynamic Programming (ADP) approach, and corresponding algorithms for the online resolution of the incentive Stackelberg solution without requiring prior knowledge of system dynamics. Last but not least, we prove the convergence of our approach to the optimum. △ Less

Submitted 13 March, 2024; originally announced March 2024.

Comments: 8 pages

Exploring Federated Self-Supervised Learning for General Purpose Audio Understanding

Authors: Yasar Abbas Ur Rehman, Kin Wai Lau, Yuyang Xie, Lan Ma, Jiajun Shen

Abstract: The integration of Federated Learning (FL) and Self-supervised Learning (SSL) offers a unique and synergetic combination to exploit the audio data for general-purpose audio understanding, without compromising user data privacy. However, rare efforts have been made to investigate the SSL models in the FL regime for general-purpose audio understanding, especially when the training data is generated… ▽ More The integration of Federated Learning (FL) and Self-supervised Learning (SSL) offers a unique and synergetic combination to exploit the audio data for general-purpose audio understanding, without compromising user data privacy. However, rare efforts have been made to investigate the SSL models in the FL regime for general-purpose audio understanding, especially when the training data is generated by large-scale heterogeneous audio sources. In this paper, we evaluate the performance of feature-matching and predictive audio-SSL techniques when integrated into large-scale FL settings simulated with non-independently identically distributed (non-iid) data. We propose a novel Federated SSL (F-SSL) framework, dubbed FASSL, that enables learning intermediate feature representations from large-scale decentralized heterogeneous clients, holding unlabelled audio data. Our study has found that audio F-SSL approaches perform on par with the centralized audio-SSL approaches on the audio-retrieval task. Extensive experiments demonstrate the effectiveness and significance of FASSL as it assists in obtaining the optimal global model for state-of-the-art FL aggregation methods. △ Less

Submitted 5 February, 2024; originally announced February 2024.

FDNet: Frequency Domain Denoising Network For Cell Segmentation in Astrocytes Derived From Induced Pluripotent Stem Cells

Authors: Haoran Li, Jiahua Shi, Huaming Chen, Bo Du, Simon Maksour, Gabrielle Phillips, Mirella Dottori, Jun Shen

Abstract: Artificially generated induced pluripotent stem cells (iPSCs) from somatic cells play an important role for disease modeling and drug screening of neurodegenerative diseases. Astrocytes differentiated from iPSCs are important targets to investigate neuronal metabolism. The astrocyte differentiation progress can be monitored through the variations of morphology observed from microscopy images at di… ▽ More Artificially generated induced pluripotent stem cells (iPSCs) from somatic cells play an important role for disease modeling and drug screening of neurodegenerative diseases. Astrocytes differentiated from iPSCs are important targets to investigate neuronal metabolism. The astrocyte differentiation progress can be monitored through the variations of morphology observed from microscopy images at different differentiation stages, then determined by molecular biology techniques upon maturation. However, the astrocytes usually ``perfectly'' blend into the background and some of them are covered by interference information (i.e., dead cells, media sediments, and cell debris), which makes astrocytes difficult to observe. Due to the lack of annotated datasets, the existing state-of-the-art deep learning approaches cannot be used to address this issue. In this paper, we introduce a new task named astrocyte segmentation with a novel dataset, called IAI704, which contains 704 images and their corresponding pixel-level annotation masks. Moreover, a novel frequency domain denoising network, named FDNet, is proposed for astrocyte segmentation. In detail, our FDNet consists of a contextual information fusion module (CIF), an attention block (AB), and a Fourier transform block (FTB). CIF and AB fuse multi-scale feature embeddings to localize the astrocytes. FTB transforms feature embeddings into the frequency domain and conducts a high-pass filter to eliminate interference information. Experimental results demonstrate the superiority of our proposed FDNet over the state-of-the-art substitutes in astrocyte segmentation, shedding insights for iPSC differentiation progress prediction. △ Less

Submitted 4 February, 2024; originally announced February 2024.

Comments: Accepted by The IEEE International Symposium on Biomedical Imaging (ISBI) 2024

Knowledge-driven Meta-learning for CSI Feedback

Authors: Han Xiao, Wenqiang Tian, Wendong Liu, Jiajia Guo, Zhi Zhang, Shi **, Zhihua Shi, Li Guo, Jia Shen

Abstract: Accurate and effective channel state information (CSI) feedback is a key technology for massive multiple-input and multiple-output systems. Recently, deep learning (DL) has been introduced for CSI feedback enhancement through massive collected training data and lengthy training time, which is quite costly and impractical for realistic deployment. In this article, a knowledge-driven meta-learning a… ▽ More Accurate and effective channel state information (CSI) feedback is a key technology for massive multiple-input and multiple-output systems. Recently, deep learning (DL) has been introduced for CSI feedback enhancement through massive collected training data and lengthy training time, which is quite costly and impractical for realistic deployment. In this article, a knowledge-driven meta-learning approach is proposed, where the DL model initialized by the meta model obtained from meta training phase is able to achieve rapid convergence when facing a new scenario during target retraining phase. Specifically, instead of training with massive data collected from various scenarios, the meta task environment is constructed based on the intrinsic knowledge of spatial-frequency characteristics of CSI for meta training. Moreover, the target task dataset is also augmented by exploiting the knowledge of statistical characteristics of wireless channel, so that the DL model can achieve higher performance with small actually collected dataset and short training time. In addition, we provide analyses of rationale for the improvement yielded by the knowledge in both phases. Simulation results demonstrate the superiority of the proposed approach from the perspective of feedback performance and convergence speed. △ Less

Submitted 25 October, 2023; v1 submitted 24 October, 2023; originally announced October 2023.

Comments: arXiv admin note: text overlap with arXiv:2301.13475

Two Degree of Freedom Adaptive Control for Hysteresis Compensation of Pneumatic Continuum Bending Actuator

Authors: Junyi Shen, Tetsuro Miyazaki, Shingo Ohno, Maina Sogabe, Kenji Kawashima

Abstract: Soft robotics, with their inherent flexibility and infinite degrees of freedom (DoF), offer promising advancements in human-machine interfaces. Particularly, pneumatic artificial muscles (PAMs) and pneumatic bending actuators have been fundamental in driving this evolution, capitalizing on their mimetic nature to natural muscle movements. However, with the versatility of these actuators comes the… ▽ More Soft robotics, with their inherent flexibility and infinite degrees of freedom (DoF), offer promising advancements in human-machine interfaces. Particularly, pneumatic artificial muscles (PAMs) and pneumatic bending actuators have been fundamental in driving this evolution, capitalizing on their mimetic nature to natural muscle movements. However, with the versatility of these actuators comes the intricate challenge of hysteresis - a nonlinear phenomenon that hampers precise positioning, especially pronounced in pneumatic actuators due to gas compressibility. In this study, we introduce a novel 2-DoF adaptive control for precise bending tracking using a pneumatic continuum actuator. Notably, our control method integrates adaptability into both the feedback and the feedforward element, enhancing trajectory tracking in the presence of profound nonlinear effects. Comparative analysis with existing approaches underscores the superior tracking accuracy of our proposed strategy. This work discusses a new way of simple yet effective control designs for soft actuators with hysteresis properties. △ Less

Submitted 17 September, 2023; originally announced September 2023.

Comments: Submitted to IEEE Conference on Robotics and Automation (ICRA 2024), Under Review

Effectively Heterogeneous Federated Learning: A Pairing and Split Learning Based Approach

Authors: **glong Shen, Xiucheng Wang, Nan Cheng, Longfei Ma, Conghao Zhou, Yuan Zhang

Abstract: As a promising paradigm federated Learning (FL) is widely used in privacy-preserving machine learning, which allows distributed devices to collaboratively train a model while avoiding data transmission among clients. Despite its immense potential, the FL suffers from bottlenecks in training speed due to client heterogeneity, leading to escalated training latency and straggling server aggregation.… ▽ More As a promising paradigm federated Learning (FL) is widely used in privacy-preserving machine learning, which allows distributed devices to collaboratively train a model while avoiding data transmission among clients. Despite its immense potential, the FL suffers from bottlenecks in training speed due to client heterogeneity, leading to escalated training latency and straggling server aggregation. To deal with this challenge, a novel split federated learning (SFL) framework that pairs clients with different computational resources is proposed, where clients are paired based on computing resources and communication rates among clients, meanwhile the neural network model is split into two parts at the logical level, and each client only computes the part assigned to it by using the SL to achieve forward inference and backward training. Moreover, to effectively deal with the client pairing problem, a heuristic greedy algorithm is proposed by reconstructing the optimization of training latency as a graph edge selection problem. Simulation results show the proposed method can significantly improve the FL training speed and achieve high performance both in independent identical distribution (IID) and Non-IID data distribution. △ Less

Submitted 26 August, 2023; originally announced August 2023.

Trajectory Tracking Control of Dual-PAM Soft Actuator with Hysteresis Compensator

Authors: Junyi Shen, Tetsuro Miyazaki, Shingo Ohno, Maina Sogabe, Kenji Kawashima

Abstract: Soft robotics is a swiftly evolving field. Pneumatic actuators are suitable for driving soft robots because of their superior performance. However, their control is challenging due to the hysteresis characteristics. In response to this challenge, we propose an adaptive control method to compensate for the hysteresis of soft actuators. Employing a novel dual pneumatic artificial muscle (PAM) bendin… ▽ More Soft robotics is a swiftly evolving field. Pneumatic actuators are suitable for driving soft robots because of their superior performance. However, their control is challenging due to the hysteresis characteristics. In response to this challenge, we propose an adaptive control method to compensate for the hysteresis of soft actuators. Employing a novel dual pneumatic artificial muscle (PAM) bending actuator, the innovative control approach abates hysteresis effects by dynamically modulating gains within a traditional PID controller corresponding to the predicted variation of the reference trajectory. Through experimental evaluation, we found that the proposed control method outperforms its conventional counterparts regarding tracking accuracy and response speed. Our work reveals a new direction for advancing model-free control in soft actuators. △ Less

Submitted 18 November, 2023; v1 submitted 23 August, 2023; originally announced August 2023.

Comments: This paper has been published in the IEEE Robotics and Automation Letters ,DOI 10.1109/LRA.2023.3334098, copyright has been transfferd to the IEEE. Final version is available at IEEE Xplore

PSRFlow: Probabilistic Super Resolution with Flow-Based Models for Scientific Data

Authors: **gyi Shen, Han-Wei Shen

Abstract: Although many deep-learning-based super-resolution approaches have been proposed in recent years, because no ground truth is available in the inference stage, few can quantify the errors and uncertainties of the super-resolved results. For scientific visualization applications, however, conveying uncertainties of the results to scientists is crucial to avoid generating misleading or incorrect info… ▽ More Although many deep-learning-based super-resolution approaches have been proposed in recent years, because no ground truth is available in the inference stage, few can quantify the errors and uncertainties of the super-resolved results. For scientific visualization applications, however, conveying uncertainties of the results to scientists is crucial to avoid generating misleading or incorrect information. In this paper, we propose PSRFlow, a novel normalizing flow-based generative model for scientific data super-resolution that incorporates uncertainty quantification into the super-resolution process. PSRFlow learns the conditional distribution of the high-resolution data based on the low-resolution counterpart. By sampling from a Gaussian latent space that captures the missing information in the high-resolution data, one can generate different plausible super-resolution outputs. The efficient sampling in the Gaussian latent space allows our model to perform uncertainty quantification for the super-resolved results. During model training, we augment the training data with samples across various scales to make the model adaptable to data of different scales, achieving flexible super-resolution for a given input. Our results demonstrate superior performance and robust uncertainty quantification compared with existing methods such as interpolation and GAN-based super-resolution networks. △ Less

Submitted 8 August, 2023; originally announced August 2023.

Comments: To be published in Proc. IEEE VIS 2023

Interpretable and Secure Trajectory Optimization for UAV-Assisted Communication

Authors: Yunhao Quan, Nan Cheng, Xiucheng Wang, **glong Shen, Longfei Ma, Zhisheng Yin

Abstract: Unmanned aerial vehicles (UAVs) have gained popularity due to their flexible mobility, on-demand deployment, and the ability to establish high probability line-of-sight wireless communication. As a result, UAVs have been extensively used as aerial base stations (ABSs) to supplement ground-based cellular networks for various applications. However, existing UAV-assisted communication schemes mainly… ▽ More Unmanned aerial vehicles (UAVs) have gained popularity due to their flexible mobility, on-demand deployment, and the ability to establish high probability line-of-sight wireless communication. As a result, UAVs have been extensively used as aerial base stations (ABSs) to supplement ground-based cellular networks for various applications. However, existing UAV-assisted communication schemes mainly focus on trajectory optimization and power allocation, while ignoring the issue of collision avoidance during UAV flight. To address this issue, this paper proposes an interpretable UAV-assisted communication scheme that decomposes reliable UAV services into two sub-problems. The first is the constrained UAV coordinates and power allocation problem, which is solved using the Dueling Double DQN (D3QN) method. The second is the constrained UAV collision avoidance and trajectory optimization problem, which is addressed through the Monte Carlo tree search (MCTS) method. This approach ensures both reliable and efficient operation of UAVs. Moreover, we propose a scalable interpretable artificial intelligence (XAI) framework that enables more transparent and reliable system decisions. The proposed scheme's interpretability generates explainable and trustworthy results, making it easier to comprehend, validate, and control UAV-assisted communication solutions. Through extensive experiments, we demonstrate that our proposed algorithm outperforms existing techniques in terms of performance and generalization. The proposed model improves the reliability, efficiency, and safety of UAV-assisted communication systems, making it a promising solution for future UAV-assisted communication applications △ Less

Submitted 4 July, 2023; originally announced July 2023.

Motion Planning for Aerial Pick-and-Place based on Geometric Feasibility Constraints

Authors: Huazi Cao, Jiahao Shen, Cunjia Liu, Bo Zhu, Shiyu Zhao

Abstract: This paper studies the motion planning problem of the pick-and-place of an aerial manipulator that consists of a quadcopter flying base and a Delta arm. We propose a novel partially decoupled motion planning framework to solve this problem. Compared to the state-of-the-art approaches, the proposed one has two novel features. First, it does not suffer from increased computation in high-dimensional… ▽ More This paper studies the motion planning problem of the pick-and-place of an aerial manipulator that consists of a quadcopter flying base and a Delta arm. We propose a novel partially decoupled motion planning framework to solve this problem. Compared to the state-of-the-art approaches, the proposed one has two novel features. First, it does not suffer from increased computation in high-dimensional configuration spaces. That is because it calculates the trajectories of the quadcopter base and the end-effector separately in the Cartesian space based on proposed geometric feasibility constraints. The geometric feasibility constraints can ensure the resulting trajectories satisfy the aerial manipulator's geometry. Second, collision avoidance for the Delta arm is achieved through an iterative approach based on a pinhole map** method, so that the feasible trajectory can be found in an efficient manner. The proposed approach is verified by three experiments on a real aerial manipulation platform. The experimental results show the effectiveness of the proposed method for the aerial pick-and-place task. △ Less

Submitted 8 June, 2023; originally announced June 2023.

A Modular and High-Resolution Time-Frequency Post-Processing Technique

Authors: **shun Shen, Deyun Wei

Abstract: In this letter, based on the variational model, we propose a novel time-frequency post-processing technique to approximate the ideal time-frequency representation. Our method has the advantage of modularity, enabling "plug and play", independent of the performance of specific time-frequency analysis tool. Therefore, it can be easily generalized to the fractional Fourier domain and the linear canon… ▽ More In this letter, based on the variational model, we propose a novel time-frequency post-processing technique to approximate the ideal time-frequency representation. Our method has the advantage of modularity, enabling "plug and play", independent of the performance of specific time-frequency analysis tool. Therefore, it can be easily generalized to the fractional Fourier domain and the linear canonical domain. Additionally, high-resolution is its merit, which depends on the specific instantaneous frequency estimation method. We reveal the relationship between instantaneous frequency estimation and reassignment method. The effectiveness of the proposed method is verified on both synthetic signals and real world signal. △ Less

Submitted 17 May, 2023; originally announced May 2023.

Data Augmentation for Environmental Sound Classification Using Diffusion Probabilistic Model with Top-k Selection Discriminator

Authors: Yunhao Chen, Yunjie Zhu, Zihui Yan, Jianlu Shen, Zhen Ren, Yifan Huang

Abstract: Despite consistent advancement in powerful deep learning techniques in recent years, large amounts of training data are still necessary for the models to avoid overfitting. Synthetic datasets using generative adversarial networks (GAN) have recently been generated to overcome this problem. Nevertheless, despite advancements, GAN-based methods are usually hard to train or fail to generate high-qual… ▽ More Despite consistent advancement in powerful deep learning techniques in recent years, large amounts of training data are still necessary for the models to avoid overfitting. Synthetic datasets using generative adversarial networks (GAN) have recently been generated to overcome this problem. Nevertheless, despite advancements, GAN-based methods are usually hard to train or fail to generate high-quality data samples. In this paper, we propose an environmental sound classification augmentation technique based on the diffusion probabilistic model with DPM-Solver$++$ for fast sampling. In addition, to ensure the quality of the generated spectrograms, we train a top-k selection discriminator on the dataset. According to the experiment results, the synthesized spectrograms have similar features to the original dataset and can significantly increase the classification accuracy of different state-of-the-art models compared with traditional data augmentation techniques. The public code is available on https://github.com/JNAIC/DPMs-for-Audio-Data-Augmentation. △ Less

Submitted 4 April, 2023; v1 submitted 27 March, 2023; originally announced March 2023.

Resilient Output Containment Control of Heterogeneous Multiagent Systems Against Composite Attacks: A Digital Twin Approach

Authors: Yukang Cui, Lingbo Cao, Michael V. Basin, Jun Shen, Tingwen Huang, Xin Gong

Abstract: This paper studies the distributed resilient output containment control of heterogeneous multiagent systems against composite attacks, including denial-of-services (DoS) attacks, false-data injection (FDI) attacks, camouflage attacks, and actuation attacks. Inspired by digital twins, a twin layer (TL) with higher security and privacy is used to decouple the above problem into two tasks: defense pr… ▽ More This paper studies the distributed resilient output containment control of heterogeneous multiagent systems against composite attacks, including denial-of-services (DoS) attacks, false-data injection (FDI) attacks, camouflage attacks, and actuation attacks. Inspired by digital twins, a twin layer (TL) with higher security and privacy is used to decouple the above problem into two tasks: defense protocols against DoS attacks on TL and defense protocols against actuation attacks on cyber-physical layer (CPL). First, considering modeling errors of leader dynamics, we introduce distributed observers to reconstruct the leader dynamics for each follower on TL under DoS attacks. Second, distributed estimators are used to estimate follower states according to the reconstructed leader dynamics on the TL. Third, according to the reconstructed leader dynamics, we design decentralized solvers that calculate the output regulator equations on CPL. Fourth, decentralized adaptive attack-resilient control schemes that resist unbounded actuation attacks are provided on CPL. Furthermore, we apply the above control protocols to prove that the followers can achieve uniformly ultimately bounded (UUB) convergence, and the upper bound of the UUB convergence is determined explicitly. Finally, two simulation examples are provided to show the effectiveness of the proposed control protocols. △ Less

Submitted 22 March, 2023; originally announced March 2023.

On the Benefits of Leveraging Structural Information in Planning Over the Learned Model

Authors: Jiajun Shen, Kananart Kuwaranancharoen, Raid Ayoub, Pietro Mercati, Shreyas Sundaram

Abstract: Model-based Reinforcement Learning (RL) integrates learning and planning and has received increasing attention in recent years. However, learning the model can incur a significant cost (in terms of sample complexity), due to the need to obtain a sufficient number of samples for each state-action pair. In this paper, we investigate the benefits of leveraging structural information about the system… ▽ More Model-based Reinforcement Learning (RL) integrates learning and planning and has received increasing attention in recent years. However, learning the model can incur a significant cost (in terms of sample complexity), due to the need to obtain a sufficient number of samples for each state-action pair. In this paper, we investigate the benefits of leveraging structural information about the system in terms of reducing sample complexity. Specifically, we consider the setting where the transition probability matrix is a known function of a number of structural parameters, whose values are initially unknown. We then consider the problem of estimating those parameters based on the interactions with the environment. We characterize the difference between the Q estimates and the optimal Q value as a function of the number of samples. Our analysis shows that there can be a significant saving in sample complexity by leveraging structural information about the model. We illustrate the findings by considering several problems including controlling a queuing system with heterogeneous servers, and seeking an optimal path in a stochastic windy gridworld. △ Less

Submitted 15 March, 2023; originally announced March 2023.

Comments: 9 pages, 5 figures

A Knowledge-Driven Meta-Learning Method for CSI Feedback

Authors: Han Xiao, Wenqiang Tian, Wendong Liu, Zhi Zhang, Zhihua Shi, Li Guo, Jia Shen

Abstract: Accurate and effective channel state information (CSI) feedback is a key technology for massive multiple-input and multiple-output (MIMO) systems. Recently, deep learning (DL) has been introduced to enhance CSI feedback in massive MIMO application, where the massive collected training data and lengthy training time are costly and impractical for realistic deployment. In this paper, a knowledge-dri… ▽ More Accurate and effective channel state information (CSI) feedback is a key technology for massive multiple-input and multiple-output (MIMO) systems. Recently, deep learning (DL) has been introduced to enhance CSI feedback in massive MIMO application, where the massive collected training data and lengthy training time are costly and impractical for realistic deployment. In this paper, a knowledge-driven meta-learning solution for CSI feedback is proposed, where the DL model initialized by the meta model obtained from meta training phase is able to achieve rapid convergence when facing a new scenario during the target retraining phase. Specifically, instead of training with massive data collected from various scenarios, the meta task environment is constructed based on the intrinsic knowledge of spatial-frequency characteristics of CSI for meta training. Moreover, the target task dataset is also augmented by exploiting the knowledge of statistical characteristics of channel, so that the DL model initialized by meta training can rapidly fit into a new target scenario with higher performance using only a few actually collected data in the target retraining phase. The method greatly reduces the demand for the number of actual collected data, as well as the cost of training time for realistic deployment. Simulation results demonstrate the superiority of the proposed approach from the perspective of feedback performance and convergence speed. △ Less

Submitted 31 January, 2023; originally announced January 2023.

Machine Fault Classification using Hamiltonian Neural Networks

Authors: Jeremy Shen, Jawad Chowdhury, Sourav Banerjee, Gabriel Terejanu

Abstract: A new approach is introduced to classify faults in rotating machinery based on the total energy signature estimated from sensor measurements. The overall goal is to go beyond using black-box models and incorporate additional physical constraints that govern the behavior of mechanical systems. Observational data is used to train Hamiltonian neural networks that describe the conserved energy of the… ▽ More A new approach is introduced to classify faults in rotating machinery based on the total energy signature estimated from sensor measurements. The overall goal is to go beyond using black-box models and incorporate additional physical constraints that govern the behavior of mechanical systems. Observational data is used to train Hamiltonian neural networks that describe the conserved energy of the system for normal and various abnormal regimes. The estimated total energy function, in the form of the weights of the Hamiltonian neural network, serves as the new feature vector to discriminate between the faults using off-the-shelf classification models. The experimental results are obtained using the MaFaulDa database, where the proposed model yields a promising area under the curve (AUC) of $0.78$ for the binary classification (normal vs abnormal) and $0.84$ for the multi-class problem (normal, and $5$ different abnormal regimes). △ Less

Submitted 4 January, 2023; originally announced January 2023.

Comments: ICPRAM 2023

Effective Audio Classification Network Based on Paired Inverse Pyramid Structure and Dense MLP Block

Authors: Yunhao Chen, Yunjie Zhu, Zihui Yan, Yifan Huang, Zhen Ren, Jianlu Shen, Lifang Chen

Abstract: Recently, massive architectures based on Convolutional Neural Network (CNN) and self-attention mechanisms have become necessary for audio classification. While these techniques are state-of-the-art, these works' effectiveness can only be guaranteed with huge computational costs and parameters, large amounts of data augmentation, transfer from large datasets and some other tricks. By utilizing the… ▽ More Recently, massive architectures based on Convolutional Neural Network (CNN) and self-attention mechanisms have become necessary for audio classification. While these techniques are state-of-the-art, these works' effectiveness can only be guaranteed with huge computational costs and parameters, large amounts of data augmentation, transfer from large datasets and some other tricks. By utilizing the lightweight nature of audio, we propose an efficient network structure called Paired Inverse Pyramid Structure (PIP) and a network called Paired Inverse Pyramid Structure MLP Network (PIPMN). The PIPMN reaches 96\% of Environmental Sound Classification (ESC) accuracy on the UrbanSound8K dataset and 93.2\% of Music Genre Classification (MGC) on the GTAZN dataset, with only 1 million parameters. Both of the results are achieved without data augmentation or model transfer. Public code is available at: https://github.com/JNAIC/PIPMN △ Less

Submitted 30 May, 2023; v1 submitted 5 November, 2022; originally announced November 2022.

Research on Self-adaptive Online Vehicle Velocity Prediction Strategy Considering Traffic Information Fusion

Authors: Ziyan Zhang, Junhao Shen, Dongwei Yao, Feng Wu

Abstract: In order to increase the prediction accuracy of the online vehicle velocity prediction (VVP) strategy, a self-adaptive velocity prediction algorithm fused with traffic information was presented for the multiple scenarios. Initially, traffic scenarios were established inside the co-simulation environment. In addition, the algorithm of a general regressive neural network (GRNN) paired with datasets… ▽ More In order to increase the prediction accuracy of the online vehicle velocity prediction (VVP) strategy, a self-adaptive velocity prediction algorithm fused with traffic information was presented for the multiple scenarios. Initially, traffic scenarios were established inside the co-simulation environment. In addition, the algorithm of a general regressive neural network (GRNN) paired with datasets of the ego-vehicle, the front vehicle, and traffic lights was used in traffic scenarios, which increasingly improved the prediction accuracy. To ameliorate the robustness of the algorithm, then the strategy was optimized by particle swarm optimization (PSO) and k-fold cross-validation to find the optimal parameters of the neural network in real-time, which constructed a self-adaptive online PSO-GRNN VVP strategy with multi-information fusion to adapt with different operating situations. The self-adaptive online PSO-GRNN VVP strategy was then deployed to a variety of simulated scenarios to test its efficacy under various operating situations. Finally, the simulation results reveal that in urban and highway scenarios, the prediction accuracy is separately increased by 27.8% and 54.5% when compared to the traditional GRNN VVP strategy with fixed parameters utilizing only the historical ego-vehicle velocity dataset. △ Less

Submitted 7 October, 2022; originally announced October 2022.

Comments: 9 pages, 7 figures

Improved Fuzzy $H_{\infty}$ Filter Design Method for Nonlinear Systems with Time-Varing Delay

Authors: Qianqian Ma, Li Li, Junhui Shen, Haowei Guan, Guangcheng Ma, Hongwei Xia

Abstract: This paper investigates the fuzzy $H_{\infty}$ filter design issue for nonlinear systems with time-varying delay. In order to obtain less conservative fuzzy $H_{\infty}$ filter design method, a novel integral inequality is employed to replace the conventional Lebniz-Newton formula to analyze the stability conditions of the filtering error system. Besides, the information of the membership function… ▽ More This paper investigates the fuzzy $H_{\infty}$ filter design issue for nonlinear systems with time-varying delay. In order to obtain less conservative fuzzy $H_{\infty}$ filter design method, a novel integral inequality is employed to replace the conventional Lebniz-Newton formula to analyze the stability conditions of the filtering error system. Besides, the information of the membership functions is introduced in the criterion to further relax the derived results. The proposed delay dependent filter design method is presented as LMI-based conditions, and corresponding definite expressions of fuzzy $H_{\infty}$ filter are given as well. Finally, a simulation example is provided to prove the effectiveness and superiority of the designed fuzzy $H_{\infty}$ filter. △ Less

Submitted 11 September, 2022; originally announced September 2022.

Comments: This paper was published in 2017 IEEE SMC. arXiv admin note: text overlap with arXiv:2209.04989. text overlap with arXiv:2209.04989

Training Text-To-Speech Systems From Synthetic Data: A Practical Approach For Accent Transfer Tasks

Authors: Lev Finkelstein, Heiga Zen, Norman Casagrande, Chun-an Chan, Ye Jia, Tom Kenter, Alexey Petelin, Jonathan Shen, Vincent Wan, Yu Zhang, Yonghui Wu, Rob Clark

Abstract: Transfer tasks in text-to-speech (TTS) synthesis - where one or more aspects of the speech of one set of speakers is transferred to another set of speakers that do not feature these aspects originally - remains a challenging task. One of the challenges is that models that have high-quality transfer capabilities can have issues in stability, making them impractical for user-facing critical tasks. T… ▽ More Transfer tasks in text-to-speech (TTS) synthesis - where one or more aspects of the speech of one set of speakers is transferred to another set of speakers that do not feature these aspects originally - remains a challenging task. One of the challenges is that models that have high-quality transfer capabilities can have issues in stability, making them impractical for user-facing critical tasks. This paper demonstrates that transfer can be obtained by training a robust TTS system on data generated by a less robust TTS system designed for a high-quality transfer task; in particular, a CHiVE-BERT monolingual TTS system is trained on the output of a Tacotron model designed for accent transfer. While some quality loss is inevitable with this approach, experimental results show that the models trained on synthetic data this way can produce high quality audio displaying accent transfer, while preserving speaker characteristics such as speaking style. △ Less

Submitted 28 August, 2022; originally announced August 2022.

Comments: To be published in Interspeech 2022

AI Enlightens Wireless Communication: A Transformer Backbone for CSI Feedback

Authors: Han Xiao, Zhiqin Wang, Dexin Li, Wenqiang Tian, Xiaofeng Liu, Wendong Liu, Shi **, Jia Shen, Zhi Zhang, Ning Yang

Abstract: This paper is based on the background of the 2nd Wireless Communication Artificial Intelligence (AI) Competition (WAIC) which is hosted by IMT-2020(5G) Promotion Group 5G+AIWork Group, where the framework of the eigenvector-based channel state information (CSI) feedback problem is firstly provided. Then a basic Transformer backbone for CSI feedback referred to EVCsiNet-T is proposed. Moreover, a s… ▽ More This paper is based on the background of the 2nd Wireless Communication Artificial Intelligence (AI) Competition (WAIC) which is hosted by IMT-2020(5G) Promotion Group 5G+AIWork Group, where the framework of the eigenvector-based channel state information (CSI) feedback problem is firstly provided. Then a basic Transformer backbone for CSI feedback referred to EVCsiNet-T is proposed. Moreover, a series of potential enhancements for deep learning based (DL-based) CSI feedback including i) data augmentation, ii) loss function design, iii) training strategy, and iv) model ensemble are introduced. The experimental results involving the comparison between EVCsiNet-T and traditional codebook methods over different channels are further provided, which show the advanced performance and a promising prospect of Transformer on DL-based CSI feedback problem. △ Less

Submitted 16 June, 2022; originally announced June 2022.

A High-Voltage Characterisation Platform For Emerging Resistive Switching Technologies

Authors: Jiawei Shen, Andrea Mifsud, Lijie Xie, Abdulaziz Alshaya, Christos Papavassiliou

Abstract: Emerging memristor-based array architectures have been effectively employed in non-volatile memories and neuromorphic computing systems due to their density, scalability and capability of storing information. Nonetheless, to demonstrate a practical on-chip memristor-based system, it is essential to have the ability to apply large programming voltage ranges during the characterisation procedures fo… ▽ More Emerging memristor-based array architectures have been effectively employed in non-volatile memories and neuromorphic computing systems due to their density, scalability and capability of storing information. Nonetheless, to demonstrate a practical on-chip memristor-based system, it is essential to have the ability to apply large programming voltage ranges during the characterisation procedures for various memristor technologies. This work presents a 16x16 high voltage memristor characterisation array employing high voltage CMOS circuitry. The proposed system has a maximum programming range of $\pm22V$ to allow on-chip electroforming and I-V sweep. In addition, a Kelvin voltage sensing system is implemented to improve the readout accuracy for low memristance measurements. This work addresses the limitation of conventional CMOS-memristor platforms which can only operate at low voltages, thus limiting the characterisation range and integration options of memristor technologies. △ Less

Submitted 17 May, 2022; originally announced May 2022.

Comments: 5 pages. To be published in ISCAS 2022 and made available on IEEEXplore

A Wide Dynamic Range Read-out System For Resistive Switching Technology

Authors: Lijie Xie, Jiawei Shen, Andrea Mifsud, Chaohan Wang, Abdulaziz Alshaya, Christos Papavassiliou

Abstract: The memristor, because of its controllability over a wide dynamic range of resistance, has emerged as a promising device for data storage and analog computation. A major challenge is the accurate measurement of memristance over a wide dynamic range. In this paper, a novel read-out circuit with feedback adjustment is proposed to measure and digitise input current in the range between 20nA and 2mA.… ▽ More The memristor, because of its controllability over a wide dynamic range of resistance, has emerged as a promising device for data storage and analog computation. A major challenge is the accurate measurement of memristance over a wide dynamic range. In this paper, a novel read-out circuit with feedback adjustment is proposed to measure and digitise input current in the range between 20nA and 2mA. The magnitude of the input currents is estimated by a 5-stage logarithmic current-to-voltage amplifier which scales a linear analog-to-digital converter. This way the least significant bit tracks the absolute input magnitude. This circuit is applicable to reading single memristor conductance, and is also preferable in analog computing where read-out accuracy is particularly critical. The circuits have been realized in Bipolar-CMOS-DMOS (BCD) Gen2 technology. △ Less

Submitted 17 May, 2022; originally announced May 2022.

Comments: 5 pages, To be published in ISCAS 2022 and made available on IEEE Xplore

A CMOS-based Characterisation Platform for Emerging RRAM Technologies

Authors: Andrea Mifsud, Jiawei Shen, Peilong Feng, Lijie Xie, Chaohan Wang, Yihan Pan, Sachin Maheshwari, Shady Agwa, Spyros Stathopoulos, Shiwei Wang, Alexander Serb, Christos Papavassiliou, Themis Prodromakis, Timothy G. Constandinou

Abstract: Mass characterisation of emerging memory devices is an essential step in modelling their behaviour for integration within a standard design flow for existing integrated circuit designers. This work develops a novel characterisation platform for emerging resistive devices with a capacity of up to 1 million devices on-chip. Split into four independent sub-arrays, it contains on-chip column-parallel… ▽ More Mass characterisation of emerging memory devices is an essential step in modelling their behaviour for integration within a standard design flow for existing integrated circuit designers. This work develops a novel characterisation platform for emerging resistive devices with a capacity of up to 1 million devices on-chip. Split into four independent sub-arrays, it contains on-chip column-parallel DACs for fast voltage programming of the DUT. On-chip readout circuits with ADCs are also available for fast read operations covering 5-decades of input current (20nA to 2mA). This allows a device's resistance range to be between 1k$Ω$ and 10M$Ω$ with a minimum voltage range of $\pm$1.5V on the device. △ Less

Submitted 17 May, 2022; originally announced May 2022.

Comments: 5 pages. To be published in ISCAS 2022 and made available on IEEE Xplore

Abandoning the Bayer-Filter to See in the Dark

Authors: Xingbo Dong, Wanyan Xu, Zhihui Miao, Lan Ma, Chao Zhang, Jiewen Yang, Zhe **, Andrew Beng ** Teoh, Jiajun Shen

Abstract: Low-light image enhancement - a pervasive but challenging problem, plays a central role in enhancing the visibility of an image captured in a poor illumination environment. Due to the fact that not all photons can pass the Bayer-Filter on the sensor of the color camera, in this work, we first present a De-Bayer-Filter simulator based on deep neural networks to generate a monochrome raw image from… ▽ More Low-light image enhancement - a pervasive but challenging problem, plays a central role in enhancing the visibility of an image captured in a poor illumination environment. Due to the fact that not all photons can pass the Bayer-Filter on the sensor of the color camera, in this work, we first present a De-Bayer-Filter simulator based on deep neural networks to generate a monochrome raw image from the colored raw image. Next, a fully convolutional network is proposed to achieve the low-light image enhancement by fusing colored raw data with synthesized monochrome raw data. Channel-wise attention is also introduced to the fusion process to establish a complementary interaction between features from colored and monochrome raw images. To train the convolutional networks, we propose a dataset with monochrome and color raw pairs named Mono-Colored Raw paired dataset (MCR) collected by using a monochrome camera without Bayer-Filter and a color camera with Bayer-Filter. The proposed pipeline take advantages of the fusion of the virtual monochrome and the color raw images and our extensive experiments indicate that significant improvement can be achieved by leveraging raw sensor data and data-driven learning. △ Less

Submitted 22 March, 2022; v1 submitted 8 March, 2022; originally announced March 2022.

Deep Learning-Based Sparse Whole-Slide Image Analysis for the Diagnosis of Gastric Intestinal Metaplasia

Authors: Jon Braatz, Pranav Rajpurkar, Stephanie Zhang, Andrew Y. Ng, Jeanne Shen

Abstract: In recent years, deep learning has successfully been applied to automate a wide variety of tasks in diagnostic histopathology. However, fast and reliable localization of small-scale regions-of-interest (ROI) has remained a key challenge, as discriminative morphologic features often occupy only a small fraction of a gigapixel-scale whole-slide image (WSI). In this paper, we propose a sparse WSI ana… ▽ More In recent years, deep learning has successfully been applied to automate a wide variety of tasks in diagnostic histopathology. However, fast and reliable localization of small-scale regions-of-interest (ROI) has remained a key challenge, as discriminative morphologic features often occupy only a small fraction of a gigapixel-scale whole-slide image (WSI). In this paper, we propose a sparse WSI analysis method for the rapid identification of high-power ROI for WSI-level classification. We develop an evaluation framework inspired by the early classification literature, in order to quantify the tradeoff between diagnostic performance and inference time for sparse analytic approaches. We test our method on a common but time-consuming task in pathology - that of diagnosing gastric intestinal metaplasia (GIM) on hematoxylin and eosin (H&E)-stained slides from endoscopic biopsy specimens. GIM is a well-known precursor lesion along the pathway to development of gastric cancer. We performed a thorough evaluation of the performance and inference time of our approach on a test set of GIM-positive and GIM-negative WSI, finding that our method successfully detects GIM in all positive WSI, with a WSI-level classification area under the receiver operating characteristic curve (AUC) of 0.98 and an average precision (AP) of 0.95. Furthermore, we show that our method can attain these metrics in under one minute on a standard CPU. Our results are applicable toward the goal of develo** neural networks that can easily be deployed in clinical settings to support pathologists in quickly localizing and diagnosing small-scale morphologic features in WSI. △ Less

Submitted 4 January, 2022; originally announced January 2022.

Expression might be enough: representing pressure and demand for reinforcement learning based traffic signal control

Authors: Liang Zhang, Qiang Wu, Jun Shen, Linyuan Lü, Bo Du, Jianqing Wu

Abstract: Many studies confirmed that a proper traffic state representation is more important than complex algorithms for the classical traffic signal control (TSC) problem. In this paper, we (1) present a novel, flexible and efficient method, namely advanced max pressure (Advanced-MP), taking both running and queuing vehicles into consideration to decide whether to change current signal phase; (2) inventiv… ▽ More Many studies confirmed that a proper traffic state representation is more important than complex algorithms for the classical traffic signal control (TSC) problem. In this paper, we (1) present a novel, flexible and efficient method, namely advanced max pressure (Advanced-MP), taking both running and queuing vehicles into consideration to decide whether to change current signal phase; (2) inventively design the traffic movement representation with the efficient pressure and effective running vehicles from Advanced-MP, namely advanced traffic state (ATS); and (3) develop a reinforcement learning (RL) based algorithm template, called Advanced-XLight, by combining ATS with the latest RL approaches, and generate two RL algorithms, namely "Advanced-MPLight" and "Advanced-CoLight" from Advanced-XLight. Comprehensive experiments on multiple real-world datasets show that: (1) the Advanced-MP outperforms baseline methods, and it is also efficient and reliable for deployment; and (2) Advanced-MPLight and Advanced-CoLight can achieve the state-of-the-art. △ Less

Submitted 9 August, 2022; v1 submitted 19 December, 2021; originally announced December 2021.

Comments: 10 pages, 5 figures

ACM Class: J.4; J.6

EasyCom: An Augmented Reality Dataset to Support Algorithms for Easy Communication in Noisy Environments

Authors: Jacob Donley, Vladimir Tourbabin, Jung-Suk Lee, Mark Broyles, Hao Jiang, Jie Shen, Maja Pantic, Vamsi Krishna Ithapu, Ravish Mehra

Abstract: Augmented Reality (AR) as a platform has the potential to facilitate the reduction of the cocktail party effect. Future AR headsets could potentially leverage information from an array of sensors spanning many different modalities. Training and testing signal processing and machine learning algorithms on tasks such as beam-forming and speech enhancement require high quality representative data. To… ▽ More Augmented Reality (AR) as a platform has the potential to facilitate the reduction of the cocktail party effect. Future AR headsets could potentially leverage information from an array of sensors spanning many different modalities. Training and testing signal processing and machine learning algorithms on tasks such as beam-forming and speech enhancement require high quality representative data. To the best of the author's knowledge, as of publication there are no available datasets that contain synchronized egocentric multi-channel audio and video with dynamic movement and conversations in a noisy environment. In this work, we describe, evaluate and release a dataset that contains over 5 hours of multi-modal data useful for training and testing algorithms for the application of improving conversations for an AR glasses wearer. We provide speech intelligibility, quality and signal-to-noise ratio improvement results for a baseline method and show improvements across all tested metrics. The dataset we are releasing contains AR glasses egocentric multi-channel microphone array audio, wide field-of-view RGB video, speech source pose, headset microphone audio, annotated voice activity, speech transcriptions, head bounding boxes, target of speech and source identification labels. We have created and are releasing this dataset to facilitate research in multi-modal AR solutions to the cocktail party problem. △ Less

Submitted 18 October, 2021; v1 submitted 8 July, 2021; originally announced July 2021.

Comments: Dataset is available at: https://github.com/facebookresearch/EasyComDataset

AI Enlightens Wireless Communication: Analyses, Solutions and Opportunities on CSI Feedback

Authors: Han Xiao, Zhiqin Wang, Wenqiang Tian, Xiaofeng Liu, Wendong Liu, Shi **, Jia Shen, Zhi Zhang, Ning Yang

Abstract: In this paper, we give a systematic description of the 1st Wireless Communication Artificial Intelligence (AI) Competition (WAIC) which is hosted by IMT-2020(5G) Promotion Group 5G+AI Work Group. Firstly, the framework of full channel state information (F-CSI) feedback problem and its corresponding channel dataset are provided. Then the enhancing schemes for DL-based F-CSI feedback including i) ch… ▽ More In this paper, we give a systematic description of the 1st Wireless Communication Artificial Intelligence (AI) Competition (WAIC) which is hosted by IMT-2020(5G) Promotion Group 5G+AI Work Group. Firstly, the framework of full channel state information (F-CSI) feedback problem and its corresponding channel dataset are provided. Then the enhancing schemes for DL-based F-CSI feedback including i) channel data analysis and preprocessing, ii) neural network design and iii) quantization enhancement are elaborated. The final competition results composed of different enhancing schemes are presented. Based on the valuable experience of 1st WAIC, we also list some challenges and potential study areas for the design of AI-based wireless communication systems. △ Less

Submitted 14 June, 2021; v1 submitted 12 June, 2021; originally announced June 2021.

GCN-MIF: Graph Convolutional Network with Multi-Information Fusion for Low-dose CT Denoising

Authors: Kecheng Chen, Jiayu Sun, Jiang Shen, Jixiang Luo, Xinyu Zhang, Xuelin Pan, Dongsheng Wu, Yue Zhao, Miguel Bento, Yazhou Ren, Xiaorong Pu

Abstract: Being low-level radiation exposure and less harmful to health, low-dose computed tomography (LDCT) has been widely adopted in the early screening of lung cancer and COVID-19. LDCT images inevitably suffer from the degradation problem caused by complex noises. It was reported that deep learning (DL)-based LDCT denoising methods using convolutional neural network (CNN) achieved impressive denoising… ▽ More Being low-level radiation exposure and less harmful to health, low-dose computed tomography (LDCT) has been widely adopted in the early screening of lung cancer and COVID-19. LDCT images inevitably suffer from the degradation problem caused by complex noises. It was reported that deep learning (DL)-based LDCT denoising methods using convolutional neural network (CNN) achieved impressive denoising performance. Although most existing DL-based methods (e.g., encoder-decoder framework) can implicitly utilize non-local and contextual information via downsampling operator and 3D CNN, the explicit multi-information (i.e., local, non-local, and contextual) integration may not be explored enough. To address this issue, we propose a novel graph convolutional network-based LDCT denoising model, namely GCN-MIF, to explicitly perform multi-information fusion for denoising purpose. Concretely, by constructing intra- and inter-slice graph, the graph convolutional network is introduced to leverage the non-local and contextual relationships among pixels. The traditional CNN is adopted for the extraction of local information. Finally, the proposed GCN-MIF model fuses all the extracted local, non-local, and contextual information. Extensive experiments show the effectiveness of our proposed GCN-MIF model by quantitative and visualized results. Furthermore, a double-blind reader study on a public clinical dataset is also performed to validate the usability of denoising results in terms of the structural fidelity, the noise suppression, and the overall score. Models and code are available at https://github.com/tonyckc/GCN-MIF_demo. △ Less

Submitted 16 April, 2022; v1 submitted 15 May, 2021; originally announced May 2021.

Comments: Submitted to TMI with under review

PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS

Authors: Ye Jia, Heiga Zen, Jonathan Shen, Yu Zhang, Yonghui Wu

Abstract: This paper introduces PnG BERT, a new encoder model for neural TTS. This model is augmented from the original BERT model, by taking both phoneme and grapheme representations of text as input, as well as the word-level alignment between them. It can be pre-trained on a large text corpus in a self-supervised manner, and fine-tuned in a TTS task. Experimental results show that a neural TTS model usin… ▽ More This paper introduces PnG BERT, a new encoder model for neural TTS. This model is augmented from the original BERT model, by taking both phoneme and grapheme representations of text as input, as well as the word-level alignment between them. It can be pre-trained on a large text corpus in a self-supervised manner, and fine-tuned in a TTS task. Experimental results show that a neural TTS model using a pre-trained PnG BERT as its encoder yields more natural prosody and more accurate pronunciation than a baseline model using only phoneme input with no pre-training. Subjective side-by-side preference evaluations show that raters have no statistically significant preference between the speech synthesized using a PnG BERT and ground truth recordings from professional speakers. △ Less

Submitted 7 June, 2021; v1 submitted 28 March, 2021; originally announced March 2021.

Comments: Accepted to Interspeech 2021

Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling

Authors: Isaac Elias, Heiga Zen, Jonathan Shen, Yu Zhang, Ye Jia, RJ Skerry-Ryan, Yonghui Wu

Abstract: This paper introduces Parallel Tacotron 2, a non-autoregressive neural text-to-speech model with a fully differentiable duration model which does not require supervised duration signals. The duration model is based on a novel attention mechanism and an iterative reconstruction loss based on Soft Dynamic Time War**, this model can learn token-frame alignments as well as token durations automatica… ▽ More This paper introduces Parallel Tacotron 2, a non-autoregressive neural text-to-speech model with a fully differentiable duration model which does not require supervised duration signals. The duration model is based on a novel attention mechanism and an iterative reconstruction loss based on Soft Dynamic Time War**, this model can learn token-frame alignments as well as token durations automatically. Experimental results show that Parallel Tacotron 2 outperforms baselines in subjective naturalness in several diverse multi speaker evaluations. Its duration control capability is also demonstrated. △ Less

Submitted 29 August, 2021; v1 submitted 26 March, 2021; originally announced March 2021.

Comments: Submitted to INTERSPEECH 2021

End-to-end Uncertainty-based Mitigation of Adversarial Attacks to Automated Lane Centering

Authors: Ruochen Jiao, Hengyi Liang, Takami Sato, Junjie Shen, Qi Alfred Chen, Qi Zhu

Abstract: In the development of advanced driver-assistance systems (ADAS) and autonomous vehicles, machine learning techniques that are based on deep neural networks (DNNs) have been widely used for vehicle perception. These techniques offer significant improvement on average perception accuracy over traditional methods, however, have been shown to be susceptible to adversarial attacks, where small perturba… ▽ More In the development of advanced driver-assistance systems (ADAS) and autonomous vehicles, machine learning techniques that are based on deep neural networks (DNNs) have been widely used for vehicle perception. These techniques offer significant improvement on average perception accuracy over traditional methods, however, have been shown to be susceptible to adversarial attacks, where small perturbations in the input may cause significant errors in the perception results and lead to system failure. Most prior works addressing such adversarial attacks focus only on the sensing and perception modules. In this work, we propose an end-to-end approach that addresses the impact of adversarial attacks throughout perception, planning, and control modules. In particular, we choose a target ADAS application, the automated lane centering system in OpenPilot, quantify the perception uncertainty under adversarial attacks, and design a robust planning and control module accordingly based on the uncertainty analysis. We evaluate our proposed approach using both the public dataset and production-grade autonomous driving simulator. The experiment results demonstrate that our approach can effectively mitigate the impact of adversarial attacks and can achieve 55% to 90% improvement over the original OpenPilot. △ Less

Submitted 27 February, 2021; originally announced March 2021.

Comments: 8 pages for conference

Learning domain-agnostic visual representation for computational pathology using medically-irrelevant style transfer augmentation

Authors: Rikiya Yamashita, ** Long, Snikitha Banda, Jeanne Shen, Daniel L. Rubin

Abstract: Suboptimal generalization of machine learning models on unseen data is a key challenge which hampers the clinical applicability of such models to medical imaging. Although various methods such as domain adaptation and domain generalization have evolved to combat this challenge, learning robust and generalizable representations is core to medical image understanding, and continues to be a problem.… ▽ More Suboptimal generalization of machine learning models on unseen data is a key challenge which hampers the clinical applicability of such models to medical imaging. Although various methods such as domain adaptation and domain generalization have evolved to combat this challenge, learning robust and generalizable representations is core to medical image understanding, and continues to be a problem. Here, we propose STRAP (Style TRansfer Augmentation for histoPathology), a form of data augmentation based on random style transfer from non-medical style source such as artistic paintings, for learning domain-agnostic visual representations in computational pathology. Style transfer replaces the low-level texture content of an image with the uninformative style of randomly selected style source image, while preserving the original high-level semantic content. This improves robustness to domain shift and can be used as a simple yet powerful tool for learning domain-agnostic representations. We demonstrate that STRAP leads to state-of-the-art performance, particularly in the presence of domain shifts, on two particular classification tasks in computational pathology. △ Less

Submitted 3 June, 2021; v1 submitted 2 February, 2021; originally announced February 2021.

Learning to Fuse Asymmetric Feature Maps in Siamese Trackers

Authors: Wencheng Han, ** Dong, Fahad Shahbaz Khan, Ling Shao, Jianbing Shen

Abstract: Recently, Siamese-based trackers have achieved promising performance in visual tracking. Most recent Siamese-based trackers typically employ a depth-wise cross-correlation (DW-XCorr) to obtain multi-channel correlation information from the two feature maps (target and search region). However, DW-XCorr has several limitations within Siamese-based tracking: it can easily be fooled by distractors, ha… ▽ More Recently, Siamese-based trackers have achieved promising performance in visual tracking. Most recent Siamese-based trackers typically employ a depth-wise cross-correlation (DW-XCorr) to obtain multi-channel correlation information from the two feature maps (target and search region). However, DW-XCorr has several limitations within Siamese-based tracking: it can easily be fooled by distractors, has fewer activated channels, and provides weak discrimination of object boundaries. Further, DW-XCorr is a handcrafted parameter-free module and cannot fully benefit from offline learning on large-scale data. We propose a learnable module, called the asymmetric convolution (ACM), which learns to better capture the semantic correlation information in offline training on large-scale data. Different from DW-XCorr and its predecessor(XCorr), which regard a single feature map as the convolution kernel, our ACM decomposes the convolution operation on a concatenated feature map into two mathematically equivalent operations, thereby avoiding the need for the feature maps to be of the same size (width and height)during concatenation. Our ACM can incorporate useful prior information, such as bounding-box size, with standard visual features. Furthermore, ACM can easily be integrated into existing Siamese trackers based on DW-XCorror XCorr. To demonstrate its generalization ability, we integrate ACM into three representative trackers: SiamFC, SiamRPN++, and SiamBAN. Our experiments reveal the benefits of the proposed ACM, which outperforms existing methods on six tracking benchmarks. On the LaSOT test set, our ACM-based tracker obtains a significant improvement of 5.8% in terms of success (AUC), over the baseline. △ Less

Submitted 30 March, 2021; v1 submitted 4 December, 2020; originally announced December 2020.

Comments: Accepted by CVPR2021

Parallel Tacotron: Non-Autoregressive and Controllable TTS

Authors: Isaac Elias, Heiga Zen, Jonathan Shen, Yu Zhang, Ye Jia, Ron Weiss, Yonghui Wu

Abstract: Although neural end-to-end text-to-speech models can synthesize highly natural speech, there is still room for improvements to its efficiency and naturalness. This paper proposes a non-autoregressive neural text-to-speech model augmented with a variational autoencoder-based residual encoder. This model, called \emph{Parallel Tacotron}, is highly parallelizable during both training and inference, a… ▽ More Although neural end-to-end text-to-speech models can synthesize highly natural speech, there is still room for improvements to its efficiency and naturalness. This paper proposes a non-autoregressive neural text-to-speech model augmented with a variational autoencoder-based residual encoder. This model, called \emph{Parallel Tacotron}, is highly parallelizable during both training and inference, allowing efficient synthesis on modern parallel hardware. The use of the variational autoencoder relaxes the one-to-many map** nature of the text-to-speech problem and improves naturalness. To further improve the naturalness, we use lightweight convolutions, which can efficiently capture local contexts, and introduce an iterative spectrogram loss inspired by iterative refinement. Experimental results show that Parallel Tacotron matches a strong autoregressive baseline in subjective evaluations with significantly decreased inference time. △ Less

Submitted 22 October, 2020; originally announced October 2020.

ASDN: A Deep Convolutional Network for Arbitrary Scale Image Super-Resolution

Authors: Jialiang Shen, Yucheng Wang, Jian Zhang

Abstract: Deep convolutional neural networks have significantly improved the peak signal-to-noise ratio of SuperResolution (SR). However, image viewer applications commonly allow users to zoom the images to arbitrary magnification scales, thus far imposing a large number of required training scales at a tremendous computational cost. To obtain a more computationally efficient model for arbitrary scale SR, t… ▽ More Deep convolutional neural networks have significantly improved the peak signal-to-noise ratio of SuperResolution (SR). However, image viewer applications commonly allow users to zoom the images to arbitrary magnification scales, thus far imposing a large number of required training scales at a tremendous computational cost. To obtain a more computationally efficient model for arbitrary scale SR, this paper employs a Laplacian pyramid method to reconstruct any-scale high-resolution (HR) images using the high-frequency image details in a Laplacian Frequency Representation. For SR of small-scales (between 1 and 2), images are constructed by interpolation from a sparse set of precalculated Laplacian pyramid levels. SR of larger scales is computed by recursion from small scales, which significantly reduces the computational cost. For a full comparison, fixed- and any-scale experiments are conducted using various benchmarks. At fixed scales, ASDN outperforms predefined upsampling methods (e.g., SRCNN, VDSR, DRRN) by about 1 dB in PSNR. At any-scale, ASDN generally exceeds Meta-SR on many scales. △ Less

Submitted 5 October, 2020; originally announced October 2020.

Enabling Remote Whole-Body Control with 5G Edge Computing

Authors: Huaijiang Zhu, Manali Sharma, Kai Pfeiffer, Marco Mezzavilla, Jia Shen, Sundeep Rangan, Ludovic Righetti

Abstract: Real-world applications require light-weight, energy-efficient, fully autonomous robots. Yet, increasing autonomy is oftentimes synonymous with escalating computational requirements. It might thus be desirable to offload intensive computation--not only sensing and planning, but also low-level whole-body control--to remote servers in order to reduce on-board computational needs. Fifth Generation (5… ▽ More Real-world applications require light-weight, energy-efficient, fully autonomous robots. Yet, increasing autonomy is oftentimes synonymous with escalating computational requirements. It might thus be desirable to offload intensive computation--not only sensing and planning, but also low-level whole-body control--to remote servers in order to reduce on-board computational needs. Fifth Generation (5G) wireless cellular technology, with its low latency and high bandwidth capabilities, has the potential to unlock cloud-based high performance control of complex robots. However, state-of-the-art control algorithms for legged robots can only tolerate very low control delays, which even ultra-low latency 5G edge computing can sometimes fail to achieve. In this work, we investigate the problem of cloud-based whole-body control of legged robots over a 5G link. We propose a novel approach that consists of a standard optimization-based controller on the network edge and a local linear, approximately optimal controller that significantly reduces on-board computational needs while increasing robustness to delay and possible loss of communication. Simulation experiments on humanoid balancing and walking tasks that includes a realistic 5G communication model demonstrate significant improvement of the reliability of robot locomotion under jitter and delays likely to experienced in 5G wireless links. △ Less

Submitted 18 August, 2020; originally announced August 2020.

PraNet: Parallel Reverse Attention Network for Polyp Segmentation

Authors: Deng-** Fan, Ge-Peng Ji, Tao Zhou, Geng Chen, Huazhu Fu, Jianbing Shen, Ling Shao

Abstract: Colonoscopy is an effective technique for detecting colorectal polyps, which are highly related to colorectal cancer. In clinical practice, segmenting polyps from colonoscopy images is of great importance since it provides valuable information for diagnosis and surgery. However, accurate polyp segmentation is a challenging task, for two major reasons: (i) the same type of polyps has a diversity of… ▽ More Colonoscopy is an effective technique for detecting colorectal polyps, which are highly related to colorectal cancer. In clinical practice, segmenting polyps from colonoscopy images is of great importance since it provides valuable information for diagnosis and surgery. However, accurate polyp segmentation is a challenging task, for two major reasons: (i) the same type of polyps has a diversity of size, color and texture; and (ii) the boundary between a polyp and its surrounding mucosa is not sharp. To address these challenges, we propose a parallel reverse attention network (PraNet) for accurate polyp segmentation in colonoscopy images. Specifically, we first aggregate the features in high-level layers using a parallel partial decoder (PPD). Based on the combined feature, we then generate a global map as the initial guidance area for the following components. In addition, we mine the boundary cues using a reverse attention (RA) module, which is able to establish the relationship between areas and boundary cues. Thanks to the recurrent cooperation mechanism between areas and boundaries, our PraNet is capable of calibrating any misaligned predictions, improving the segmentation accuracy. Quantitative and qualitative evaluations on five challenging datasets across six metrics show that our PraNet improves the segmentation accuracy significantly, and presents a number of advantages in terms of generalizability, and real-time segmentation efficiency. △ Less

Submitted 3 July, 2020; v1 submitted 13 June, 2020; originally announced June 2020.

Comments: Accepted to MICCAI 2020

M2Net: Multi-modal Multi-channel Network for Overall Survival Time Prediction of Brain Tumor Patients

Authors: Tao Zhou, Huazhu Fu, Yu Zhang, Changqing Zhang, Xiankai Lu, Jianbing Shen, Ling Shao

Abstract: Early and accurate prediction of overall survival (OS) time can help to obtain better treatment planning for brain tumor patients. Although many OS time prediction methods have been developed and obtain promising results, there are still several issues. First, conventional prediction methods rely on radiomic features at the local lesion area of a magnetic resonance (MR) volume, which may not repre… ▽ More Early and accurate prediction of overall survival (OS) time can help to obtain better treatment planning for brain tumor patients. Although many OS time prediction methods have been developed and obtain promising results, there are still several issues. First, conventional prediction methods rely on radiomic features at the local lesion area of a magnetic resonance (MR) volume, which may not represent the full image or model complex tumor patterns. Second, different types of scanners (i.e., multi-modal data) are sensitive to different brain regions, which makes it challenging to effectively exploit the complementary information across multiple modalities and also preserve the modality-specific properties. Third, existing methods focus on prediction models, ignoring complex data-to-label relationships. To address the above issues, we propose an end-to-end OS time prediction model; namely, Multi-modal Multi-channel Network (M2Net). Specifically, we first project the 3D MR volume onto 2D images in different directions, which reduces computational costs, while preserving important information and enabling pre-trained models to be transferred from other tasks. Then, we use a modality-specific network to extract implicit and high-level features from different MR scans. A multi-modal shared network is built to fuse these features using a bilinear pooling model, exploiting their correlations to provide complementary information. Finally, we integrate the outputs from each modality-specific network and the multi-modal shared network to generate the final prediction result. Experimental results demonstrate the superiority of our M2Net model over other methods. △ Less

Submitted 14 July, 2020; v1 submitted 1 June, 2020; originally announced June 2020.

Comments: Accepted by MICCAI'20

Modeling and Enhancing Low-quality Retinal Fundus Images

Authors: Ziyi Shen, Huazhu Fu, Jianbing Shen, Ling Shao

Abstract: Retinal fundus images are widely used for the clinical screening and diagnosis of eye diseases. However, fundus images captured by operators with various levels of experience have a large variation in quality. Low-quality fundus images increase uncertainty in clinical observation and lead to the risk of misdiagnosis. However, due to the special optical beam of fundus imaging and structure of the r… ▽ More Retinal fundus images are widely used for the clinical screening and diagnosis of eye diseases. However, fundus images captured by operators with various levels of experience have a large variation in quality. Low-quality fundus images increase uncertainty in clinical observation and lead to the risk of misdiagnosis. However, due to the special optical beam of fundus imaging and structure of the retina, natural image enhancement methods cannot be utilized directly to address this. In this paper, we first analyze the ophthalmoscope imaging system and simulate a reliable degradation of major inferior-quality factors, including uneven illumination, image blurring, and artifacts. Then, based on the degradation model, a clinically oriented fundus enhancement network (cofe-Net) is proposed to suppress global degradation factors, while simultaneously preserving anatomical retinal structures and pathological characteristics for clinical observation and analysis. Experiments on both synthetic and real images demonstrate that our algorithm effectively corrects low-quality fundus images without losing retinal details. Moreover, we also show that the fundus correction method can benefit medical image analysis applications, e.g., retinal vessel segmentation and optic disc/cup detection. △ Less

Submitted 9 December, 2020; v1 submitted 12 May, 2020; originally announced May 2020.

Inf-Net: Automatic COVID-19 Lung Infection Segmentation from CT Images

Authors: Deng-** Fan, Tao Zhou, Ge-Peng Ji, Yi Zhou, Geng Chen, Huazhu Fu, Jianbing Shen, Ling Shao

Abstract: Coronavirus Disease 2019 (COVID-19) spread globally in early 2020, causing the world to face an existential health crisis. Automated detection of lung infections from computed tomography (CT) images offers a great potential to augment the traditional healthcare strategy for tackling COVID-19. However, segmenting infected regions from CT slices faces several challenges, including high variation in… ▽ More Coronavirus Disease 2019 (COVID-19) spread globally in early 2020, causing the world to face an existential health crisis. Automated detection of lung infections from computed tomography (CT) images offers a great potential to augment the traditional healthcare strategy for tackling COVID-19. However, segmenting infected regions from CT slices faces several challenges, including high variation in infection characteristics, and low intensity contrast between infections and normal tissues. Further, collecting a large amount of data is impractical within a short time period, inhibiting the training of a deep model. To address these challenges, a novel COVID-19 Lung Infection Segmentation Deep Network (Inf-Net) is proposed to automatically identify infected regions from chest CT slices. In our Inf-Net, a parallel partial decoder is used to aggregate the high-level features and generate a global map. Then, the implicit reverse attention and explicit edge-attention are utilized to model the boundaries and enhance the representations. Moreover, to alleviate the shortage of labeled data, we present a semi-supervised segmentation framework based on a randomly selected propagation strategy, which only requires a few labeled images and leverages primarily unlabeled data. Our semi-supervised framework can improve the learning ability and achieve a higher performance. Extensive experiments on our COVID-SemiSeg and real CT volumes demonstrate that the proposed Inf-Net outperforms most cutting-edge segmentation models and advances the state-of-the-art performance. △ Less

Submitted 21 May, 2020; v1 submitted 22 April, 2020; originally announced April 2020.

Comments: To appear in IEEE TMI. The code is released in: https://github.com/Deng**Fan/Inf-Net

COVID-19 Chest CT Image Segmentation -- A Deep Convolutional Neural Network Solution

Authors: Qingsen Yan, Bo Wang, Dong Gong, Chuan Luo, Wei Zhao, Jianhu Shen, Qinfeng Shi, Shuo **, Liang Zhang, Zheng You

Abstract: A novel coronavirus disease 2019 (COVID-19) was detected and has spread rapidly across various countries around the world since the end of the year 2019, Computed Tomography (CT) images have been used as a crucial alternative to the time-consuming RT-PCR test. However, pure manual segmentation of CT images faces a serious challenge with the increase of suspected cases, resulting in urgent requirem… ▽ More A novel coronavirus disease 2019 (COVID-19) was detected and has spread rapidly across various countries around the world since the end of the year 2019, Computed Tomography (CT) images have been used as a crucial alternative to the time-consuming RT-PCR test. However, pure manual segmentation of CT images faces a serious challenge with the increase of suspected cases, resulting in urgent requirements for accurate and automatic segmentation of COVID-19 infections. Unfortunately, since the imaging characteristics of the COVID-19 infection are diverse and similar to the backgrounds, existing medical image segmentation methods cannot achieve satisfactory performance. In this work, we try to establish a new deep convolutional neural network tailored for segmenting the chest CT images with COVID-19 infections. We firstly maintain a large and new chest CT image dataset consisting of 165,667 annotated chest CT images from 861 patients with confirmed COVID-19. Inspired by the observation that the boundary of the infected lung can be enhanced by adjusting the global intensity, in the proposed deep CNN, we introduce a feature variation block which adaptively adjusts the global properties of the features for segmenting COVID-19 infection. The proposed FV block can enhance the capability of feature representation effectively and adaptively for diverse cases. We fuse features at different scales by proposing Progressive Atrous Spatial Pyramid Pooling to handle the sophisticated infection areas with diverse appearance and shapes. We conducted experiments on the data collected in China and Germany and show that the proposed deep CNN can produce impressive performance effectively. △ Less

Submitted 25 April, 2020; v1 submitted 23 April, 2020; originally announced April 2020.

Cascaded Human-Object Interaction Recognition

Authors: Tianfei Zhou, Wenguan Wang, Siyuan Qi, Haibin Ling, Jianbing Shen

Abstract: Rapid progress has been witnessed for human-object interaction (HOI) recognition, but most existing models are confined to single-stage reasoning pipelines. Considering the intrinsic complexity of the task, we introduce a cascade architecture for a multi-stage, coarse-to-fine HOI understanding. At each stage, an instance localization network progressively refines HOI proposals and feeds them into… ▽ More Rapid progress has been witnessed for human-object interaction (HOI) recognition, but most existing models are confined to single-stage reasoning pipelines. Considering the intrinsic complexity of the task, we introduce a cascade architecture for a multi-stage, coarse-to-fine HOI understanding. At each stage, an instance localization network progressively refines HOI proposals and feeds them into an interaction recognition network. Each of the two networks is also connected to its predecessor at the previous stage, enabling cross-stage information propagation. The interaction recognition network has two crucial parts: a relation ranking module for high-quality HOI proposal selection and a triple-stream classifier for relation prediction. With our carefully-designed human-centric relation features, these two modules work collaboratively towards effective interaction understanding. Further beyond relation detection on a bounding-box level, we make our framework flexible to perform fine-grained pixel-wise relation segmentation; this provides a new glimpse into better relation modeling. Our approach reached the $1^{st}$ place in the ICCV2019 Person in Context Challenge, on both relation detection and segmentation tasks. It also shows promising results on V-COCO. △ Less

Submitted 11 March, 2020; v1 submitted 9 March, 2020; originally announced March 2020.

Comments: Accepted to CVPR 2020. Winner of the ICCV-2019 PIC Challenge on both HOIW and PIC tracks. Code: https://github.com/tfzhou/C-HOI

Depression Detection using Resting State Three-channel EEG Signal

Authors: Qiuxia Shi, Ang Liu, Rongyan Chen, Jian Shen, Qinglin Zhao, Bin Hu

Abstract: In universal environment, a patient-friendly inexpensive method is needed to realize the early diagnosis of depression, which is believed to be an effective way to reduce the mortality of depression. The purpose of this study is only to collect EEG signal from three electrodes Fp1, Fpz and Fp2, then the linear and nonlinear features of EEG used to classify depression patients and healthy controls.… ▽ More In universal environment, a patient-friendly inexpensive method is needed to realize the early diagnosis of depression, which is believed to be an effective way to reduce the mortality of depression. The purpose of this study is only to collect EEG signal from three electrodes Fp1, Fpz and Fp2, then the linear and nonlinear features of EEG used to classify depression patients and healthy controls. The EEG recordings were carried out on a group of 18 medication-free depressive patients and 25 gender and age matched controls. In this paper, the selected features include three linear (maximum, mean and center values of the power) and three nonlinear features (correlation dimension, Renyi entropy and C0 complexity). The accuracy and effectiveness of classification model between depressive and control subjects were calculated using leave-one-out cross-validation. The experimental results indicate that selected three channel EEG and features can distinguish the subjects between depression and normal beings, the classification accuracy is 72.25%. It is hoped that the performed results can provide more choices for the early diagnosis of depression in a universal environment. △ Less

Submitted 26 February, 2020; v1 submitted 21 February, 2020; originally announced February 2020.

Comments: 12 pages, 2figures, 1 table

Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesis

Authors: Tao Zhou, Huazhu Fu, Geng Chen, Jianbing Shen, Ling Shao

Abstract: Magnetic resonance imaging (MRI) is a widely used neuroimaging technique that can provide images of different contrasts (i.e., modalities). Fusing this multi-modal data has proven particularly effective for boosting model performance in many tasks. However, due to poor data quality and frequent patient dropout, collecting all modalities for every patient remains a challenge. Medical image synthesi… ▽ More Magnetic resonance imaging (MRI) is a widely used neuroimaging technique that can provide images of different contrasts (i.e., modalities). Fusing this multi-modal data has proven particularly effective for boosting model performance in many tasks. However, due to poor data quality and frequent patient dropout, collecting all modalities for every patient remains a challenge. Medical image synthesis has been proposed as an effective solution to this, where any missing modalities are synthesized from the existing ones. In this paper, we propose a novel Hybrid-fusion Network (Hi-Net) for multi-modal MR image synthesis, which learns a map** from multi-modal source images (i.e., existing modalities) to target images (i.e., missing modalities). In our Hi-Net, a modality-specific network is utilized to learn representations for each individual modality, and a fusion network is employed to learn the common latent representation of multi-modal data. Then, a multi-modal synthesis network is designed to densely combine the latent representation with hierarchical features from each modality, acting as a generator to synthesize the target images. Moreover, a layer-wise multi-modal fusion strategy is presented to effectively exploit the correlations among multiple modalities, in which a Mixed Fusion Block (MFB) is proposed to adaptively weight different fusion strategies (i.e., element-wise summation, product, and maximization). Extensive experiments demonstrate that the proposed model outperforms other state-of-the-art medical image synthesis methods. △ Less

Submitted 11 February, 2020; originally announced February 2020.

Comments: has been accepted by IEEE TMI