-
Applying Fine-Tuned LLMs for Reducing Data Needs in Load Profile Analysis
Authors:
Yi Hu,
Hyeon** Kim,
Kai Ye,
Ning Lu
Abstract:
This paper presents a novel method for utilizing fine-tuned Large Language Models (LLMs) to minimize data requirements in load profile analysis, demonstrated through the restoration of missing data in power system load profiles. A two-stage fine-tuning strategy is proposed to adapt a pre-trained LLMs, i.e., GPT-3.5, for missing data restoration tasks. Through empirical evaluation, we demonstrate t…
▽ More
This paper presents a novel method for utilizing fine-tuned Large Language Models (LLMs) to minimize data requirements in load profile analysis, demonstrated through the restoration of missing data in power system load profiles. A two-stage fine-tuning strategy is proposed to adapt a pre-trained LLMs, i.e., GPT-3.5, for missing data restoration tasks. Through empirical evaluation, we demonstrate the effectiveness of the fine-tuned model in accurately restoring missing data, achieving comparable performance to state-of-the-art specifically designed models such as BERT-PIN. Key findings include the importance of prompt engineering and the optimal utilization of fine-tuning samples, highlighting the efficiency of few-shot learning in transferring knowledge from general user cases to specific target users. Furthermore, the proposed approach demonstrates notable cost-effectiveness and time efficiency compared to training models from scratch, making it a practical solution for scenarios with limited data availability and computing resources. This research has significant potential for application to other power system load profile analysis tasks. Consequently, it advances the use of LLMs in power system analytics, offering promising implications for enhancing the resilience and efficiency of power distribution systems.
△ Less
Submitted 2 June, 2024;
originally announced June 2024.
-
NeurTV: Total Variation on the Neural Domain
Authors:
Yisi Luo,
Xile Zhao,
Kai Ye,
Deyu Meng
Abstract:
Recently, we have witnessed the success of total variation (TV) for many imaging applications. However, traditional TV is defined on the original pixel domain, which limits its potential. In this work, we suggest a new TV regularization defined on the neural domain. Concretely, the discrete data is continuously and implicitly represented by a deep neural network (DNN), and we use the derivatives o…
▽ More
Recently, we have witnessed the success of total variation (TV) for many imaging applications. However, traditional TV is defined on the original pixel domain, which limits its potential. In this work, we suggest a new TV regularization defined on the neural domain. Concretely, the discrete data is continuously and implicitly represented by a deep neural network (DNN), and we use the derivatives of DNN outputs w.r.t. input coordinates to capture local correlations of data. As compared with classical TV on the original domain, the proposed TV on the neural domain (termed NeurTV) enjoys two advantages. First, NeurTV is not limited to meshgrid but is suitable for both meshgrid and non-meshgrid data. Second, NeurTV can more exactly capture local correlations across data for any direction and any order of derivatives attributed to the implicit and continuous nature of neural domain. We theoretically reinterpret NeurTV under the variational approximation framework, which allows us to build the connection between classical TV and NeurTV and inspires us to develop variants (e.g., NeurTV with arbitrary resolution and space-variant NeurTV). Extensive numerical experiments with meshgrid data (e.g., color and hyperspectral images) and non-meshgrid data (e.g., point clouds and spatial transcriptomics) showcase the effectiveness of the proposed methods.
△ Less
Submitted 27 May, 2024;
originally announced May 2024.
-
Underdetermined DOA Estimation of Off-Grid Sources Based on the Generalized Double Pareto Prior
Authors:
Yongfeng Huang,
Zhendong Chen,
Kun Ye,
Lang Zhou,
Haixin Sun
Abstract:
In this letter, we investigate a new generalized double Pareto based on off-grid sparse Bayesian learning (GDPOGSBL) approach to improve the performance of direction of arrival (DOA) estimation in underdetermined scenarios. The method aims to enhance the sparsity of source signal by utilizing the generalized double Pareto (GDP) prior. Firstly, we employ a first-order linear Taylor expansion to mod…
▽ More
In this letter, we investigate a new generalized double Pareto based on off-grid sparse Bayesian learning (GDPOGSBL) approach to improve the performance of direction of arrival (DOA) estimation in underdetermined scenarios. The method aims to enhance the sparsity of source signal by utilizing the generalized double Pareto (GDP) prior. Firstly, we employ a first-order linear Taylor expansion to model the real array manifold matrix, and Bayesian inference is utilized to calculate the off-grid error, which mitigates the grid dictionary mismatch problem in underdetermined scenarios. Secondly, an innovative grid refinement method is introduced, treating grid points as iterative parameters to minimize the modeling error between the source and grid points. The numerical simulation results verify the superiority of the proposed strategy, especially when dealing with a coarse grid and few snapshots.
△ Less
Submitted 17 May, 2024; v1 submitted 18 April, 2024;
originally announced May 2024.
-
Towards Evaluating the Robustness of Automatic Speech Recognition Systems via Audio Style Transfer
Authors:
Weifei **,
Yuxin Cao,
Junjie Su,
Qi Shen,
Kai Ye,
Derui Wang,
Jie Hao,
Ziyao Liu
Abstract:
In light of the widespread application of Automatic Speech Recognition (ASR) systems, their security concerns have received much more attention than ever before, primarily due to the susceptibility of Deep Neural Networks. Previous studies have illustrated that surreptitiously crafting adversarial perturbations enables the manipulation of speech recognition systems, resulting in the production of…
▽ More
In light of the widespread application of Automatic Speech Recognition (ASR) systems, their security concerns have received much more attention than ever before, primarily due to the susceptibility of Deep Neural Networks. Previous studies have illustrated that surreptitiously crafting adversarial perturbations enables the manipulation of speech recognition systems, resulting in the production of malicious commands. These attack methods mostly require adding noise perturbations under $\ell_p$ norm constraints, inevitably leaving behind artifacts of manual modifications. Recent research has alleviated this limitation by manipulating style vectors to synthesize adversarial examples based on Text-to-Speech (TTS) synthesis audio. However, style modifications based on optimization objectives significantly reduce the controllability and editability of audio styles. In this paper, we propose an attack on ASR systems based on user-customized style transfer. We first test the effect of Style Transfer Attack (STA) which combines style transfer and adversarial attack in sequential order. And then, as an improvement, we propose an iterative Style Code Attack (SCA) to maintain audio quality. Experimental results show that our method can meet the need for user-customized styles and achieve a success rate of 82% in attacks, while kee** sound naturalness due to our user study.
△ Less
Submitted 15 May, 2024;
originally announced May 2024.
-
A Novel Vision Transformer based Load Profile Analysis using Load Images as Inputs
Authors:
Hyeon** Kim,
Yi Hu,
Kai Ye,
Ning Lu
Abstract:
This paper introduces ViT4LPA, an innovative Vision Transformer (ViT) based approach for Load Profile Analysis (LPA). We transform time-series load profiles into load images. This allows us to leverage the ViT architecture, originally designed for image processing, as a pre-trained image encoder to uncover latent patterns within load data. ViT is pre-trained using an extensive load image dataset,…
▽ More
This paper introduces ViT4LPA, an innovative Vision Transformer (ViT) based approach for Load Profile Analysis (LPA). We transform time-series load profiles into load images. This allows us to leverage the ViT architecture, originally designed for image processing, as a pre-trained image encoder to uncover latent patterns within load data. ViT is pre-trained using an extensive load image dataset, comprising 1M load images derived from smart meter data collected over a two-year period from 2,000 residential users. The training methodology is self-supervised, masked image modeling, wherein masked load images are restored to reveal hidden relationships among image patches. The pre-trained ViT encoder is then applied to various downstream tasks, including the identification of electric vehicle (EV) charging loads and behind-the-meter solar photovoltaic (PV) systems and load disaggregation. Simulation results illustrate ViT4LPA's superior performance compared to existing neural network models in downstream tasks. Additionally, we conduct an in-depth analysis of the attention weights within the ViT4LPA model to gain insights into its information flow mechanisms.
△ Less
Submitted 11 April, 2024;
originally announced April 2024.
-
BERT-PIN: A BERT-based Framework for Recovering Missing Data Segments in Time-series Load Profiles
Authors:
Yi Hu,
Kai Ye,
Hyeon** Kim,
Ning Lu
Abstract:
Inspired by the success of the Transformer model in natural language processing and computer vision, this paper introduces BERT-PIN, a Bidirectional Encoder Representations from Transformers (BERT) powered Profile Inpainting Network. BERT-PIN recovers multiple missing data segments (MDSs) using load and temperature time-series profiles as inputs. To adopt a standard Transformer model structure for…
▽ More
Inspired by the success of the Transformer model in natural language processing and computer vision, this paper introduces BERT-PIN, a Bidirectional Encoder Representations from Transformers (BERT) powered Profile Inpainting Network. BERT-PIN recovers multiple missing data segments (MDSs) using load and temperature time-series profiles as inputs. To adopt a standard Transformer model structure for profile inpainting, we segment the load and temperature profiles into line segments, treating each segment as a word and the entire profile as a sentence. We incorporate a top candidates selection process in BERT-PIN, enabling it to produce a sequence of probability distributions, based on which users can generate multiple plausible imputed data sets, each reflecting different confidence levels. We develop and evaluate BERT-PIN using real-world dataset for two applications: multiple MDSs recovery and demand response baseline estimation. Simulation results show that BERT-PIN outperforms the existing methods in accuracy while is capable of restoring multiple MDSs within a longer window. BERT-PIN, served as a pre-trained model, can be fine-tuned for conducting many downstream tasks, such as classification and super resolution.
△ Less
Submitted 26 October, 2023;
originally announced October 2023.
-
Linearized Integrated Microwave Photonic Circuit for Filtering and Phase Shifting
Authors:
Gaojian Liu,
Kaixuan Ye,
Okky Daulay,
Qinggui Tan,
Hongxi Yu,
David Marpaung
Abstract:
Photonic integration, advanced functionality, reconfigurability, and high RF performance are key features in integrated microwave photonic systems that are still difficult to achieve simultaneously. In this work, we demonstrate an integrated microwave photonic circuit that can be reconfigured for two distinct RF functions, namely, a tunable notch filter and a phase shifter. We achieved $>$50dB hig…
▽ More
Photonic integration, advanced functionality, reconfigurability, and high RF performance are key features in integrated microwave photonic systems that are still difficult to achieve simultaneously. In this work, we demonstrate an integrated microwave photonic circuit that can be reconfigured for two distinct RF functions, namely, a tunable notch filter and a phase shifter. We achieved $>$50dB high-extinction notch filtering over 6-16 GHz and 2$π$ continuously tunable phase shifting over 12-20 GHz frequencies. At the same time, we implemented an on-chip linearization technique to achieve a spurious-free dynamic range of more than 120$\rm{dB}\cdot \rm{Hz}^{4/5}$ for both functions. Our work combines multi-functionality and linearization in one photonic integrated circuit, and paves the way to reconfigurable RF photonic front-ends with very high performance.
△ Less
Submitted 26 February, 2023;
originally announced February 2023.
-
Evaluation of Legged Robot Landing Capability Under Aggressive Linear and Angular Velocities
Authors:
Keran Ye,
Konstantinos Karydis
Abstract:
This paper proposes a method to evaluate the capability of aggressive legged robot landing under significant touchdown linear and angular velocities upon impact. Our approach builds upon the Planar Inverted Pendulum with Flywheel (PIPF) model and introduces a landing framework for the first stance step on a non-dimensional basis. We develop a nonlinear framework with iterative constrained trajecto…
▽ More
This paper proposes a method to evaluate the capability of aggressive legged robot landing under significant touchdown linear and angular velocities upon impact. Our approach builds upon the Planar Inverted Pendulum with Flywheel (PIPF) model and introduces a landing framework for the first stance step on a non-dimensional basis. We develop a nonlinear framework with iterative constrained trajectory optimization to stabilize the first stance step prior to N-step Capturability analysis. Performance maps across many different initial conditions reveal approximately linear boundaries as well as the effect of inertia, body incidence angle and leg attacking angle on the boundary shape. Our method also yields the engineering insight that body inertia affects the performance map the most, hence its optimization can be prioritized when the target is to improve robot landing efficacy.
△ Less
Submitted 23 February, 2023;
originally announced February 2023.
-
Design Considerations of a Coordinative Demand Charge Mitigation Strategy
Authors:
Rongxing Hu,
Kai Ye,
Hyeon** Kim,
Hanpyo Lee,
Ning Lu,
Di Wu,
PJ Rehm
Abstract:
This paper presents a coordinative demand charge mitigation (DCM) strategy for reducing electricity consumption during system peak periods. Available DCM resources include batteries, diesel generators, controllable loads, and conservation voltage reduction. All resources are directly controlled by load serving entities. A mixed integer linear programming based energy management algorithm is develo…
▽ More
This paper presents a coordinative demand charge mitigation (DCM) strategy for reducing electricity consumption during system peak periods. Available DCM resources include batteries, diesel generators, controllable loads, and conservation voltage reduction. All resources are directly controlled by load serving entities. A mixed integer linear programming based energy management algorithm is developed to optimally coordinate of DCM resources considering the load payback effect. To better capture system peak periods, two different kinds of load forecast are used: the day-ahead load forecast and the peak-hour probability forecast. Five DCM strategies are compared for reconciling the discrepancy between the two forecasting results. The DCM strategies are tested using actual utility data. Simulation results show that the proposed algorithm can effectively mitigate the demand charge while preventing the system peak from being shifted to the payback hours. We also identify the diminishing return effect, which can help load serving entities optimize the size of their DCM resources.
△ Less
Submitted 1 February, 2023; v1 submitted 16 December, 2022;
originally announced December 2022.
-
A Modified Sequence-to-point HVAC Load Disaggregation Algorithm
Authors:
Kai Ye,
Hyeon** Kim,
Yi Hu,
Ning Lu,
Di Wu,
PJ Rehm
Abstract:
This paper presents a modified sequence-to-point (S2P) algorithm for disaggregating the heat, ventilation, and air conditioning (HVAC) load from the total building electricity consumption. The original S2P model is convolutional neural network (CNN) based, which uses load profiles as inputs. We propose three modifications. First, the input convolution layer is changed from 1D to 2D so that normali…
▽ More
This paper presents a modified sequence-to-point (S2P) algorithm for disaggregating the heat, ventilation, and air conditioning (HVAC) load from the total building electricity consumption. The original S2P model is convolutional neural network (CNN) based, which uses load profiles as inputs. We propose three modifications. First, the input convolution layer is changed from 1D to 2D so that normalized temperature profiles are also used as inputs to the S2P model. Second, a drop-out layer is added to improve adaptability and generalizability so that the model trained in one area can be transferred to other geographical areas without labelled HVAC data. Third, a fine-tuning process is proposed for areas with a small amount of labelled HVAC data so that the pre-trained S2P model can be fine-tuned to achieve higher disaggregation accuracy (i.e., better transferability) in other areas. The model is first trained and tested using smart meter and sub-metered HVAC data collected in Austin, Texas. Then, the trained model is tested on two other areas: Boulder, Colorado and San Diego, California. Simulation results show that the proposed modified S2P algorithm outperforms the original S2P model and the support-vector machine based approach in accuracy, adaptability, and transferability.
△ Less
Submitted 24 February, 2023; v1 submitted 9 December, 2022;
originally announced December 2022.
-
An ICA-Based HVAC Load Disaggregation Method Using Smart Meter Data
Authors:
Hyeon** Kim,
Kai Ye,
Han Pyo Lee,
Rongxing Hu,
Ning Lu,
Di Wu,
PJ Rehm
Abstract:
This paper presents an independent component analysis (ICA) based unsupervised-learning method for heat, ventilation, and air-conditioning (HVAC) load disaggregation using low-resolution (e.g., 15 minutes) smart meter data. We first demonstrate that electricity consumption profiles on mild-temperature days can be used to estimate the non-HVAC base load on hot days. A residual load profile can then…
▽ More
This paper presents an independent component analysis (ICA) based unsupervised-learning method for heat, ventilation, and air-conditioning (HVAC) load disaggregation using low-resolution (e.g., 15 minutes) smart meter data. We first demonstrate that electricity consumption profiles on mild-temperature days can be used to estimate the non-HVAC base load on hot days. A residual load profile can then be calculated by subtracting the mild-day load profile from the hot-day load profile. The residual load profiles are processed using ICA for HVAC load extraction. An optimization-based algorithm is proposed for post-adjustment of the ICA results, considering two bounding factors for enhancing the robustness of the ICA algorithm. First, we use the hourly HVAC energy bounds computed based on the relationship between HVAC load and temperature to remove unrealistic HVAC load spikes. Second, we exploit the dependency between the daily nocturnal and diurnal loads extracted from historical meter data to smooth the base load profile. Pecan Street data with sub-metered HVAC data were used to test and validate the proposed methods.Simulation results demonstrated that the proposed method is computationally efficient and robust across multiple customers.
△ Less
Submitted 19 September, 2022;
originally announced September 2022.
-
KiPA22 Report: U-Net with Contour Regularization for Renal Structures Segmentation
Authors:
Kangqing Ye,
Peng Liu,
Xiaoyang Zou,
Qin Zhou,
Guoyan Zheng
Abstract:
Three-dimensional (3D) integrated renal structures (IRS) segmentation is important in clinical practice. With the advancement of deep learning techniques, many powerful frameworks focusing on medical image segmentation are proposed. In this challenge, we utilized the nnU-Net framework, which is the state-of-the-art method for medical image segmentation. To reduce the outlier prediction for the tum…
▽ More
Three-dimensional (3D) integrated renal structures (IRS) segmentation is important in clinical practice. With the advancement of deep learning techniques, many powerful frameworks focusing on medical image segmentation are proposed. In this challenge, we utilized the nnU-Net framework, which is the state-of-the-art method for medical image segmentation. To reduce the outlier prediction for the tumor label, we combine contour regularization (CR) loss of the tumor label with Dice loss and cross-entropy loss to improve this phenomenon.
△ Less
Submitted 6 September, 2022; v1 submitted 10 August, 2022;
originally announced August 2022.
-
Modeling and Trajectory Optimization for Standing Long Jum** of a Quadruped with A Preloaded Elastic Prismatic Spine
Authors:
Keran Ye,
Konstantinos Karydis
Abstract:
This paper presents a novel methodology to model and optimize trajectories of a quadrupedal robot with spinal compliance to improve standing jump performance compared to quadrupeds with a rigid spine. We introduce an elastic model for a prismatic robotic spine that is actively preloaded and mechanically lock-enabled at initial and maximum length, and develop a constrained trajectory optimization m…
▽ More
This paper presents a novel methodology to model and optimize trajectories of a quadrupedal robot with spinal compliance to improve standing jump performance compared to quadrupeds with a rigid spine. We introduce an elastic model for a prismatic robotic spine that is actively preloaded and mechanically lock-enabled at initial and maximum length, and develop a constrained trajectory optimization method to co-optimize the elastic parameters and motion trajectories toward enhanced jum** distance. Results reveal that a less stiff spring is likely to facilitate jum** performance not as a direct propelling source but as a means to unleash more motor power for propelling by trading-off overall energy efficiency. We also visualize the impact of spring coefficients on the overall optimization routine from energetic perspectives to identify the suitable parameter region.
△ Less
Submitted 31 August, 2021;
originally announced September 2021.
-
Independent Encoder for Deep Hierarchical Unsupervised Image-to-Image Translation
Authors:
Kai Ye,
Yinru Ye,
Minqiang Yang,
Bin Hu
Abstract:
The main challenges of image-to-image (I2I) translation are to make the translated image realistic and retain as much information from the source domain as possible. To address this issue, we propose a novel architecture, termed as IEGAN, which removes the encoder of each network and introduces an encoder that is independent of other networks. Compared with previous models, it embodies three advan…
▽ More
The main challenges of image-to-image (I2I) translation are to make the translated image realistic and retain as much information from the source domain as possible. To address this issue, we propose a novel architecture, termed as IEGAN, which removes the encoder of each network and introduces an encoder that is independent of other networks. Compared with previous models, it embodies three advantages of our model: Firstly, it is more directly and comprehensively to grasp image information since the encoder no longer receives loss from generator and discriminator. Secondly, the independent encoder allows each network to focus more on its own goal which makes the translated image more realistic. Thirdly, the reduction in the number of encoders performs more unified image representation. However, when the independent encoder applies two down-sampling blocks, it's hard to extract semantic information. To tackle this problem, we propose deep and shallow information space containing characteristic and semantic information, which can guide the model to translate high-quality images under the task with significant shape or texture change. We compare IEGAN with other previous models, and conduct researches on semantic information consistency and component ablation at the same time. These experiments show the superiority and effectiveness of our architecture. Our code is published on: https://github.com/Elvinky/IEGAN.
△ Less
Submitted 6 July, 2021;
originally announced July 2021.
-
How to Build a Graph-Based Deep Learning Architecture in Traffic Domain: A Survey
Authors:
Jiexia Ye,
Juanjuan Zhao,
Kejiang Ye,
Chengzhong Xu
Abstract:
In recent years, various deep learning architectures have been proposed to solve complex challenges (e.g. spatial dependency, temporal dependency) in traffic domain, which have achieved satisfactory performance. These architectures are composed of multiple deep learning techniques in order to tackle various challenges in traffic tasks. Traditionally, convolution neural networks (CNNs) are utilized…
▽ More
In recent years, various deep learning architectures have been proposed to solve complex challenges (e.g. spatial dependency, temporal dependency) in traffic domain, which have achieved satisfactory performance. These architectures are composed of multiple deep learning techniques in order to tackle various challenges in traffic tasks. Traditionally, convolution neural networks (CNNs) are utilized to model spatial dependency by decomposing the traffic network as grids. However, many traffic networks are graph-structured in nature. In order to utilize such spatial information fully, it's more appropriate to formulate traffic networks as graphs mathematically. Recently, various novel deep learning techniques have been developed to process graph data, called graph neural networks (GNNs). More and more works combine GNNs with other deep learning techniques to construct an architecture dealing with various challenges in a complex traffic task, where GNNs are responsible for extracting spatial correlations in traffic network. These graph-based architectures have achieved state-of-the-art performance. To provide a comprehensive and clear picture of such emerging trend, this survey carefully examines various graph-based deep learning architectures in many traffic applications. We first give guidelines to formulate a traffic problem based on graph and construct graphs from various kinds of traffic datasets. Then we decompose these graph-based architectures to discuss their shared deep learning techniques, clarifying the utilization of each technique in traffic tasks. What's more, we summarize some common traffic challenges and the corresponding graph-based deep learning solutions to each challenge. Finally, we provide benchmark datasets, open source codes and future research directions in this rapidly growing field.
△ Less
Submitted 10 October, 2020; v1 submitted 24 May, 2020;
originally announced May 2020.
