-
Enhancing In-context Learning via Linear Probe Calibration
Authors:
Momin Abbas,
Yi Zhou,
Parikshit Ram,
Nathalie Baracaldo,
Horst Samulowitz,
Theodoros Salonidis,
Tianyi Chen
Abstract:
In-context learning (ICL) is a new paradigm for natural language processing that utilizes Generative Pre-trained Transformer (GPT)-like models. This approach uses prompts that include in-context demonstrations to generate the corresponding output for a new query input. However, applying ICL in real cases does not scale with the number of samples, and lacks robustness to different prompt templates…
▽ More
In-context learning (ICL) is a new paradigm for natural language processing that utilizes Generative Pre-trained Transformer (GPT)-like models. This approach uses prompts that include in-context demonstrations to generate the corresponding output for a new query input. However, applying ICL in real cases does not scale with the number of samples, and lacks robustness to different prompt templates and demonstration permutations. In this paper, we first show that GPT-like models using ICL result in unreliable predictions based on a new metric based on Shannon entropy. Then, to solve this problem, we propose a new technique called the Linear Probe Calibration (LinC), a method that calibrates the model's output probabilities, resulting in reliable predictions and improved performance, while requiring only minimal additional samples (as few as five labeled data samples). LinC significantly enhances the ICL test performance of GPT models on various benchmark datasets, with an average improvement of up to 21%, and up to a 50% improvement in some cases, and significantly boosts the performance of PEFT methods, especially in the low resource regime. Moreover, LinC achieves lower expected calibration error, and is highly robust to varying label proportions, prompt templates, and demonstration permutations. Our code is available at \url{https://github.com/mominabbass/LinC}.
△ Less
Submitted 22 January, 2024;
originally announced January 2024.
-
Single-shot Hyper-parameter Optimization for Federated Learning: A General Algorithm & Analysis
Authors:
Yi Zhou,
Parikshit Ram,
Theodoros Salonidis,
Nathalie Baracaldo,
Horst Samulowitz,
Heiko Ludwig
Abstract:
We address the relatively unexplored problem of hyper-parameter optimization (HPO) for federated learning (FL-HPO). We introduce Federated Loss SuRface Aggregation (FLoRA), a general FL-HPO solution framework that can address use cases of tabular data and any Machine Learning (ML) model including gradient boosting training algorithms and therefore further expands the scope of FL-HPO. FLoRA enables…
▽ More
We address the relatively unexplored problem of hyper-parameter optimization (HPO) for federated learning (FL-HPO). We introduce Federated Loss SuRface Aggregation (FLoRA), a general FL-HPO solution framework that can address use cases of tabular data and any Machine Learning (ML) model including gradient boosting training algorithms and therefore further expands the scope of FL-HPO. FLoRA enables single-shot FL-HPO: identifying a single set of good hyper-parameters that are subsequently used in a single FL training. Thus, it enables FL-HPO solutions with minimal additional communication overhead compared to FL training without HPO. We theoretically characterize the optimality gap of FL-HPO, which explicitly accounts for the heterogeneous non-IID nature of the parties' local data distributions, a dominant characteristic of FL systems. Our empirical evaluation of FLoRA for multiple ML algorithms on seven OpenML datasets demonstrates significant model accuracy improvements over the considered baseline, and robustness to increasing number of parties involved in FL-HPO training.
△ Less
Submitted 16 February, 2022;
originally announced February 2022.
-
Selective Edge Computing for Mobile Analytics
Authors:
Apostolos Galanopoulos,
George Iosifidis,
Theodoros Salonidis,
Douglas J. Leith
Abstract:
An increasing number of mobile applications rely on Machine Learning (ML) routines for analyzing data. Executing such tasks at the user devices saves the energy spent on transmitting and processing large data volumes at distant cloud-deployed servers. However, due to memory and computing limitations, the devices often cannot support the required resource-intensive routines and fail to accurately e…
▽ More
An increasing number of mobile applications rely on Machine Learning (ML) routines for analyzing data. Executing such tasks at the user devices saves the energy spent on transmitting and processing large data volumes at distant cloud-deployed servers. However, due to memory and computing limitations, the devices often cannot support the required resource-intensive routines and fail to accurately execute the tasks. In this work, we address the problem of edge-assisted analytics in resource-constrained systems by proposing and evaluating a rigorous selective offloading framework. The devices execute their tasks locally and outsource them to cloudlet servers only when they predict a significant performance improvement. We consider the practical scenario where the offloading gain and resource costs are time-varying; and propose an online optimization algorithm that maximizes the service performance without requiring to know this information. Our approach relies on an approximate dual subgradient method combined with a primal-averaging scheme, and works under minimal assumptions about the system stochasticity. We fully implement the proposed algorithm in a wireless testbed and evaluate its performance using a state-of-the-art image recognition application, finding significant performance gains and cost savings.
△ Less
Submitted 8 January, 2022;
originally announced January 2022.
-
FLoRA: Single-shot Hyper-parameter Optimization for Federated Learning
Authors:
Yi Zhou,
Parikshit Ram,
Theodoros Salonidis,
Nathalie Baracaldo,
Horst Samulowitz,
Heiko Ludwig
Abstract:
We address the relatively unexplored problem of hyper-parameter optimization (HPO) for federated learning (FL-HPO). We introduce Federated Loss suRface Aggregation (FLoRA), the first FL-HPO solution framework that can address use cases of tabular data and gradient boosting training algorithms in addition to stochastic gradient descent/neural networks commonly addressed in the FL literature. The fr…
▽ More
We address the relatively unexplored problem of hyper-parameter optimization (HPO) for federated learning (FL-HPO). We introduce Federated Loss suRface Aggregation (FLoRA), the first FL-HPO solution framework that can address use cases of tabular data and gradient boosting training algorithms in addition to stochastic gradient descent/neural networks commonly addressed in the FL literature. The framework enables single-shot FL-HPO, by first identifying a good set of hyper-parameters that are used in a **single** FL training. Thus, it enables FL-HPO solutions with minimal additional communication overhead compared to FL training without HPO. Our empirical evaluation of FLoRA for Gradient Boosted Decision Trees on seven OpenML data sets demonstrates significant model accuracy improvements over the considered baseline, and robustness to increasing number of parties involved in FL-HPO training.
△ Less
Submitted 15 December, 2021;
originally announced December 2021.
-
Improving IoT Analytics through Selective Edge Execution
Authors:
A. Galanopoulos,
A. G. Tasiopoulos,
G. Iosifidis,
T. Salonidis,
D. J. Leith
Abstract:
A large number of emerging IoT applications rely on machine learning routines for analyzing data. Executing such tasks at the user devices improves response time and economizes network resources. However, due to power and computing limitations, the devices often cannot support such resource-intensive routines and fail to accurately execute the analytics. In this work, we propose to improve the per…
▽ More
A large number of emerging IoT applications rely on machine learning routines for analyzing data. Executing such tasks at the user devices improves response time and economizes network resources. However, due to power and computing limitations, the devices often cannot support such resource-intensive routines and fail to accurately execute the analytics. In this work, we propose to improve the performance of analytics by leveraging edge infrastructure. We devise an algorithm that enables the IoT devices to execute their routines locally; and then outsource them to cloudlet servers, only if they predict they will gain a significant performance improvement. It uses an approximate dual subgradient method, making minimal assumptions about the statistical properties of the system's parameters. Our analysis demonstrates that our proposed algorithm can intelligently leverage the cloudlet, adapting to the service requirements.
△ Less
Submitted 7 March, 2020;
originally announced March 2020.
-
Anonymizing Data for Privacy-Preserving Federated Learning
Authors:
Olivia Choudhury,
Aris Gkoulalas-Divanis,
Theodoros Salonidis,
Issa Sylla,
Yoonyoung Park,
Grace Hsu,
Amar Das
Abstract:
Federated learning enables training a global machine learning model from data distributed across multiple sites, without having to move the data. This is particularly relevant in healthcare applications, where data is rife with personal, highly-sensitive information, and data analysis methods must provably comply with regulatory guidelines. Although federated learning prevents sharing raw data, it…
▽ More
Federated learning enables training a global machine learning model from data distributed across multiple sites, without having to move the data. This is particularly relevant in healthcare applications, where data is rife with personal, highly-sensitive information, and data analysis methods must provably comply with regulatory guidelines. Although federated learning prevents sharing raw data, it is still possible to launch privacy attacks on the model parameters that are exposed during the training process, or on the generated machine learning model. In this paper, we propose the first syntactic approach for offering privacy in the context of federated learning. Unlike the state-of-the-art differential privacy-based frameworks, our approach aims to maximize utility or model performance, while supporting a defensible level of privacy, as demanded by GDPR and HIPAA. We perform a comprehensive empirical evaluation on two important problems in the healthcare domain, using real-world electronic health data of 1 million patients. The results demonstrate the effectiveness of our approach in achieving high model performance, while offering the desired level of privacy. Through comparative studies, we also show that, for varying datasets, experimental setups, and privacy budgets, our approach offers higher model performance than differential privacy-based techniques in federated learning.
△ Less
Submitted 20 February, 2020;
originally announced February 2020.
-
Differential Privacy-enabled Federated Learning for Sensitive Health Data
Authors:
Olivia Choudhury,
Aris Gkoulalas-Divanis,
Theodoros Salonidis,
Issa Sylla,
Yoonyoung Park,
Grace Hsu,
Amar Das
Abstract:
Leveraging real-world health data for machine learning tasks requires addressing many practical challenges, such as distributed data silos, privacy concerns with creating a centralized database from person-specific sensitive data, resource constraints for transferring and integrating data from multiple sites, and risk of a single point of failure. In this paper, we introduce a federated learning f…
▽ More
Leveraging real-world health data for machine learning tasks requires addressing many practical challenges, such as distributed data silos, privacy concerns with creating a centralized database from person-specific sensitive data, resource constraints for transferring and integrating data from multiple sites, and risk of a single point of failure. In this paper, we introduce a federated learning framework that can learn a global model from distributed health data held locally at different sites. The framework offers two levels of privacy protection. First, it does not move or share raw data across sites or with a centralized server during the model training process. Second, it uses a differential privacy mechanism to further protect the model from potential privacy attacks. We perform a comprehensive evaluation of our approach on two healthcare applications, using real-world electronic health data of 1 million patients. We demonstrate the feasibility and effectiveness of the federated learning framework in offering an elevated level of privacy and maintaining utility of the global model.
△ Less
Submitted 27 February, 2020; v1 submitted 6 October, 2019;
originally announced October 2019.
-
Maximum Lifetime Analytics in IoT Networks
Authors:
Victor Valls,
George Iosifidis,
Theodoros Salonidis
Abstract:
This paper studies the problem of allocating bandwidth and computation resources to data analytics tasks in Internet of Things (IoT) networks. IoT nodes are powered by batteries, can process (some of) the data locally, and the quality grade or performance of how data analytics tasks are carried out depends on where these are executed. The goal is to design a resource allocation algorithm that join…
▽ More
This paper studies the problem of allocating bandwidth and computation resources to data analytics tasks in Internet of Things (IoT) networks. IoT nodes are powered by batteries, can process (some of) the data locally, and the quality grade or performance of how data analytics tasks are carried out depends on where these are executed. The goal is to design a resource allocation algorithm that jointly maximizes the network lifetime and the performance of the data analytics tasks subject to energy constraints. This joint maximization problem is challenging with coupled resource constraints that induce non-convexity. We first show that the problem can be mapped to an equivalent convex problem, and then propose an online algorithm that provably solves the problem and does not require any a priori knowledge of the time-varying wireless link capacities and data analytics arrival process statistics. The algorithm's optimality properties are derived using an analysis which, to the best of our knowledge, proves for the first time the convergence of the dual subgradient method with time-varying sets. Our simulations seeded by real IoT device energy measurements, show that the network connectivity plays a crucial role in network lifetime maximization, that the algorithm can obtain both maximum network lifetime and maximum data analytics performance in addition to maximizing the joint objective, and that the algorithm increases the network lifetime by approximately 50% compared to an algorithm that minimizes the total energy consumption.
△ Less
Submitted 22 April, 2019;
originally announced April 2019.
-
Adaptive Federated Learning in Resource Constrained Edge Computing Systems
Authors:
Shiqiang Wang,
Tiffany Tuor,
Theodoros Salonidis,
Kin K. Leung,
Christian Makaya,
Ting He,
Kevin Chan
Abstract:
Emerging technologies and applications including Internet of Things (IoT), social networking, and crowd-sourcing generate large amounts of data at the network edge. Machine learning models are often built from the collected data, to enable the detection, classification, and prediction of future events. Due to bandwidth, storage, and privacy concerns, it is often impractical to send all the data to…
▽ More
Emerging technologies and applications including Internet of Things (IoT), social networking, and crowd-sourcing generate large amounts of data at the network edge. Machine learning models are often built from the collected data, to enable the detection, classification, and prediction of future events. Due to bandwidth, storage, and privacy concerns, it is often impractical to send all the data to a centralized location. In this paper, we consider the problem of learning model parameters from data distributed across multiple edge nodes, without sending raw data to a centralized place. Our focus is on a generic class of machine learning models that are trained using gradient-descent based approaches. We analyze the convergence bound of distributed gradient descent from a theoretical point of view, based on which we propose a control algorithm that determines the best trade-off between local update and global parameter aggregation to minimize the loss function under a given resource budget. The performance of the proposed algorithm is evaluated via extensive experiments with real datasets, both on a networked prototype system and in a larger-scale simulated environment. The experimentation results show that our proposed approach performs near to the optimum with various machine learning models and different data distributions.
△ Less
Submitted 16 February, 2019; v1 submitted 14 April, 2018;
originally announced April 2018.
-
Live Service Migration in Mobile Edge Clouds
Authors:
Andrew Machen,
Shiqiang Wang,
Kin K. Leung,
Bong Jun Ko,
Theodoros Salonidis
Abstract:
Mobile edge clouds (MECs) bring the benefits of the cloud closer to the user, by installing small cloud infrastructures at the network edge. This enables a new breed of real-time applications, such as instantaneous object recognition and safety assistance in intelligent transportation systems, that require very low latency. One key issue that comes with proximity is how to ensure that users always…
▽ More
Mobile edge clouds (MECs) bring the benefits of the cloud closer to the user, by installing small cloud infrastructures at the network edge. This enables a new breed of real-time applications, such as instantaneous object recognition and safety assistance in intelligent transportation systems, that require very low latency. One key issue that comes with proximity is how to ensure that users always receive good performance as they move across different locations. Migrating services between MECs is seen as the means to achieve this. This article presents a layered framework for migrating active service applications that are encapsulated either in virtual machines (VMs) or containers. This layering approach allows a substantial reduction in service downtime. The framework is easy to implement using readily available technologies, and one of its key advantages is that it supports containers, which is a promising emerging technology that offers tangible benefits over VMs. The migration performance of various real applications is evaluated by experiments under the presented framework. Insights drawn from the experimentation results are discussed.
△ Less
Submitted 2 August, 2017; v1 submitted 13 June, 2017;
originally announced June 2017.
-
On the Complexity of Optimal Routing and Content Caching in Heterogeneous Networks
Authors:
Mostafa Dehghan,
Anand Seetharam,
Bo Jiang,
Ting He,
Theodoros Salonidis,
Jim Kurose,
Don Towsley,
Ramesh Sitaraman
Abstract:
We investigate the problem of optimal request routing and content caching in a heterogeneous network supporting in-network content caching with the goal of minimizing average content access delay. Here, content can either be accessed directly from a back-end server (where content resides permanently) or be obtained from one of multiple in-network caches. To access a piece of content, a user must d…
▽ More
We investigate the problem of optimal request routing and content caching in a heterogeneous network supporting in-network content caching with the goal of minimizing average content access delay. Here, content can either be accessed directly from a back-end server (where content resides permanently) or be obtained from one of multiple in-network caches. To access a piece of content, a user must decide whether to route its request to a cache or to the back-end server. Additionally, caches must decide which content to cache. We investigate the problem complexity of two problem formulations, where the direct path to the back-end server is modeled as i) a congestion-sensitive or ii) a congestion-insensitive path, reflecting whether or not the delay of the uncached path to the back-end server depends on the user request load, respectively. We show that the problem is NP-complete in both cases. We prove that under the congestion-insensitive model the problem can be solved optimally in polynomial time if each piece of content is requested by only one user, or when there are at most two caches in the network. We also identify a structural property of the user-cache graph that potentially makes the problem NP-complete. For the congestion-sensitive model, we prove that the problem remains NP-complete even if there is only one cache in the network and each content is requested by only one user. We show that approximate solutions can be found for both models within a (1-1/e) factor of the optimal solution, and demonstrate a greedy algorithm that is found to be within 1% of optimal for small problem sizes. Through trace-driven simulations we evaluate the performance of our greedy algorithms, which show up to a 50% reduction in average delay over solutions based on LRU content caching.
△ Less
Submitted 31 December, 2014;
originally announced January 2015.
-
Optimal Caching and Routing in Hybrid Networks
Authors:
Mostafa Dehghan,
Anand Seetharam,
Ting He,
Theodoros Salonidis,
Jim Kurose,
Don Towsley
Abstract:
Hybrid networks consisting of MANET nodes and cellular infrastructure have been recently proposed to improve the performance of military networks. Prior work has demonstrated the benefits of in-network content caching in a wired, Internet context. We investigate the problem of develo** optimal routing and caching policies in a hybrid network supporting in-network caching with the goal of minimiz…
▽ More
Hybrid networks consisting of MANET nodes and cellular infrastructure have been recently proposed to improve the performance of military networks. Prior work has demonstrated the benefits of in-network content caching in a wired, Internet context. We investigate the problem of develo** optimal routing and caching policies in a hybrid network supporting in-network caching with the goal of minimizing overall content-access delay. Here, needed content may always be accessed at a back-end server via the cellular infrastructure; alternatively, content may also be accessed via cache-equipped "cluster" nodes within the MANET. To access content, MANET nodes must thus decide whether to route to in-MANET cluster nodes or to back-end servers via the cellular infrastructure; the in-MANET cluster nodes must additionally decide which content to cache. We model the cellular path as either i) a congestion-insensitive fixed-delay path or ii) a congestion-sensitive path modeled as an M/M/1 queue. We demonstrate that under the assumption of stationary, independent requests, it is optimal to adopt static caching (i.e., to keep a cache's content fixed over time) based on content popularity. We also show that it is optimal to route to in-MANET caches for content cached there, but to route requests for remaining content via the cellular infrastructure for the congestion-insensitive case and to split traffic between the in-MANET caches and cellular infrastructure for the congestion-sensitive case. We develop a simple distributed algorithm for the joint routing/caching problem and demonstrate its efficacy via simulation.
△ Less
Submitted 7 July, 2014;
originally announced July 2014.
-
Optimal control of end-user energy storage
Authors:
Peter M. van de Ven,
Nidhi Hegde,
Laurent Massoulie,
Theodoros Salonidis
Abstract:
An increasing number of retail energy markets show price fluctuations, providing users with the opportunity to buy energy at lower than average prices. We propose to temporarily store this inexpensive energy in a battery, and use it to satisfy demand when energy prices are high, thus allowing users to exploit the price variations without having to shift their demand to the low-price periods. We st…
▽ More
An increasing number of retail energy markets show price fluctuations, providing users with the opportunity to buy energy at lower than average prices. We propose to temporarily store this inexpensive energy in a battery, and use it to satisfy demand when energy prices are high, thus allowing users to exploit the price variations without having to shift their demand to the low-price periods. We study the battery control policy that yields the best performance, i.e., minimizes the total discounted costs. The optimal policy is shown to have a threshold structure, and we derive these thresholds in a few special cases. The cost savings obtained from energy storage are demonstrated through extensive numerical experiments, and we offer various directions for future research.
△ Less
Submitted 5 December, 2012; v1 submitted 8 March, 2012;
originally announced March 2012.
