-
Swee** Heterogeneity with Smart MoPs: Mixture of Prompts for LLM Task Adaptation
Authors:
Chen Dun,
Mirian Hipolito Garcia,
Guoqing Zheng,
Ahmed Hassan Awadallah,
Anastasios Kyrillidis,
Robert Sim
Abstract:
Large Language Models (LLMs) have the ability to solve a variety of tasks, such as text summarization and mathematical questions, just out of the box, but they are often trained with a single task in mind. Due to high computational costs, the current trend is to use prompt instruction tuning to better adjust monolithic, pretrained LLMs for new -- but often individual -- downstream tasks. Thus, how…
▽ More
Large Language Models (LLMs) have the ability to solve a variety of tasks, such as text summarization and mathematical questions, just out of the box, but they are often trained with a single task in mind. Due to high computational costs, the current trend is to use prompt instruction tuning to better adjust monolithic, pretrained LLMs for new -- but often individual -- downstream tasks. Thus, how one would expand prompt tuning to handle -- concomitantly -- heterogeneous tasks and data distributions is a widely open question. To address this gap, we suggest the use of \emph{Mixture of Prompts}, or MoPs, associated with smart gating functionality: the latter -- whose design is one of the contributions of this paper -- can identify relevant skills embedded in different groups of prompts and dynamically assign combined experts (i.e., collection of prompts), based on the target task. Additionally, MoPs are empirically agnostic to any model compression technique applied -- for efficiency reasons -- as well as instruction data source and task composition. In practice, MoPs can simultaneously mitigate prompt training "interference" in multi-task, multi-source scenarios (e.g., task and data heterogeneity across sources), as well as possible implications from model approximations. As a highlight, MoPs manage to decrease final perplexity from $\sim20\%$ up to $\sim70\%$, as compared to baselines, in the federated scenario, and from $\sim 3\%$ up to $\sim30\%$ in the centralized scenario.
△ Less
Submitted 5 October, 2023; v1 submitted 4 October, 2023;
originally announced October 2023.
-
FedJETs: Efficient Just-In-Time Personalization with Federated Mixture of Experts
Authors:
Chen Dun,
Mirian Hipolito Garcia,
Guoqing Zheng,
Ahmed Hassan Awadallah,
Robert Sim,
Anastasios Kyrillidis,
Dimitrios Dimitriadis
Abstract:
One of the goals in Federated Learning (FL) is to create personalized models that can adapt to the context of each participating client, while utilizing knowledge from a shared global model. Yet, often, personalization requires a fine-tuning step using clients' labeled data in order to achieve good performance. This may not be feasible in scenarios where incoming clients are fresh and/or have priv…
▽ More
One of the goals in Federated Learning (FL) is to create personalized models that can adapt to the context of each participating client, while utilizing knowledge from a shared global model. Yet, often, personalization requires a fine-tuning step using clients' labeled data in order to achieve good performance. This may not be feasible in scenarios where incoming clients are fresh and/or have privacy concerns. It, then, remains open how one can achieve just-in-time personalization in these scenarios. We propose FedJETs, a novel solution by using a Mixture-of-Experts (MoE) framework within a FL setup. Our method leverages the diversity of the clients to train specialized experts on different subsets of classes, and a gating function to route the input to the most relevant expert(s). Our gating function harnesses the knowledge of a pretrained model common expert to enhance its routing decisions on-the-fly. As a highlight, our approach can improve accuracy up to 18\% in state of the art FL settings, while maintaining competitive zero-shot performance. In practice, our method can handle non-homogeneous data distributions, scale more efficiently, and improve the state-of-the-art performance on common FL benchmarks.
△ Less
Submitted 4 October, 2023; v1 submitted 14 June, 2023;
originally announced June 2023.
-
Non-uniform array design for robust LoS MIMO via convex optimization
Authors:
Michail Palaiologos,
Mario H. Castãneda García,
Anastasios Kakkavas,
Richard A. Stirling-Gallacher,
Giuseppe Caire
Abstract:
The array design problem of multiple-input multiple-output (MIMO) systems in a line-of-sight (LoS) transmit environment is examined. As uniform array configurations at the transmitter (Tx) and receiver (Rx) are optimal at specific transmit distances only, they lead to reduced spectral efficiency over a range of transmit distances. To that end, the joint design of nonuniform Tx and Rx arrays toward…
▽ More
The array design problem of multiple-input multiple-output (MIMO) systems in a line-of-sight (LoS) transmit environment is examined. As uniform array configurations at the transmitter (Tx) and receiver (Rx) are optimal at specific transmit distances only, they lead to reduced spectral efficiency over a range of transmit distances. To that end, the joint design of nonuniform Tx and Rx arrays towards maximizing the minimum capacity of a LoS MIMO system across a range of transmit distances is investigated in this paper. By introducing convex relaxation, the joint Tx and Rx array design is cast as a convex optimization problem, which is solved in a iterative manner. In addition, we also implement a local search to obtain a refined solution that achieves an improved performance. It is shown that the non-uniform configurations designed with our proposed approach outperform uniform and non-uniform array designs of the literature in terms of capacity and/or complexity.
△ Less
Submitted 14 February, 2023;
originally announced February 2023.
-
Federated Multilingual Models for Medical Transcript Analysis
Authors:
Andre Manoel,
Mirian Hipolito Garcia,
Tal Baumel,
Shize Su,
Jialei Chen,
Dan Miller,
Danny Karmon,
Robert Sim,
Dimitrios Dimitriadis
Abstract:
Federated Learning (FL) is a novel machine learning approach that allows the model trainer to access more data samples, by training the model across multiple decentralized data sources, while data access constraints are in place. Such trained models can achieve significantly higher performance beyond what can be done when trained on a single data source. As part of FL's promises, none of the train…
▽ More
Federated Learning (FL) is a novel machine learning approach that allows the model trainer to access more data samples, by training the model across multiple decentralized data sources, while data access constraints are in place. Such trained models can achieve significantly higher performance beyond what can be done when trained on a single data source. As part of FL's promises, none of the training data is ever transmitted to any central location, ensuring that sensitive data remains local and private. These characteristics make FL perfectly suited for large-scale applications in healthcare, where a variety of compliance constraints restrict how data may be handled, processed, and stored. Despite the apparent benefits of federated learning, the heterogeneity in the local data distributions pose significant challenges, and such challenges are even more pronounced in the case of multilingual data providers. In this paper we present a federated learning system for training a large-scale multi-lingual model suitable for fine-tuning on downstream tasks such as medical entity tagging. Our work represents one of the first such production-scale systems, capable of training across multiple highly heterogeneous data providers, and achieving levels of accuracy that could not be otherwise achieved by using central training with public data. Finally, we show that the global model performance can be further improved by a training step performed locally.
△ Less
Submitted 3 November, 2022;
originally announced November 2022.
-
Multi-Environment based Meta-Learning with CSI Fingerprints for Radio Based Positioning
Authors:
Anastasios Foliadis,
Mario H. Castañeda Garcia,
Richard A. Stirling-Gallacher,
Reiner S. Thomä
Abstract:
Radio based positioning of a user equipment (UE) based on deep learning (DL) methods using channel state information (CSI) fingerprints have shown promising results. DL models are able to capture complex properties embedded in the CSI about a particular environment and map UE's CSI to the UE's position. However, the CSI fingerprints and the DL models trained on such fingerprints are highly depende…
▽ More
Radio based positioning of a user equipment (UE) based on deep learning (DL) methods using channel state information (CSI) fingerprints have shown promising results. DL models are able to capture complex properties embedded in the CSI about a particular environment and map UE's CSI to the UE's position. However, the CSI fingerprints and the DL models trained on such fingerprints are highly dependent on a particular propagation environment, which generally limits the transfer of knowledge of the DL models from one environment to another. In this paper, we propose a DL model consisting of two parts: the first part aims to learn environment independent features while the second part combines those features depending on the particular environment. To improve transfer learning, we propose a meta learning scheme for training the first part over multiple environments. We show that for positioning in a new environment, initializing a DL model with the meta learned environment independent function achieves higher UE positioning accuracy compared to regular transfer learning from one environment to the new environment, or compared to training the DL model from scratch with only fingerprints from the new environment. Our proposed scheme is able to create an environment independent function which can embed knowledge from multiple environments and more effectively learn from a new environment.
△ Less
Submitted 26 October, 2022;
originally announced October 2022.
-
FLUTE: A Scalable, Extensible Framework for High-Performance Federated Learning Simulations
Authors:
Mirian Hipolito Garcia,
Andre Manoel,
Daniel Madrigal Diaz,
Fatemehsadat Mireshghallah,
Robert Sim,
Dimitrios Dimitriadis
Abstract:
In this paper we introduce "Federated Learning Utilities and Tools for Experimentation" (FLUTE), a high-performance open-source platform for federated learning research and offline simulations. The goal of FLUTE is to enable rapid prototy** and simulation of new federated learning algorithms at scale, including novel optimization, privacy, and communications strategies. We describe the architect…
▽ More
In this paper we introduce "Federated Learning Utilities and Tools for Experimentation" (FLUTE), a high-performance open-source platform for federated learning research and offline simulations. The goal of FLUTE is to enable rapid prototy** and simulation of new federated learning algorithms at scale, including novel optimization, privacy, and communications strategies. We describe the architecture of FLUTE, enabling arbitrary federated modeling schemes to be realized. We compare the platform with other state-of-the-art platforms and describe available features of FLUTE for experimentation in core areas of active research, such as optimization, privacy, and scalability. A comparison with other established platforms shows speed-ups of up to 42x and savings in memory footprint of 3x. A sample of the platform capabilities is also presented for a range of tasks, as well as other functionality, such as linear scaling for the number of participating clients, and a variety of federated optimizers, including FedAdam, DGA, etcetera.
△ Less
Submitted 14 November, 2022; v1 submitted 25 March, 2022;
originally announced March 2022.
-
Reliable Deep Learning based Localization with CSI Fingerprints and Multiple Base Stations
Authors:
Anastasios Foliadis,
Mario H. Castañeda Garcia,
Richard A. Stirling-Gallacher,
Reiner S. Thomä
Abstract:
Deep learning (DL) methods have been recently proposed for user equipment (UE) localization in wireless communication networks, based on the channel state information (CSI) between a UE and each base station (BS) in the uplink. With the CSI from the available BSs, UE localization can be performed in different ways. One the one hand, a single neural network (NN) can be trained for the UE localizati…
▽ More
Deep learning (DL) methods have been recently proposed for user equipment (UE) localization in wireless communication networks, based on the channel state information (CSI) between a UE and each base station (BS) in the uplink. With the CSI from the available BSs, UE localization can be performed in different ways. One the one hand, a single neural network (NN) can be trained for the UE localization by considering the CSI from all the available BSs as one overall fingerprint of the user's location. On the other hand, the CSI at each BS can be used to obtain an estimate of the UE's position with a separate NN at each BS, and then the position estimates of all BSs are combined to obtain an overall estimate of the UE position. In this work, we show that UE localization with the latter approach can achieve a higher positioning accuracy. We propose to consider the uncertainty in the UE localization at each BS, such that overall UE's position is determined by combining the position estimates of the different BSs based on the uncertainty at each BS. With this approach, a more reliable position estimate can be obtained in case of variations in the channel.
△ Less
Submitted 23 November, 2021;
originally announced November 2021.
-
A Tutorial on 5G NR V2X Communications
Authors:
Mario H. Castañeda Garcia,
Alejandro Molina-Galan,
Mate Boban,
Javier Gozalvez,
Baldomero Coll-Perales,
Taylan Şahin,
Apostolos Kousaridas
Abstract:
The Third Generation Partnership Project (3GPP) has recently published its Release 16 that includes the first Vehicle to-Everything (V2X) standard based on the 5G New Radio (NR) air interface. 5G NR V2X introduces advanced functionalities on top of the 5G NR air interface to support connected and automated driving use cases with stringent requirements. This paper presents an in-depth tutorial of t…
▽ More
The Third Generation Partnership Project (3GPP) has recently published its Release 16 that includes the first Vehicle to-Everything (V2X) standard based on the 5G New Radio (NR) air interface. 5G NR V2X introduces advanced functionalities on top of the 5G NR air interface to support connected and automated driving use cases with stringent requirements. This paper presents an in-depth tutorial of the 3GPP Release 16 5G NR V2X standard for V2X communications, with a particular focus on the sidelink, since it is the most significant part of 5G NR V2X. The main part of the paper is an in-depth treatment of the key aspects of 5G NR V2X: the physical layer, the resource allocation, the quality of service management, the enhancements introduced to the Uu interface and the mobility management for V2N (Vehicle to Network) communications, as well as the co-existence mechanisms between 5G NR V2X and LTE V2X. We also review the use cases, the system architecture, and describe the evaluation methodology and simulation assumptions for 5G NR V2X. Finally, we provide an outlook on possible 5G NR V2X enhancements, including those identified within Release 17.
△ Less
Submitted 8 February, 2021;
originally announced February 2021.
-
CSI-Based Localization with CNNs Exploiting Phase Information
Authors:
Anastasios Foliadis,
Mario H. Castañeda Garcia,
Richard A. Stirling-Gallacher,
Reiner S. Thomä
Abstract:
In this paper we study the use of the Channel State Information (CSI) as fingerprint inputs of a Convolutional Neural Network (CNN) for localization. We examine whether the CSI can be used as a distinct fingerprint corresponding to a single position by considering the inconsistencies with its raw phase that cause the CSI to be unreliable. We propose two methods to produce reliable fingerprints inc…
▽ More
In this paper we study the use of the Channel State Information (CSI) as fingerprint inputs of a Convolutional Neural Network (CNN) for localization. We examine whether the CSI can be used as a distinct fingerprint corresponding to a single position by considering the inconsistencies with its raw phase that cause the CSI to be unreliable. We propose two methods to produce reliable fingerprints including the phase information. Furthermore, we examine the structure of the CNN and more specifically the impact of pooling on the positioning performance, and show that pooling over the subcarriers can be more beneficial than over the antennas.
△ Less
Submitted 22 January, 2021;
originally announced January 2021.
-
Position Information from Single-Bounce Reflections
Authors:
Anastasios Kakkavas,
Mario H. Castañeda García,
Gonzalo Seco-Granados,
Henk Wymeersch,
Richard A. Stirling-Gallacher,
Josef A. Nossek
Abstract:
In the context of positioning a target with a single-anchor, this contribution focuses on the Fisher information about the position, orientation and clock offset of the target provided by single-bounce reflections. The availability of prior knowledge of the target's environment is taken into account via a prior distribution of the position of virtual anchors, and the rank, intensity and direction…
▽ More
In the context of positioning a target with a single-anchor, this contribution focuses on the Fisher information about the position, orientation and clock offset of the target provided by single-bounce reflections. The availability of prior knowledge of the target's environment is taken into account via a prior distribution of the position of virtual anchors, and the rank, intensity and direction of provided information is studied. We show that when no prior knowledge is available, single-bounce reflections offer position information in the direction parallel to the reflecting surface, irrespective of the target's and anchor's locations. We provide a geometrically intuitive explanation of the results and present numerical examples demonstrating their potential implications.
△ Less
Submitted 2 December, 2020;
originally announced December 2020.
-
Power Allocation and Parameter Estimation for Multipath-based 5G Positioning
Authors:
Anastasios Kakkavas,
Henk Wymeersch,
Gonzalo Seco-Granados,
Mario H. Castañeda García,
Richard A. Stirling-Gallacher,
Josef A. Nossek
Abstract:
We consider a single-anchor multiple-input multiple-output (MIMO) orthogonal frequency-division multiplexing (OFDM) system with imperfectly synchronized transmitter (Tx) and receiver (Rx) clocks, where the Rx estimates its position based on the received reference signals. The Tx, having (imperfect) prior knowledge about the Rx location and the surrounding geometry, transmits the reference signals…
▽ More
We consider a single-anchor multiple-input multiple-output (MIMO) orthogonal frequency-division multiplexing (OFDM) system with imperfectly synchronized transmitter (Tx) and receiver (Rx) clocks, where the Rx estimates its position based on the received reference signals. The Tx, having (imperfect) prior knowledge about the Rx location and the surrounding geometry, transmits the reference signals based on a set of fixed beams. In this work, we develop strategies for the power allocation among the beams aiming to minimize the expected Cramér-Rao lower bound (CRLB) for Rx positioning. Additional constraints on the design are included to ensure that the line-of-sight (LOS) path is detected with high probability. Furthermore, the effect of clock asynchronism on the resulting allocation strategies is also studied. We also propose a gridless compressed sensing-based position estimation algorithm, which exploits the information on the clock offset provided by non-line-of-sight paths, and show that it is asymptotically efficient.
△ Less
Submitted 2 December, 2020;
originally announced December 2020.
-
A new consistent Neutron Star Equation of State from a Generalized Skyrme model
Authors:
Christoph Adam,
Alberto García Martín-Caro,
Miguel Huidobro García,
Ricardo Vázquez,
Andrzej Wereszczynski
Abstract:
We propose a new equation of state for nuclear matter based on a generalized Skyrme model which is consistent with all current constraints on the observed properties of neutron stars. This generalized model depends only on two free parameters related to the ranges of pressure values at which different submodels are dominant, and which can be adjusted so that mass-radius and deformability constrain…
▽ More
We propose a new equation of state for nuclear matter based on a generalized Skyrme model which is consistent with all current constraints on the observed properties of neutron stars. This generalized model depends only on two free parameters related to the ranges of pressure values at which different submodels are dominant, and which can be adjusted so that mass-radius and deformability constraints from astrophysical and gravitational wave measurements can be met. Our results support the Skyrme model and its generalizations as good candidates for a low energy effective field-theoretic description of nuclear matter even at extreme conditions such as those inside neutron stars.
△ Less
Submitted 17 November, 2020; v1 submitted 14 June, 2020;
originally announced June 2020.
-
Cooperative Localization with Angular Measurements and Posterior Linearization
Authors:
Yibo Wu,
Bile Peng,
Henk Wymeersch,
Gonzalo Seco-Granados,
Anastasios Kakkavas,
Mario H. Castañeda Garcia,
Richard A. Stirling-Gallacher
Abstract:
The application of cooperative localization in vehicular networks is attractive to improve accuracy and coverage. Conventional distance measurements between vehicles are limited by the need for synchronization and provide no heading information of the vehicle. To address this, we present a cooperative localization algorithm using posterior linearization belief propagation (PLBP) utilizing angle-of…
▽ More
The application of cooperative localization in vehicular networks is attractive to improve accuracy and coverage. Conventional distance measurements between vehicles are limited by the need for synchronization and provide no heading information of the vehicle. To address this, we present a cooperative localization algorithm using posterior linearization belief propagation (PLBP) utilizing angle-of-arrival (AoA)-only measurements. Simulation results show that both directional and positional root mean squared error (RMSE) of vehicles can be decreased significantly and converge to a low value in a few iterations. Furthermore, the influence of parameters for the vehicular network, such as vehicle density, communication radius, prior uncertainty and AoA measurements noise, is analyzed.
△ Less
Submitted 10 July, 2019;
originally announced July 2019.
-
5G Downlink Multi-Beam Signal Design for LOS Positioning
Authors:
Anastasios Kakkavas,
Gonzalo Seco-Granados,
Henk Wymeersch,
Mario H. Castañeda García,
Richard A. Stirling-Gallacher,
Josef A. Nossek
Abstract:
In this work, we study optimal transmit strategies for minimizing the positioning error bound in a line-of-sight scenario, under different levels of prior knowledge of the channel parameters. For the case of perfect prior knowledge, we prove that two beams are optimal, and determine their beam directions and optimal power allocation. For the imperfect prior knowledge case, we compute the optimal p…
▽ More
In this work, we study optimal transmit strategies for minimizing the positioning error bound in a line-of-sight scenario, under different levels of prior knowledge of the channel parameters. For the case of perfect prior knowledge, we prove that two beams are optimal, and determine their beam directions and optimal power allocation. For the imperfect prior knowledge case, we compute the optimal power allocation among the beams of a codebook for two different robustness-related objectives, namely average or maximum squared position error bound minimization. Our numerical results show that our low-complexity approach can outperform existing methods that entail higher signaling and computational overhead.
△ Less
Submitted 2 December, 2020; v1 submitted 4 June, 2019;
originally announced June 2019.
-
LOS MIMO Design based on Multiple Optimum Antenna Separations
Authors:
Mario H. Castañeda García,
Marcin Iwanow,
Richard A. Stirling-Gallacher
Abstract:
The use of multiple antennas in a transmit and receive antenna array for MIMO wireless communication allows the spatial degrees of freedom in rich scattering environments to be exploited. However, for line-of-sight (LOS) MIMO channels with uniform linear arrays (ULAs) at the transmitter and receiver, the antenna separations at the transmit and receive array need to be optimized to maximize the spa…
▽ More
The use of multiple antennas in a transmit and receive antenna array for MIMO wireless communication allows the spatial degrees of freedom in rich scattering environments to be exploited. However, for line-of-sight (LOS) MIMO channels with uniform linear arrays (ULAs) at the transmitter and receiver, the antenna separations at the transmit and receive array need to be optimized to maximize the spatial degrees of freedom and the channel capacity. In this paper, we first revisit the derivation of the optimum antenna separation at the transmit and receive ULAs in a LOS MIMO system, and provide the general expression for the optimum antenna separation product, which consists of multiple solutions. Although only the solution corresponding to the smallest antenna separation product is usually considered in the literature, we exploit the multiple solutions for a LOS MIMO design over a range of distances between the transmitter and receiver. In particular, we consider the LOS MIMO design in a vehicle-to-vehicle (V2V) communication scenario, over a range of distances between the transmit and receive vehicle.
△ Less
Submitted 11 September, 2018;
originally announced September 2018.
-
Performance Limits of Single-Anchor Millimeter-Wave Positioning
Authors:
Anastasios Kakkavas,
Mario H. Castañeda García,
Richard A. Strirling-Gallacher,
Josef A. Nossek
Abstract:
The fundamental limits of single-anchor multi-antenna positioning are investigated. Exploiting the structure of the multiple input-multiple output-orthogonal frequency division multiplexing (MIMO-OFDM) channel at millimeter-wave frequencies, we present geometrically intuitive asymptotic expressions for the Fisher information on position, orientation and velocity for large bandwidth and number of a…
▽ More
The fundamental limits of single-anchor multi-antenna positioning are investigated. Exploiting the structure of the multiple input-multiple output-orthogonal frequency division multiplexing (MIMO-OFDM) channel at millimeter-wave frequencies, we present geometrically intuitive asymptotic expressions for the Fisher information on position, orientation and velocity for large bandwidth and number of antennas. The effects of synchronization errors and mobility are studied and it is shown that non-line-of-sight (NLOS) paths can be used to estimate the synchronization error and drastically improve the positioning performance. We also find that, in the presence of line-of-sight (LOS), mobility has a small impact on the achievable positioning accuracy, but in the NLOS-only scenario it can significantly improve the achievable performance, depending on the variance of the synchronization error. Finally, considering a communication system with device-specific transmission and reception constraints, we compare the positioning accuracy between the downlink and the uplink and show that they are equivalent under the same signal-to-noise ratio (SNR).
△ Less
Submitted 2 December, 2020; v1 submitted 24 August, 2018;
originally announced August 2018.
-
Multi-Array 5G V2V Relative Positioning: Performance Bounds
Authors:
Anastasios Kakkavas,
Mario H. Castañeda García,
Richard A. Stirling-Gallacher,
Josef A. Nossek
Abstract:
We study the performance bounds of vehicle-to-vehicle (V2V) relative positioning for vehicles with multiple antenna arrays. The Cramér-Rao bound for the estimation of the relative position and the orientation of the Tx vehicle is derived, when angle of arrival (AOA) measurements with or without time-difference of arrival (TDOA) measurements are used. In addition, geometrically intuitive expression…
▽ More
We study the performance bounds of vehicle-to-vehicle (V2V) relative positioning for vehicles with multiple antenna arrays. The Cramér-Rao bound for the estimation of the relative position and the orientation of the Tx vehicle is derived, when angle of arrival (AOA) measurements with or without time-difference of arrival (TDOA) measurements are used. In addition, geometrically intuitive expressions for the corresponding Fisher information are provided. The derived bounds are numerically evaluated for different carrier frequencies, bandwidths and array configurations under different V2V scenarios, i.e. overtaking and platooning. The significance of the AOA and TDOA measurements for position estimation is investigated. The achievable positioning accuracy is then compared with the present requirements of the 3rd Generation Partnership Project (3GPP) 5G New Radio (NR) vehicle-to-everything (V2X) standardization.
△ Less
Submitted 4 June, 2019; v1 submitted 22 August, 2018;
originally announced August 2018.
