-
Federated Knowledge Transfer Fine-tuning Large Server Model with Resource-Constrained IoT Clients
Authors:
Shaoyuan Chen,
Linlin You,
Rui Liu,
Shuo Yu,
Ahmed M. Abdelmoniem
Abstract:
The training of large models, involving fine-tuning, faces the scarcity of high-quality data. Compared to the solutions based on centralized data centers, updating large models in the Internet of Things (IoT) faces challenges in coordinating knowledge from distributed clients by using their private and heterogeneous data. To tackle such a challenge, we propose KOALA (Federated Knowledge Transfer F…
▽ More
The training of large models, involving fine-tuning, faces the scarcity of high-quality data. Compared to the solutions based on centralized data centers, updating large models in the Internet of Things (IoT) faces challenges in coordinating knowledge from distributed clients by using their private and heterogeneous data. To tackle such a challenge, we propose KOALA (Federated Knowledge Transfer Fine-tuning Large Server Model with Resource-Constrained IoT Clients) to impel the training of large models in IoT. Since the resources obtained by IoT clients are limited and restricted, it is infeasible to locally execute large models and also update them in a privacy-preserving manner. Therefore, we leverage federated learning and knowledge distillation to update large models through collaboration with their small models, which can run locally at IoT clients to process their private data separately and enable large-small model knowledge transfer through iterative learning between the server and clients. Moreover, to support clients with similar or different computing capacities, KOALA is designed with two kinds of large-small model joint learning modes, namely to be homogeneous or heterogeneous. Experimental results demonstrate that compared to the conventional approach, our method can not only achieve similar training performance but also significantly reduce the need for local storage and computing power resources.
△ Less
Submitted 7 July, 2024;
originally announced July 2024.
-
Decentralised Moderation for Interoperable Social Networks: A Conversation-based Approach for Pleroma and the Fediverse
Authors:
Vibhor Agarwal,
Aravindh Raman,
Nishanth Sastry,
Ahmed M. Abdelmoniem,
Gareth Tyson,
Ignacio Castro
Abstract:
The recent development of decentralised and interoperable social networks (such as the "fediverse") creates new challenges for content moderators. This is because millions of posts generated on one server can easily "spread" to another, even if the recipient server has very different moderation policies. An obvious solution would be to leverage moderation tools to automatically tag (and filter) po…
▽ More
The recent development of decentralised and interoperable social networks (such as the "fediverse") creates new challenges for content moderators. This is because millions of posts generated on one server can easily "spread" to another, even if the recipient server has very different moderation policies. An obvious solution would be to leverage moderation tools to automatically tag (and filter) posts that contravene moderation policies, e.g. related to toxic speech. Recent work has exploited the conversational context of a post to improve this automatic tagging, e.g. using the replies to a post to help classify if it contains toxic speech. This has shown particular potential in environments with large training sets that contain complete conversations. This, however, creates challenges in a decentralised context, as a single conversation may be fragmented across multiple servers. Thus, each server only has a partial view of an entire conversation because conversations are often federated across servers in a non-synchronized fashion. To address this, we propose a decentralised conversation-aware content moderation approach suitable for the fediverse. Our approach employs a graph deep learning model (GraphNLI) trained locally on each server. The model exploits local data to train a model that combines post and conversational information captured through random walks to detect toxicity. We evaluate our approach with data from Pleroma, a major decentralised and interoperable micro-blogging network containing 2 million conversations. Our model effectively detects toxicity on larger instances, exclusively trained using their local post information (0.8837 macro-F1). Our approach has considerable scope to improve moderation in decentralised and interoperable social networks such as Pleroma or Mastodon.
△ Less
Submitted 16 April, 2024; v1 submitted 3 April, 2024;
originally announced April 2024.
-
Flashback: Understanding and Mitigating Forgetting in Federated Learning
Authors:
Mohammed Aljahdali,
Ahmed M. Abdelmoniem,
Marco Canini,
Samuel Horváth
Abstract:
In Federated Learning (FL), forgetting, or the loss of knowledge across rounds, hampers algorithm convergence, particularly in the presence of severe data heterogeneity among clients. This study explores the nuances of this issue, emphasizing the critical role of forgetting in FL's inefficient learning within heterogeneous data contexts. Knowledge loss occurs in both client-local updates and serve…
▽ More
In Federated Learning (FL), forgetting, or the loss of knowledge across rounds, hampers algorithm convergence, particularly in the presence of severe data heterogeneity among clients. This study explores the nuances of this issue, emphasizing the critical role of forgetting in FL's inefficient learning within heterogeneous data contexts. Knowledge loss occurs in both client-local updates and server-side aggregation steps; addressing one without the other fails to mitigate forgetting. We introduce a metric to measure forgetting granularly, ensuring distinct recognition amid new knowledge acquisition. Leveraging these insights, we propose Flashback, an FL algorithm with a dynamic distillation approach that is used to regularize the local models, and effectively aggregate their knowledge. Across different benchmarks, Flashback outperforms other methods, mitigates forgetting, and achieves faster round-to-target-accuracy, by converging in 6 to 16 rounds.
△ Less
Submitted 8 February, 2024;
originally announced February 2024.
-
FairQ: Fair and Fast Rate Allocation in Data Centers
Authors:
Ahmed M. Abdelmoniem,
Brahim Bensaou
Abstract:
The peculiar congestion patterns in data centers are caused by the bursty and composite nature of traffic, the small bandwidth-delay product, and the tiny switch buffers. It is not practical to modify TCP to adapt to data centers, especially in public clouds where multiple congestion control protocols coexist. In this work, we design a switch-based method to address such congestion issues; our app…
▽ More
The peculiar congestion patterns in data centers are caused by the bursty and composite nature of traffic, the small bandwidth-delay product, and the tiny switch buffers. It is not practical to modify TCP to adapt to data centers, especially in public clouds where multiple congestion control protocols coexist. In this work, we design a switch-based method to address such congestion issues; our approach does not require any modification to TCP, which enables easy and seamless deployment in public data centers via switch software update. We first present a simple analysis to demonstrate the stability and effectiveness of the scheme, and then we discuss a hardware NetFPGA switch-based prototype. The experimental results from real deployments in a small testbed cluster show the effectiveness of our approach.
△ Less
Submitted 9 January, 2024;
originally announced January 2024.
-
An Empirical Study of Efficiency and Privacy of Federated Learning Algorithms
Authors:
Sofia Zahri,
Hajar Bennouri,
Ahmed M. Abdelmoniem
Abstract:
In today's world, the rapid expansion of IoT networks and the proliferation of smart devices in our daily lives, have resulted in the generation of substantial amounts of heterogeneous data. These data forms a stream which requires special handling. To handle this data effectively, advanced data processing technologies are necessary to guarantee the preservation of both privacy and efficiency. Fed…
▽ More
In today's world, the rapid expansion of IoT networks and the proliferation of smart devices in our daily lives, have resulted in the generation of substantial amounts of heterogeneous data. These data forms a stream which requires special handling. To handle this data effectively, advanced data processing technologies are necessary to guarantee the preservation of both privacy and efficiency. Federated learning emerged as a distributed learning method that trains models locally and aggregates them on a server to preserve data privacy. This paper showcases two illustrative scenarios that highlight the potential of federated learning (FL) as a key to delivering efficient and privacy-preserving machine learning within IoT networks. We first give the mathematical foundations for key aggregation algorithms in federated learning, i.e., FedAvg and FedProx. Then, we conduct simulations, using Flower Framework, to show the \textit{efficiency} of these algorithms by training deep neural networks on common datasets and show a comparison between the accuracy and loss metrics of FedAvg and FedProx. Then, we present the results highlighting the trade-off between maintaining privacy versus accuracy via simulations - involving the implementation of the differential privacy (DP) method - in Pytorch and Opacus ML frameworks on common FL datasets and data distributions for both FedAvg and FedProx strategies.
△ Less
Submitted 23 December, 2023;
originally announced December 2023.
-
A Meta-learning based Stacked Regression Approach for Customer Lifetime Value Prediction
Authors:
Karan Gadgil,
Sukhpal Singh Gill,
Ahmed M. Abdelmoniem
Abstract:
Companies across the globe are keen on targeting potential high-value customers in an attempt to expand revenue and this could be achieved only by understanding the customers more. Customer Lifetime Value (CLV) is the total monetary value of transactions/purchases made by a customer with the business over an intended period of time and is used as means to estimate future customer interactions. CLV…
▽ More
Companies across the globe are keen on targeting potential high-value customers in an attempt to expand revenue and this could be achieved only by understanding the customers more. Customer Lifetime Value (CLV) is the total monetary value of transactions/purchases made by a customer with the business over an intended period of time and is used as means to estimate future customer interactions. CLV finds application in a number of distinct business domains such as Banking, Insurance, Online-entertainment, Gaming, and E-Commerce. The existing distribution-based and basic (recency, frequency & monetary) based models face a limitation in terms of handling a wide variety of input features. Moreover, the more advanced Deep learning approaches could be superfluous and add an undesirable element of complexity in certain application areas. We, therefore, propose a system which is able to qualify both as effective, and comprehensive yet simple and interpretable. With that in mind, we develop a meta-learning-based stacked regression model which combines the predictions from bagging and boosting models that each is found to perform well individually. Empirical tests have been carried out on an openly available Online Retail dataset to evaluate various models and show the efficacy of the proposed approach.
△ Less
Submitted 7 August, 2023;
originally announced August 2023.
-
Stock Market Price Prediction: A Hybrid LSTM and Sequential Self-Attention based Approach
Authors:
Karan Pardeshi,
Sukhpal Singh Gill,
Ahmed M. Abdelmoniem
Abstract:
One of the most enticing research areas is the stock market, and projecting stock prices may help investors profit by making the best decisions at the correct time. Deep learning strategies have emerged as a critical technique in the field of the financial market. The stock market is impacted due to two aspects, one is the geo-political, social and global events on the bases of which the price tre…
▽ More
One of the most enticing research areas is the stock market, and projecting stock prices may help investors profit by making the best decisions at the correct time. Deep learning strategies have emerged as a critical technique in the field of the financial market. The stock market is impacted due to two aspects, one is the geo-political, social and global events on the bases of which the price trends could be affected. Meanwhile, the second aspect purely focuses on historical price trends and seasonality, allowing us to forecast stock prices. In this paper, our aim is to focus on the second aspect and build a model that predicts future prices with minimal errors. In order to provide better prediction results of stock price, we propose a new model named Long Short-Term Memory (LSTM) with Sequential Self-Attention Mechanism (LSTM-SSAM). Finally, we conduct extensive experiments on the three stock datasets: SBIN, HDFCBANK, and BANKBARODA. The experimental results prove the effectiveness and feasibility of the proposed model compared to existing models. The experimental findings demonstrate that the root-mean-squared error (RMSE), and R-square (R2) evaluation indicators are giving the best results.
△ Less
Submitted 7 August, 2023;
originally announced August 2023.
-
Fortaleza: The emergence of a network hub
Authors:
Eric Bragion,
Habiba Akter,
Mohit Kumar,
Minxian Xu,
Ahmed M. Abdelmoniem,
Sukhpal Singh Gill
Abstract:
Digitalisation, accelerated by the pandemic, has brought the opportunity for companies to expand their businesses beyond their geographic location and has considerably affected networks around the world. Cloud services have a better acceptance nowadays, and it is foreseen that this industry will grow exponentially in the following years. With more distributed networks that need to support customer…
▽ More
Digitalisation, accelerated by the pandemic, has brought the opportunity for companies to expand their businesses beyond their geographic location and has considerably affected networks around the world. Cloud services have a better acceptance nowadays, and it is foreseen that this industry will grow exponentially in the following years. With more distributed networks that need to support customers in different locations, the model of one-single server in big financial centres has become outdated and companies tend to look for alternatives that will meet their needs, and this seems to be the case with Fortaleza, in Brazil. With several submarine cables connections available, the city has stood out as a possible hub to different regions, and this is what this paper explores. Making use of real traffic data through looking glasses, we established a latency classification that ranges from exceptionally low to high and analysed 800 latencies from Roubaix, Fortaleza and Sao Paulo to Miami, Mexico City, Frankfurt, Paris, Milan, Prague, Sao Paulo, Santiago, Buenos Aires and Luanda. We found that non-developed countries have a big dependence on the United States to route Internet traffic. Despite this, Fortaleza proves to be an alternative for serving different regions with relatively low latencies.
△ Less
Submitted 28 June, 2023;
originally announced July 2023.
-
Leveraging The Edge-to-Cloud Continuum for Scalable Machine Learning on Decentralized Data
Authors:
Ahmed M. Abdelmoniem
Abstract:
With mobile, IoT and sensor devices becoming pervasive in our life and recent advances in Edge Computational Intelligence (e.g., Edge AI/ML), it became evident that the traditional methods for training AI/ML models are becoming obsolete, especially with the growing concerns over privacy and security. This work tries to highlight the key challenges that prohibit Edge AI/ML from seeing wide-range ad…
▽ More
With mobile, IoT and sensor devices becoming pervasive in our life and recent advances in Edge Computational Intelligence (e.g., Edge AI/ML), it became evident that the traditional methods for training AI/ML models are becoming obsolete, especially with the growing concerns over privacy and security. This work tries to highlight the key challenges that prohibit Edge AI/ML from seeing wide-range adoption in different sectors, especially for large-scale scenarios. Therefore, we focus on the main challenges acting as adoption barriers for the existing methods and propose a design with a drastic shift from the current ill-suited approaches. The new design is envisioned to be model-centric in which the trained models are treated as a commodity driving the exchange dynamics of collaborative learning in decentralized settings. It is expected that this design will provide a decentralized framework for efficient collaborative learning at scale.
△ Less
Submitted 19 June, 2023;
originally announced June 2023.
-
AI-based Fog and Edge Computing: A Systematic Review, Taxonomy and Future Directions
Authors:
Sundas Iftikhar,
Sukhpal Singh Gill,
Chenghao Song,
Minxian Xu,
Mohammad Sadegh Aslanpour,
Adel N. Toosi,
Junhui Du,
Huaming Wu,
Shreya Ghosh,
Deepraj Chowdhury,
Muhammed Golec,
Mohit Kumar,
Ahmed M. Abdelmoniem,
Felix Cuadrado,
Blesson Varghese,
Omer Rana,
Schahram Dustdar,
Steve Uhlig
Abstract:
Resource management in computing is a very challenging problem that involves making sequential decisions. Resource limitations, resource heterogeneity, dynamic and diverse nature of workload, and the unpredictability of fog/edge computing environments have made resource management even more challenging to be considered in the fog landscape. Recently Artificial Intelligence (AI) and Machine Learnin…
▽ More
Resource management in computing is a very challenging problem that involves making sequential decisions. Resource limitations, resource heterogeneity, dynamic and diverse nature of workload, and the unpredictability of fog/edge computing environments have made resource management even more challenging to be considered in the fog landscape. Recently Artificial Intelligence (AI) and Machine Learning (ML) based solutions are adopted to solve this problem. AI/ML methods with the capability to make sequential decisions like reinforcement learning seem most promising for these type of problems. But these algorithms come with their own challenges such as high variance, explainability, and online training. The continuously changing fog/edge environment dynamics require solutions that learn online, adopting changing computing environment. In this paper, we used standard review methodology to conduct this Systematic Literature Review (SLR) to analyze the role of AI/ML algorithms and the challenges in the applicability of these algorithms for resource management in fog/edge computing environments. Further, various machine learning, deep learning and reinforcement learning techniques for edge AI management have been discussed. Furthermore, we have presented the background and current status of AI/ML-based Fog/Edge Computing. Moreover, a taxonomy of AI/ML-based resource management techniques for fog/edge computing has been proposed and compared the existing techniques based on the proposed taxonomy. Finally, open challenges and promising future research directions have been identified and discussed in the area of AI/ML-based fog/edge computing.
△ Less
Submitted 8 December, 2022;
originally announced December 2022.
-
The Switch from Conventional to SDN: The Case for Transport-Agnostic Congestion Control
Authors:
Ahmed M. Abdelmoniem,
Brahim Bensaou
Abstract:
To meet the timing requirements of interactive applications, the no-frills congestion-agnostic transport protocols like UDP are increasingly deployed side-by-side in the same network with congestion-responsive TCP. In cloud platforms, even though the computation and storage is totally virtualized, they lack a true virtualization mechanism for the network (i.e., the underlying data centers networks…
▽ More
To meet the timing requirements of interactive applications, the no-frills congestion-agnostic transport protocols like UDP are increasingly deployed side-by-side in the same network with congestion-responsive TCP. In cloud platforms, even though the computation and storage is totally virtualized, they lack a true virtualization mechanism for the network (i.e., the underlying data centers networks). The impact of such lack of isolation services, may result into frequent outages (for some applications) when such diverse traffics contend for the small buffers in the commodity switches used in data centers. In this paper, we explore the design space of a simple, practical and transport-agnostic scheme to enable a scalable and flexible end-to-end congestion control in data centers. Then, we present the the shortcomings of coupling the monitoring and control of congestion in the conventional system and discuss how a Software-Defined Network (SDN) would provide an appealing alternative to circumvent the problems of the conventional system. The two systems implements a software-based congestion control mechanisms that perform monitoring, control decisions and traffic control enforcement functions. Both systems are designed with a major assumption that the applications (or transport protocols) are non-cooperative with the system, ultimately making it deployable in existing data centers without any service disruption or hardware upgrade. Both systems are implemented and evaluated via simulation in NS2 as well as real-life small-scale test-bed deployment and experiments.
△ Less
Submitted 10 September, 2022;
originally announced September 2022.
-
Towards Energy-Aware Federated Learning on Battery-Powered Clients
Authors:
Amna Arouj,
Ahmed M. Abdelmoniem
Abstract:
Federated learning (FL) is a newly emerged branch of AI that facilitates edge devices to collaboratively train a global machine learning model without centralizing data and with privacy by default. However, despite the remarkable advancement, this paradigm comes with various challenges. Specifically, in large-scale deployments, client heterogeneity is the norm which impacts training quality such a…
▽ More
Federated learning (FL) is a newly emerged branch of AI that facilitates edge devices to collaboratively train a global machine learning model without centralizing data and with privacy by default. However, despite the remarkable advancement, this paradigm comes with various challenges. Specifically, in large-scale deployments, client heterogeneity is the norm which impacts training quality such as accuracy, fairness, and time. Moreover, energy consumption across these battery-constrained devices is largely unexplored and a limitation for wide-adoption of FL. To address this issue, we develop EAFL, an energy-aware FL selection method that considers energy consumption to maximize the participation of heterogeneous target devices. EAFL is a power-aware training algorithm that cherry-picks clients with higher battery levels in conjunction with its ability to maximize the system efficiency. Our design jointly minimizes the time-to-accuracy and maximizes the remaining on-device battery levels. EAFLimproves the testing model accuracy by up to 85\% and decreases the drop-out of clients by up to 2.45$\times$.
△ Less
Submitted 24 August, 2022; v1 submitted 8 August, 2022;
originally announced August 2022.
-
Resource-Efficient Federated Learning
Authors:
Ahmed M. Abdelmoniem,
Atal Narayan Sahu,
Marco Canini,
Suhaib A. Fahmy
Abstract:
Federated Learning (FL) enables distributed training by learners using local data, thereby enhancing privacy and reducing communication. However, it presents numerous challenges relating to the heterogeneity of the data distribution, device capabilities, and participant availability as deployments scale, which can impact both model convergence and bias. Existing FL schemes use random participant s…
▽ More
Federated Learning (FL) enables distributed training by learners using local data, thereby enhancing privacy and reducing communication. However, it presents numerous challenges relating to the heterogeneity of the data distribution, device capabilities, and participant availability as deployments scale, which can impact both model convergence and bias. Existing FL schemes use random participant selection to improve fairness; however, this can result in inefficient use of resources and lower quality training. In this work, we systematically address the question of resource efficiency in FL, showing the benefits of intelligent participant selection, and incorporation of updates from straggling participants. We demonstrate how these factors enable resource efficiency while also improving trained model quality.
△ Less
Submitted 4 November, 2022; v1 submitted 1 November, 2021;
originally announced November 2021.
-
Rethinking gradient sparsification as total error minimization
Authors:
Atal Narayan Sahu,
Aritra Dutta,
Ahmed M. Abdelmoniem,
Trambak Banerjee,
Marco Canini,
Panos Kalnis
Abstract:
Gradient compression is a widely-established remedy to tackle the communication bottleneck in distributed training of large deep neural networks (DNNs). Under the error-feedback framework, Top-$k$ sparsification, sometimes with $k$ as little as $0.1\%$ of the gradient size, enables training to the same model quality as the uncompressed case for a similar iteration count. From the optimization pers…
▽ More
Gradient compression is a widely-established remedy to tackle the communication bottleneck in distributed training of large deep neural networks (DNNs). Under the error-feedback framework, Top-$k$ sparsification, sometimes with $k$ as little as $0.1\%$ of the gradient size, enables training to the same model quality as the uncompressed case for a similar iteration count. From the optimization perspective, we find that Top-$k$ is the communication-optimal sparsifier given a per-iteration $k$ element budget. We argue that to further the benefits of gradient sparsification, especially for DNNs, a different perspective is necessary -- one that moves from per-iteration optimality to consider optimality for the entire training.
We identify that the total error -- the sum of the compression errors for all iterations -- encapsulates sparsification throughout training. Then, we propose a communication complexity model that minimizes the total error under a communication budget for the entire training. We find that the hard-threshold sparsifier, a variant of the Top-$k$ sparsifier with $k$ determined by a constant hard-threshold, is the optimal sparsifier for this model. Motivated by this, we provide convex and non-convex convergence analyses for the hard-threshold sparsifier with error-feedback. Unlike with Top-$k$ sparsifier, we show that hard-threshold has the same asymptotic convergence and linear speedup property as SGD in the convex case and has no impact on the data-heterogeneity in the non-convex case. Our diverse experiments on various DNNs and a logistic regression model demonstrated that the hard-threshold sparsifier is more communication-efficient than Top-$k$.
△ Less
Submitted 2 August, 2021;
originally announced August 2021.
-
Implementation and Evaluation of Data Center Congestion Controller with Switch Assistance
Authors:
Ahmed M. Abdelmoniem,
Brahim Bensaou
Abstract:
In this work, we provide the design and implementation of a switch-assisted congestion control algorithm for data center networks (DCNs). In particular, we provide a prototype of the switch-driven congestion control algorithm and deploy it in a real data center. The prototype is based on few simple modifications to the switch software. The modifications imposed by the algorithm on the switch are t…
▽ More
In this work, we provide the design and implementation of a switch-assisted congestion control algorithm for data center networks (DCNs). In particular, we provide a prototype of the switch-driven congestion control algorithm and deploy it in a real data center. The prototype is based on few simple modifications to the switch software. The modifications imposed by the algorithm on the switch are to enable the switch to modify the TCP receive-window field in the packet headers. By doing so, the algorithm can enforce a pre-calculated (or target rate) to limit the sending rate at the sources. Therefore, the algorithm requires no modifications to the TCP source or receiver code which considered out of the DCN operators' control (e.g., in the public cloud where the VM is maintained by the tenant). This paper describes in detail two implementations, one as a Linux kernel module and the second as an added feature to the well-known software switch, Open vSwitch. Then we present evaluation results based on experiments of the deployment of both designs in a small testbed to demonstrate the effectiveness of the proposed technique in achieving high throughput, good fairness, and short flow completion times for delay-sensitive flows.
△ Less
Submitted 26 June, 2021;
originally announced June 2021.
-
On the Impact of Device and Behavioral Heterogeneity in Federated Learning
Authors:
Ahmed M. Abdelmoniem,
Chen-Yu Ho,
Pantelis Papageorgiou,
Muhammad Bilal,
Marco Canini
Abstract:
Federated learning (FL) is becoming a popular paradigm for collaborative learning over distributed, private datasets owned by non-trusting entities. FL has seen successful deployment in production environments, and it has been adopted in services such as virtual keyboards, auto-completion, item recommendation, and several IoT applications. However, FL comes with the challenge of performing trainin…
▽ More
Federated learning (FL) is becoming a popular paradigm for collaborative learning over distributed, private datasets owned by non-trusting entities. FL has seen successful deployment in production environments, and it has been adopted in services such as virtual keyboards, auto-completion, item recommendation, and several IoT applications. However, FL comes with the challenge of performing training over largely heterogeneous datasets, devices, and networks that are out of the control of the centralized FL server. Motivated by this inherent setting, we make a first step towards characterizing the impact of device and behavioral heterogeneity on the trained model. We conduct an extensive empirical study spanning close to 1.5K unique configurations on five popular FL benchmarks. Our analysis shows that these sources of heterogeneity have a major impact on both model performance and fairness, thus sheds light on the importance of considering heterogeneity in FL system design.
△ Less
Submitted 15 February, 2021;
originally announced February 2021.
-
T-RACKs: A Faster Recovery Mechanism for TCP in Data Center Networks
Authors:
Ahmed M. Abdelmoniem,
Brahim Bensaou
Abstract:
Cloud interactive data-driven applications generate swarms of small TCP flows that compete for the small buffer space in data-center switches. Such applications require a short flow completion time (FCT) to perform their jobs effectively. However, TCP is oblivious to the composite nature of application data and artificially inflates the FCT of such flows by several orders of magnitude. This is due…
▽ More
Cloud interactive data-driven applications generate swarms of small TCP flows that compete for the small buffer space in data-center switches. Such applications require a short flow completion time (FCT) to perform their jobs effectively. However, TCP is oblivious to the composite nature of application data and artificially inflates the FCT of such flows by several orders of magnitude. This is due to TCP's Internet-centric design that fixes the retransmission timeout (RTO) to be at least hundreds of milliseconds. To better understand this problem, in this paper, we use empirical measurements in a small testbed to study, at a microscopic level, the effects of various types of packet losses on TCP's performance. In particular, we single out packet losses that impact the tail end of small flows, as well as bursty losses, that span a significant fraction of the small congestion window of TCP flows in data-centers, to show a non-negligible effect on the FCT. Based on this, we propose the so-called, timely-retransmitted ACKs (or T-RACKs), a simple loss recovery mechanism to conceal the drawbacks of the long RTO even in the presence of heavy packet losses. Interestingly enough, T-RACKS achieves this transparently to TCP itself as it does not require any change to TCP in the tenant's virtual machine (VM). T-RACKs can be implemented as a software shim layer in the hypervisor between the VMs and server's NIC or in hardware as a networking function in a SmartNIC. Simulation and real testbed results show that T-RACKs achieves remarkable performance improvements.
△ Less
Submitted 15 February, 2021;
originally announced February 2021.
-
An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems
Authors:
Ahmed M. Abdelmoniem,
Ahmed Elzanaty,
Mohamed-Slim Alouini,
Marco Canini
Abstract:
The recent many-fold increase in the size of deep neural networks makes efficient distributed training challenging. Many proposals exploit the compressibility of the gradients and propose lossy compression techniques to speed up the communication stage of distributed training. Nevertheless, compression comes at the cost of reduced model quality and extra computation overhead. In this work, we desi…
▽ More
The recent many-fold increase in the size of deep neural networks makes efficient distributed training challenging. Many proposals exploit the compressibility of the gradients and propose lossy compression techniques to speed up the communication stage of distributed training. Nevertheless, compression comes at the cost of reduced model quality and extra computation overhead. In this work, we design an efficient compressor with minimal overhead. Noting the sparsity of the gradients, we propose to model the gradients as random variables distributed according to some sparsity-inducing distributions (SIDs). We empirically validate our assumption by studying the statistical characteristics of the evolution of gradient vectors over the training process. We then propose Sparsity-Inducing Distribution-based Compression (SIDCo), a threshold-based sparsification scheme that enjoys similar threshold estimation quality to deep gradient compression (DGC) while being faster by imposing lower compression overhead. Our extensive evaluation of popular machine learning benchmarks involving both recurrent neural network (RNN) and convolution neural network (CNN) models shows that SIDCo speeds up training by up to 41:7%, 7:6%, and 1:9% compared to the no-compression baseline, Topk, and DGC compressors, respectively.
△ Less
Submitted 17 March, 2021; v1 submitted 26 January, 2021;
originally announced January 2021.
-
Design and Implementation of Fair Congestion Control for Data Centers Networks
Authors:
Ahmed M. Abdelmoniem,
Brahim Bensaou
Abstract:
In data centers, the nature of the composite bursty traffic along with the small bandwidth-delay product and switch buffers lead to several congestion problems that are not handled well by traditional congestion control mechanisms such as TCP. Existing work try to address the problem by modifying TCP to suit the operational nature of data centers. This is practically feasible in private settings,…
▽ More
In data centers, the nature of the composite bursty traffic along with the small bandwidth-delay product and switch buffers lead to several congestion problems that are not handled well by traditional congestion control mechanisms such as TCP. Existing work try to address the problem by modifying TCP to suit the operational nature of data centers. This is practically feasible in private settings, however, in public environments, such modifications are prohibited. Therefore, in this work, we design simple switch-based queue management to deal with such congestion issues adequately. This approach entails no modification to the TCP sender and receiver algorithms which enables easy and seamless deployment in public data centers. We present a theoretical analysis to show the stability and effectiveness of the scheme. We also present, three different real implementations (as a Linux kernel module and as an added feature to OpenvSwitch) and give numerical results from both NS-2 simulation and experiments of real deployment in a small test-bed cluster to show its effectiveness in achieving high throughput overall, a good fairness and short flow completion times for delay-sensitive flows.
△ Less
Submitted 1 December, 2020;
originally announced December 2020.
-
On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learning
Authors:
Aritra Dutta,
El Houcine Bergou,
Ahmed M. Abdelmoniem,
Chen-Yu Ho,
Atal Narayan Sahu,
Marco Canini,
Panos Kalnis
Abstract:
Compressed communication, in the form of sparsification or quantization of stochastic gradients, is employed to reduce communication costs in distributed data-parallel training of deep neural networks. However, there exists a discrepancy between theory and practice: while theoretical analysis of most existing compression methods assumes compression is applied to the gradients of the entire model,…
▽ More
Compressed communication, in the form of sparsification or quantization of stochastic gradients, is employed to reduce communication costs in distributed data-parallel training of deep neural networks. However, there exists a discrepancy between theory and practice: while theoretical analysis of most existing compression methods assumes compression is applied to the gradients of the entire model, many practical implementations operate individually on the gradients of each layer of the model. In this paper, we prove that layer-wise compression is, in theory, better, because the convergence rate is upper bounded by that of entire-model compression for a wide range of biased and unbiased compression methods. However, despite the theoretical bound, our experimental study of six well-known methods shows that convergence, in practice, may or may not be better, depending on the actual trained model and compression ratio. Our findings suggest that it would be advantageous for deep learning frameworks to include support for both layer-wise and entire-model compression.
△ Less
Submitted 19 November, 2019;
originally announced November 2019.
