-
Multi-Objective Optimization of Consumer Group Autoscaling in Message Broker Systems
Authors:
Diogo Landau,
Nishant Saurabh,
Xavier Andrade,
Jorge G Barbosa
Abstract:
Message brokers often mediate communication between data producers and consumers by adding variable-sized messages to ordered distributed queues. Our goal is to determine the number of consumers and consumer-partition assignments needed to ensure that the rate of data consumption keeps up with the rate of data production. We model the problem as a variable item size bin packing problem. As the rat…
▽ More
Message brokers often mediate communication between data producers and consumers by adding variable-sized messages to ordered distributed queues. Our goal is to determine the number of consumers and consumer-partition assignments needed to ensure that the rate of data consumption keeps up with the rate of data production. We model the problem as a variable item size bin packing problem. As the rate of production varies, new consumer-partition assignments are computed, which may require rebalancing a partition from one consumer to another. While rebalancing a queue, the data being produced into the queue is not read leading to additional latency costs. As such, we focus on the multi-objective optimization cost of minimizing both the number of consumers and queue migrations. We present a variety of algorithms and compare them to established bin packing heuristics for this application. Comparing our proposed consumer group assignment strategy with Kafka's, a commonly employed strategy, our strategy presents a 90th percentile latency of 4.52s compared to Kafka's 217s with both using the same amount of consumers. Kafka's assignment strategy only improved the consumer group's performance with regards to latency with configurations that used at least 60% more resources than our approach.
△ Less
Submitted 8 February, 2024;
originally announced February 2024.
-
Kafka Consumer Group Autoscaler
Authors:
Diogo Landau,
Xavier Andrade,
Jorge G. Barbosa
Abstract:
Message brokers enable asynchronous communication between data producers and consumers in distributed environments by assigning messages to ordered queues. Message broker systems often provide with mechanisms to parallelize tasks between consumers to increase the rate at which data is consumed. The consumption rate must exceed the production rate or queues would grow indefinitely. Still, consumers…
▽ More
Message brokers enable asynchronous communication between data producers and consumers in distributed environments by assigning messages to ordered queues. Message broker systems often provide with mechanisms to parallelize tasks between consumers to increase the rate at which data is consumed. The consumption rate must exceed the production rate or queues would grow indefinitely. Still, consumers are costly and their number should be minimized. We model the problem of determining the required number of consumers, and the partition-consumer assignments, as a variable item size bin packing variant. Data cannot be read when a queue is being migrated to another consumer. Hence, we propose the R-score metric to account for these rebalancing costs. Then, we introduce an assortment of R-score based algorithms, and compare their performance to established heuristics for the Bin Packing Problem for this application. We instantiate our method within an existing system, demonstrating its effectiveness. Our approach guarantees adequate consumption rates something the previous system was unable to at lower operational costs.
△ Less
Submitted 22 June, 2022;
originally announced June 2022.
-
Semantics-aware Virtual Machine Image Management in IaaS Clouds
Authors:
Nishant Saurabh,
Julian Remmers,
Dragi Kimovski,
Radu Prodan,
Jorge G. Barbosa
Abstract:
Infrastructure-as-a-service (IaaS) Clouds concurrently accommodate diverse sets of user requests, requiring an efficient strategy for storing and retrieving virtual machine images (VMIs) at a large scale. The VMI storage management require dealing with multiple VMIs, typically in the magnitude of gigabytes, which entails VMI sprawl issues hindering the elastic resource management and provisioning.…
▽ More
Infrastructure-as-a-service (IaaS) Clouds concurrently accommodate diverse sets of user requests, requiring an efficient strategy for storing and retrieving virtual machine images (VMIs) at a large scale. The VMI storage management require dealing with multiple VMIs, typically in the magnitude of gigabytes, which entails VMI sprawl issues hindering the elastic resource management and provisioning. Nevertheless, existing techniques to facilitate VMI management overlook VMI semantics (i.e at the level of base image and software packages) with either restricted possibility to identify and extract reusable functionalities or with higher VMI publish and retrieval overheads. In this paper, we design, implement and evaluate Expelliarmus, a novel VMI management system that helps to minimize storage, publish and retrieval overheads. To achieve this goal, Expelliarmus incorporates three complementary features. First, it makes use of VMIs modelled as semantic graphs to expedite the similarity computation between multiple VMIs. Second, Expelliarmus provides a semantic aware VMI decomposition and base image selection to extract and store non-redundant base image and software packages. Third, Expelliarmus can also assemble VMIs based on the required software packages upon user request. We evaluate Expelliarmus through a representative set of synthetic Cloud VMIs on the real test-bed. Experimental results show that our semantic-centric approach is able to optimize repository size by 2.2-16 times compared to state-of-the-art systems (e.g. IBM's Mirage and Hemera) with significant VMI publish and slight retrieval performance improvement.
△ Less
Submitted 29 July, 2019; v1 submitted 21 June, 2019;
originally announced June 2019.
-
A State-of-the-art Integrated Transportation Simulation Platform
Authors:
Tiago Azevedo,
Rosaldo J. F. Rossetti,
Jorge G. Barbosa
Abstract:
Nowadays, universities and companies have a huge need for simulation and modelling methodologies. In the particular case of traffic and transportation, making physical modifications to the real traffic networks could be highly expensive, dependent on political decisions and could be highly disruptive to the environment. However, while studying a specific domain or problem, analysing a problem thro…
▽ More
Nowadays, universities and companies have a huge need for simulation and modelling methodologies. In the particular case of traffic and transportation, making physical modifications to the real traffic networks could be highly expensive, dependent on political decisions and could be highly disruptive to the environment. However, while studying a specific domain or problem, analysing a problem through simulation may not be trivial and may need several simulation tools, hence raising interoperability issues. To overcome these problems, we propose an agent-directed transportation simulation platform, through the cloud, by means of services. We intend to use the IEEE standard HLA (High Level Architecture) for simulators interoperability and agents for controlling and coordination. Our motivations are to allow multiresolution analysis of complex domains, to allow experts to collaborate on the analysis of a common problem and to allow co-simulation and synergy of different application domains. This paper will start by presenting some preliminary background concepts to help better understand the scope of this work. After that, the results of a literature review is shown. Finally, the general architecture of a transportation simulation platform is proposed.
△ Less
Submitted 29 January, 2016;
originally announced January 2016.
-
Densifying the sparse cloud SimSaaS: The need of a synergy among agent-directed simulation, SimSaaS and HLA
Authors:
Tiago Azevedo,
Rosaldo J. F. Rossetti,
Jorge G. Barbosa
Abstract:
Modelling & Simulation (M&S) is broadly used in real scenarios where making physical modifications could be highly expensive. With the so-called Simulation Software-as-a-Service (SimSaaS), researchers could take advantage of the huge amount of resource that cloud computing provides. Even so, studying and analysing a problem through simulation may need several simulation tools, hence raising intero…
▽ More
Modelling & Simulation (M&S) is broadly used in real scenarios where making physical modifications could be highly expensive. With the so-called Simulation Software-as-a-Service (SimSaaS), researchers could take advantage of the huge amount of resource that cloud computing provides. Even so, studying and analysing a problem through simulation may need several simulation tools, hence raising interoperability issues. Having this in mind, IEEE developed a standard for interoperability among simulators named High Level Architecture (HLA). Moreover, the multi-agent system approach has become recognised as a convenient approach for modelling and simulating complex systems. Despite all the recent works and acceptance of these technologies, there is still a great lack of work regarding synergies among them. This paper shows by means of a literature review this lack of work or, in other words, the sparse Cloud SimSaaS. The literature review and the resulting taxonomy are the main contributions of this paper, as they provide a research agenda illustrating future research opportunities and trends.
△ Less
Submitted 29 January, 2016;
originally announced January 2016.
