-
Compass: A Decentralized Scheduler for Latency-Sensitive ML Workflows
Authors:
Yuting Yang,
Andrea Merlina,
Weijia Song,
Tiancheng Yuan,
Ken Birman,
Roman Vitenberg
Abstract:
We consider ML query processing in distributed systems where GPU-enabled workers coordinate to execute complex queries: a computing style often seen in applications that interact with users in support of image processing and natural language processing. In such systems, coscheduling of GPU memory management and task placement represents a promising opportunity. We propose Compass, a novel framewor…
▽ More
We consider ML query processing in distributed systems where GPU-enabled workers coordinate to execute complex queries: a computing style often seen in applications that interact with users in support of image processing and natural language processing. In such systems, coscheduling of GPU memory management and task placement represents a promising opportunity. We propose Compass, a novel framework that unifies these functions to reduce job latency while using resources efficiently, placing tasks where data dependencies will be satisfied, collocating tasks from the same job (when this will not overload the host or its GPU), and efficiently managing GPU memory. Comparison with other state of the art schedulers shows a significant reduction in completion times while requiring the same amount or even fewer resources. In one case, just half the servers were needed for processing the same workload.
△ Less
Submitted 28 February, 2024; v1 submitted 27 February, 2024;
originally announced February 2024.
-
Low-Latency ML Inference by Grou** Correlated Data Objects and Computation
Authors:
Thiago Garrett,
Weijia Song,
Roman Vitenberg,
Ken Birman
Abstract:
ML inference workflows often require low latency and high throughput, yet we lack good options for addressing this need. Techniques that reduce latency in other streaming settings (such as caching and optimization-driven scheduling) are of limited value because ML data dependencies are often very large and can change dramatically depending on the triggering event. In this work, we propose a novel…
▽ More
ML inference workflows often require low latency and high throughput, yet we lack good options for addressing this need. Techniques that reduce latency in other streaming settings (such as caching and optimization-driven scheduling) are of limited value because ML data dependencies are often very large and can change dramatically depending on the triggering event. In this work, we propose a novel correlation grou** mechanism that makes it easier for developers to express application-specific data access correlations, enabling coordinated management of data objects in server clusters hosting streaming inference tasks. Experiments based on a latency-sensitive ML-based application confirm the limitations of standard techniques while showing that our solution yields dramatically better performance. The proposed mechanism is able to maintain significantly lower and more consistent latency, achieves higher node utilization as workload and scale-out increase, and yet requires only minor changes to the code implementing the application.
△ Less
Submitted 30 November, 2023;
originally announced December 2023.
-
Cascade: A Platform for Delay-Sensitive Edge Intelligence
Authors:
Weijia Song,
Thiago Garrett,
Yuting Yang,
Mingzhao Liu,
Edward Tremel,
Lorenzo Rosa,
Andrea Merlina,
Roman Vitenberg,
Ken Birman
Abstract:
Interactive intelligent computing applications are increasingly prevalent, creating a need for AI/ML platforms optimized to reduce per-event latency while maintaining high throughput and efficient resource management. Yet many intelligent applications run on AI/ML platforms that optimize for high throughput even at the cost of high tail-latency. Cascade is a new AI/ML hosting platform intended to…
▽ More
Interactive intelligent computing applications are increasingly prevalent, creating a need for AI/ML platforms optimized to reduce per-event latency while maintaining high throughput and efficient resource management. Yet many intelligent applications run on AI/ML platforms that optimize for high throughput even at the cost of high tail-latency. Cascade is a new AI/ML hosting platform intended to untangle this puzzle. Innovations include a legacy-friendly storage layer that moves data with minimal copying and a "fast path" that collocates data and computation to maximize responsiveness. Our evaluation shows that Cascade reduces latency by orders of magnitude with no loss of throughput.
△ Less
Submitted 28 November, 2023;
originally announced November 2023.
-
Spindle: Techniques for Optimizing Atomic Multicast on RDMA
Authors:
Sagar Jha,
Lorenzo Rosa,
Ken Birman
Abstract:
Leveraging one-sided RDMA for applications that replicate small data objects can be surprisingly difficult: such uses amplify any protocol overheads. Spindle is a set of optimization techniques for systematically tackling this class of challenges for atomic multicast over RDMA. These include memory polling optimizations using novel sender and receiver batching techniques, null-message send logic,…
▽ More
Leveraging one-sided RDMA for applications that replicate small data objects can be surprisingly difficult: such uses amplify any protocol overheads. Spindle is a set of optimization techniques for systematically tackling this class of challenges for atomic multicast over RDMA. These include memory polling optimizations using novel sender and receiver batching techniques, null-message send logic, and improved multi-thread synchronization. We applied Spindle to Derecho, an open-source C++ library for atomic multicast, and obtained significant performance improvements both for the library itself and for an OMG-compliant avionics DDS built over Derecho. Derecho's multicast bandwidth utilization for 10KB messages rose from 1GB/s to 9.7GB/s on a 12.5GB/s network, and it became more robust to delays. Interestingly, although some of our techniques employ batching, latency dropped by nearly two orders of magnitude. Spindle optimizations should also be of value in other RDMA applications limited by the speed of coordination.
△ Less
Submitted 2 October, 2021;
originally announced October 2021.
-
Cache Serializability: Reducing Inconsistency in Edge Transactions
Authors:
Ittay Eyal,
Ken Birman,
Robbert van Renesse
Abstract:
Read-only caches are widely used in cloud infrastructures to reduce access latency and load on backend databases. Operators view coherent caches as impractical at genuinely large scale and many client-facing caches are updated in an asynchronous manner with best-effort pipelines. Existing solutions that support cache consistency are inapplicable to this scenario since they require a round trip to…
▽ More
Read-only caches are widely used in cloud infrastructures to reduce access latency and load on backend databases. Operators view coherent caches as impractical at genuinely large scale and many client-facing caches are updated in an asynchronous manner with best-effort pipelines. Existing solutions that support cache consistency are inapplicable to this scenario since they require a round trip to the database on every cache transaction.
Existing incoherent cache technologies are oblivious to transactional data access, even if the backend database supports transactions. We propose T-Cache, a novel caching policy for read-only transactions in which inconsistency is tolerable (won't cause safety violations) but undesirable (has a cost). T-Cache improves cache consistency despite asynchronous and unreliable communication between the cache and the database. We define cache-serializability, a variant of serializability that is suitable for incoherent caches, and prove that with unbounded resources T-Cache implements this new specification. With limited resources, T-Cache allows the system manager to choose a trade-off between performance and consistency.
Our evaluation shows that T-Cache detects many inconsistencies with only nominal overhead. We use synthetic workloads to demonstrate the efficacy of T-Cache when data accesses are clustered and its adaptive reaction to workload changes. With workloads based on the real-world topologies, T-Cache detects 43-70% of the inconsistencies and increases the rate of consistent transactions by 33-58%.
△ Less
Submitted 26 April, 2015; v1 submitted 29 September, 2014;
originally announced September 2014.
-
Practical Experience Report: The Performance of Paxos in the Cloud
Authors:
Parisa Jalili Marandi,
Samuel Benz,
Fernando Pedone,
Ken Birman
Abstract:
This experience report presents the results of an extensive performance evaluation conducted using four open-source implementations of Paxos deployed in Amazon's EC2. Paxos is a fundamental algorithm for building fault-tolerant services, at the core of state-machine replication. Implementations of Paxos are currently used in many prototypes and production systems in both academia and industry. Alt…
▽ More
This experience report presents the results of an extensive performance evaluation conducted using four open-source implementations of Paxos deployed in Amazon's EC2. Paxos is a fundamental algorithm for building fault-tolerant services, at the core of state-machine replication. Implementations of Paxos are currently used in many prototypes and production systems in both academia and industry. Although all protocols surveyed in the paper implement Paxos, they are optimized in a number of different ways, resulting in very different behavior, as we show in the paper. We have considered a variety of configurations and failure-free and faulty executions. In addition to reporting our findings, we propose and assess additional optimizations to existing implementations.
△ Less
Submitted 27 April, 2014;
originally announced April 2014.
-
The Design and Architecture of the Microsoft Cluster Service -- A Practical Approach to High-Availability and Scalability
Authors:
Werner Vogels,
Dan Dumitriu,
Ken Birman,
Rod Gamache,
Mike Massa,
Rob Short,
John Vert,
Joe Barrera
Abstract:
Microsoft Cluster Service (MSCS) extends the Win-dows NT operating system to support high-availability services. The goal is to offer an execution environment where off-the-shelf server applications can continue to operate, even in the presence of node failures. Later ver-sions of MSCS will provide scalability via a node and application management system that allows applications to scale to hund…
▽ More
Microsoft Cluster Service (MSCS) extends the Win-dows NT operating system to support high-availability services. The goal is to offer an execution environment where off-the-shelf server applications can continue to operate, even in the presence of node failures. Later ver-sions of MSCS will provide scalability via a node and application management system that allows applications to scale to hundreds of nodes. This paper provides a de-tailed description of the MSCS architecture and the de-sign decisions that have driven the implementation of the service. The paper also describes how some major appli-cations use the MSCS features, and describes features added to make it easier to implement and manage fault-tolerant applications on MSCS.
△ Less
Submitted 2 September, 1998;
originally announced September 1998.
