-
CHESS: Contextual Harnessing for Efficient SQL Synthesis
Authors:
Shayan Talaei,
Mohammadreza Pourreza,
Yu-Chen Chang,
Azalia Mirhoseini,
Amin Saberi
Abstract:
Utilizing large language models (LLMs) for transforming natural language questions into SQL queries (text-to-SQL) is a promising yet challenging approach, particularly when applied to real-world databases with complex and extensive schemas. In particular, effectively incorporating data catalogs and database values for SQL generation remains an obstacle, leading to suboptimal solutions. We address…
▽ More
Utilizing large language models (LLMs) for transforming natural language questions into SQL queries (text-to-SQL) is a promising yet challenging approach, particularly when applied to real-world databases with complex and extensive schemas. In particular, effectively incorporating data catalogs and database values for SQL generation remains an obstacle, leading to suboptimal solutions. We address this problem by proposing a new pipeline that effectively retrieves relevant data and context, selects an efficient schema, and synthesizes correct and efficient SQL queries. To increase retrieval precision, our pipeline introduces a hierarchical retrieval method leveraging model-generated keywords, locality-sensitive hashing indexing, and vector databases. Additionally, we have developed an adaptive schema pruning technique that adjusts based on the complexity of the problem and the model's context size. Our approach generalizes to both frontier proprietary models like GPT-4 and open-source models such as Llama-3-70B. Through a series of ablation studies, we demonstrate the effectiveness of each component of our pipeline and its impact on the end-to-end performance. Our method achieves new state-of-the-art performance on the cross-domain challenging BIRD dataset.
△ Less
Submitted 27 June, 2024; v1 submitted 26 May, 2024;
originally announced May 2024.
-
Hybrid Decentralized Optimization: First- and Zeroth-Order Optimizers Can Be Jointly Leveraged For Faster Convergence
Authors:
Shayan Talaei,
Giorgi Nadiradze,
Dan Alistarh
Abstract:
Distributed optimization has become one of the standard ways of speeding up machine learning training, and most of the research in the area focuses on distributed first-order, gradient-based methods. Yet, there are settings where some computationally-bounded nodes may not be able to implement first-order, gradient-based optimization, while they could still contribute to joint optimization tasks. I…
▽ More
Distributed optimization has become one of the standard ways of speeding up machine learning training, and most of the research in the area focuses on distributed first-order, gradient-based methods. Yet, there are settings where some computationally-bounded nodes may not be able to implement first-order, gradient-based optimization, while they could still contribute to joint optimization tasks. In this paper, we initiate the study of hybrid decentralized optimization, studying settings where nodes with zeroth-order and first-order optimization capabilities co-exist in a distributed system, and attempt to jointly solve an optimization task over some data distribution. We essentially show that, under reasonable parameter settings, such a system can not only withstand noisier zeroth-order agents but can even benefit from integrating such agents into the optimization process, rather than ignoring their information. At the core of our approach is a new analysis of distributed optimization with noisy and possibly-biased gradient estimators, which may be of independent interest. Experimental results on standard optimization tasks confirm our analysis, showing that hybrid first-zeroth order optimization can be practical.
△ Less
Submitted 14 October, 2022;
originally announced October 2022.
-
Communication-Efficient Federated Learning With Data and Client Heterogeneity
Authors:
Hossein Zakerinia,
Shayan Talaei,
Giorgi Nadiradze,
Dan Alistarh
Abstract:
Federated Learning (FL) enables large-scale distributed training of machine learning models, while still allowing individual nodes to maintain data locally.
However, executing FL at scale comes with inherent practical challenges:
1) heterogeneity of the local node data distributions,
2) heterogeneity of node computational speeds (asynchrony),
but also 3) constraints in the amount of commun…
▽ More
Federated Learning (FL) enables large-scale distributed training of machine learning models, while still allowing individual nodes to maintain data locally.
However, executing FL at scale comes with inherent practical challenges:
1) heterogeneity of the local node data distributions,
2) heterogeneity of node computational speeds (asynchrony),
but also 3) constraints in the amount of communication between the clients and the server.
In this work, we present the first variant of the classic federated averaging (FedAvg) algorithm
which, at the same time, supports data heterogeneity, partial client asynchrony, and communication compression.
Our algorithm comes with a rigorous analysis showing that, in spite of these system relaxations,
it can provide similar convergence to FedAvg in interesting parameter regimes.
Experimental results in the rigorous LEAF benchmark on setups of up to $300$ nodes show that our algorithm ensures fast convergence for standard federated tasks, improving upon prior quantized and asynchronous approaches.
△ Less
Submitted 3 June, 2023; v1 submitted 20 June, 2022;
originally announced June 2022.
