-
Little Giants: Exploring the Potential of Small LLMs as Evaluation Metrics in Summarization in the Eval4NLP 2023 Shared Task
Authors:
Neema Kotonya,
Saran Krishnasamy,
Joel Tetreault,
Alejandro Jaimes
Abstract:
This paper describes and analyzes our participation in the 2023 Eval4NLP shared task, which focuses on assessing the effectiveness of prompt-based techniques to empower Large Language Models to handle the task of quality estimation, particularly in the context of evaluating machine translations and summaries. We conducted systematic experiments with various prompting techniques, including standard…
▽ More
This paper describes and analyzes our participation in the 2023 Eval4NLP shared task, which focuses on assessing the effectiveness of prompt-based techniques to empower Large Language Models to handle the task of quality estimation, particularly in the context of evaluating machine translations and summaries. We conducted systematic experiments with various prompting techniques, including standard prompting, prompts informed by annotator instructions, and innovative chain-of-thought prompting. In addition, we integrated these approaches with zero-shot and one-shot learning methods to maximize the efficacy of our evaluation procedures. Our work reveals that combining these approaches using a "small", open source model (orca_mini_v3_7B) yields competitive results.
△ Less
Submitted 1 November, 2023;
originally announced November 2023.
-
Sample complexity of partition identification using multi-armed bandits
Authors:
Sandeep Juneja,
Subhashini Krishnasamy
Abstract:
Given a vector of probability distributions, or arms, each of which can be sampled independently, we consider the problem of identifying the partition to which this vector belongs from a finitely partitioned universe of such vector of distributions. We study this as a pure exploration problem in multi armed bandit settings and develop sample complexity bounds on the total mean number of samples re…
▽ More
Given a vector of probability distributions, or arms, each of which can be sampled independently, we consider the problem of identifying the partition to which this vector belongs from a finitely partitioned universe of such vector of distributions. We study this as a pure exploration problem in multi armed bandit settings and develop sample complexity bounds on the total mean number of samples required for identifying the correct partition with high probability. This framework subsumes well studied problems such as finding the best arm or the best few arms. We consider distributions belonging to the single parameter exponential family and primarily consider partitions where the vector of means of arms lie either in a given set or its complement. The sets considered correspond to distributions where there exists a mean above a specified threshold, where the set is a half space and where either the set or its complement is a polytope, or more generally, a convex set. In these settings, we characterize the lower bounds on mean number of samples for each arm highlighting their dependence on the problem geometry. Further, inspired by the lower bounds, we propose algorithms that can match these bounds asymptotically with decreasing probability of error. Applications of this framework may be diverse. We briefly discuss one associated with finance.
△ Less
Submitted 5 February, 2019; v1 submitted 14 November, 2018;
originally announced November 2018.
-
On Learning the $cμ$ Rule in Single and Parallel Server Networks
Authors:
Subhashini Krishnasamy,
Ari Arapostathis,
Ramesh Johari,
Sanjay Shakkottai
Abstract:
We consider learning-based variants of the $c μ$ rule for scheduling in single and parallel server settings of multi-class queueing systems.
In the single server setting, the $c μ$ rule is known to minimize the expected holding-cost (weighted queue-lengths summed over classes and a fixed time horizon). We focus on the problem where the service rates $μ$ are unknown with the holding-cost regret (…
▽ More
We consider learning-based variants of the $c μ$ rule for scheduling in single and parallel server settings of multi-class queueing systems.
In the single server setting, the $c μ$ rule is known to minimize the expected holding-cost (weighted queue-lengths summed over classes and a fixed time horizon). We focus on the problem where the service rates $μ$ are unknown with the holding-cost regret (regret against the $c μ$ rule with known $μ$) as our objective. We show that the greedy algorithm that uses empirically learned service rates results in a constant holding-cost regret (the regret is independent of the time horizon). This free exploration can be explained in the single server setting by the fact that any work-conserving policy obtains the same number of samples in a busy cycle.
In the parallel server setting, we show that the $c μ$ rule may result in unstable queues, even for arrival rates within the capacity region. We then present sufficient conditions for geometric ergodicity under the $c μ$ rule. Using these results, we propose an almost greedy algorithm that explores only when the number of samples falls below a threshold. We show that this algorithm delivers constant holding-cost regret because a free exploration condition is eventually satisfied.
△ Less
Submitted 2 July, 2018; v1 submitted 2 February, 2018;
originally announced February 2018.
-
ADWISERv2: A Plug-and-play Controller for Managing TCP Transfers in IEEE~802.11 Infrastructure WLANs with Multiple Access Points
Authors:
Albert Sunny,
Sumankumar Panchal,
Nikhil Vidhani,
Subhashini Krishnasamy,
S. V. R. Anand,
Malati Hegde,
Joy Kuri,
Anurag Kumar
Abstract:
In this paper, we present a generic plug-and-play controller that ensures fair and efficient operation of IEEE~802.11 infrastructure wireless local area networks with multiple co-channel access points, without any change to hardware/firmware of the network devices. Our controller addresses performance issues of TCP transfers in multi-AP WLANs, by overlaying a coarse time-slicing scheduler on top o…
▽ More
In this paper, we present a generic plug-and-play controller that ensures fair and efficient operation of IEEE~802.11 infrastructure wireless local area networks with multiple co-channel access points, without any change to hardware/firmware of the network devices. Our controller addresses performance issues of TCP transfers in multi-AP WLANs, by overlaying a coarse time-slicing scheduler on top of a cascaded fair queuing scheduler. The time slices and queue weights, used in our controller, are obtained from the solution of a constrained utility optimization formulation. A study of the impact of coarse time-slicing on TCP is also presented in this paper. We present an improved algorithm for adaptation of the service rate of the fair queuing scheduler and provide experimental results to illustrate its efficacy. We also present the changes that need to be incorporated to the proposed approach, to handle short-lived and interactive TCP flows. Finally, we report the results of experiments performed on a real testbed, demonstrating the efficacy of our controller.
△ Less
Submitted 14 September, 2016;
originally announced September 2016.
-
Detecting Sponsored Recommendations
Authors:
Subhashini Krishnasamy,
Rajat Sen,
Sewoong Oh,
Sanjay Shakkottai
Abstract:
With a vast number of items, web-pages, and news to choose from, online services and the customers both benefit tremendously from personalized recommender systems. Such systems however provide great opportunities for targeted advertisements, by displaying ads alongside genuine recommendations. We consider a biased recommendation system where such ads are displayed without any tags (disguised as ge…
▽ More
With a vast number of items, web-pages, and news to choose from, online services and the customers both benefit tremendously from personalized recommender systems. Such systems however provide great opportunities for targeted advertisements, by displaying ads alongside genuine recommendations. We consider a biased recommendation system where such ads are displayed without any tags (disguised as genuine recommendations), rendering them indistinguishable to a single user. We ask whether it is possible for a small subset of collaborating users to detect such a bias. We propose an algorithm that can detect such a bias through statistical analysis on the collaborating users' feedback. The algorithm requires only binary information indicating whether a user was satisfied with each of the recommended item or not. This makes the algorithm widely appealing to real world issues such as identification of search engine bias and pharmaceutical lobbying. We prove that the proposed algorithm detects the bias with high probability for a broad class of recommendation systems when sufficient number of users provide feedback on sufficient number of recommendations. We provide extensive simulations with real data sets and practical recommender systems, which confirm the trade offs in the theoretical guarantees.
△ Less
Submitted 14 April, 2015;
originally announced April 2015.
-
On the Scaling of Interference Alignment Under Delay and Power Constraints
Authors:
Subhashini Krishnasamy,
Urs Niesen,
Piyush Gupta
Abstract:
Future wireless standards such as 5G envision dense wireless networks with large number of simultaneously connected devices. In this context, interference management becomes critical in achieving high spectral efficiency. Orthogonal signaling, which limits the number of users utilizing the resource simultaneously, gives a sum-rate that remains constant with increasing number of users. An alternati…
▽ More
Future wireless standards such as 5G envision dense wireless networks with large number of simultaneously connected devices. In this context, interference management becomes critical in achieving high spectral efficiency. Orthogonal signaling, which limits the number of users utilizing the resource simultaneously, gives a sum-rate that remains constant with increasing number of users. An alternative approach called interference alignment promises a throughput that scales linearly with the number of users. However, this approach requires very high SNR or long time duration for sufficient channel variation, and therefore may not be feasible in real wireless systems. We explore ways to manage interference in large networks with delay and power constraints. Specifically, we devise an interference phase alignment strategy that combines precoding and scheduling without using power control to exploit the diversity inherent in a system with large number of users. We show that this scheme achieves a sum-rate that scales almost logarithmically with the number of users. We also show that no scheme using single symbol phase alignment, which is asymmetric complex signaling restricted to a single complex symbol, can achieve better than logarithmic scaling of the sum-rate.
△ Less
Submitted 13 April, 2015;
originally announced April 2015.
-
The Behavior of Epidemics under Bounded Susceptibility
Authors:
Subhashini Krishnasamy,
Siddhartha Banerjee,
Sanjay Shakkottai
Abstract:
We investigate the sensitivity of epidemic behavior to a bounded susceptibility constraint -- susceptible nodes are infected by their neighbors via the regular SI/SIS dynamics, but subject to a cap on the infection rate. Such a constraint is motivated by modern social networks, wherein messages are broadcast to all neighbors, but attention spans are limited. Bounded susceptibility also arises in d…
▽ More
We investigate the sensitivity of epidemic behavior to a bounded susceptibility constraint -- susceptible nodes are infected by their neighbors via the regular SI/SIS dynamics, but subject to a cap on the infection rate. Such a constraint is motivated by modern social networks, wherein messages are broadcast to all neighbors, but attention spans are limited. Bounded susceptibility also arises in distributed computing applications with download bandwidth constraints, and in human epidemics under quarantine policies.
Network epidemics have been extensively studied in literature; prior work characterizes the graph structures required to ensure fast spreading under the SI dynamics, and long lifetime under the SIS dynamics. In particular, these conditions turn out to be meaningful for two classes of networks of practical relevance -- dense, uniform (i.e., clique-like) graphs, and sparse, structured (i.e., star-like) graphs. We show that bounded susceptibility has a surprising impact on epidemic behavior in these graph families. For the SI dynamics, bounded susceptibility has no effect on star-like networks, but dramatically alters the spreading time in clique-like networks. In contrast, for the SIS dynamics, clique-like networks are unaffected, but star-like networks exhibit a sharp change in extinction times under bounded susceptibility.
Our findings are useful for the design of disease-resistant networks and infrastructure networks. More generally, they show that results for existing epidemic models are sensitive to modeling assumptions in non-intuitive ways, and suggest caution in directly using these as guidelines for real systems.
△ Less
Submitted 17 February, 2015;
originally announced February 2015.
