-
Value-based Resource Matching with Fairness Criteria: Application to Agricultural Water Trading
Authors:
Abhi** Adiga,
Yohai Trabelsi,
Tanvir Ferdousi,
Madhav Marathe,
S. S. Ravi,
Samarth Swarup,
Anil Kumar Vullikanti,
Mandy L. Wilson,
Sarit Kraus,
Reetwika Basu,
Supriya Savalkar,
Matthew Yourek,
Michael Brady,
Kirti Rajagopalan,
Jonathan Yoder
Abstract:
Optimal allocation of agricultural water in the event of droughts is an important global problem. In addressing this problem, many aspects, including the welfare of farmers, the economy, and the environment, must be considered. Under this backdrop, our work focuses on several resource-matching problems accounting for agents with multi-crop portfolios, geographic constraints, and fairness. First, w…
▽ More
Optimal allocation of agricultural water in the event of droughts is an important global problem. In addressing this problem, many aspects, including the welfare of farmers, the economy, and the environment, must be considered. Under this backdrop, our work focuses on several resource-matching problems accounting for agents with multi-crop portfolios, geographic constraints, and fairness. First, we address a matching problem where the goal is to maximize a welfare function in two-sided markets where buyers' requirements and sellers' supplies are represented by value functions that assign prices (or costs) to specified volumes of water. For the setting where the value functions satisfy certain monotonicity properties, we present an efficient algorithm that maximizes a social welfare function. When there are minimum water requirement constraints, we present a randomized algorithm which ensures that the constraints are satisfied in expectation. For a single seller--multiple buyers setting with fairness constraints, we design an efficient algorithm that maximizes the minimum level of satisfaction of any buyer. We also present computational complexity results that highlight the limits on the generalizability of our results. We evaluate the algorithms developed in our work with experiments on both real-world and synthetic data sets with respect to drought severity, value functions, and seniority of agents.
△ Less
Submitted 11 February, 2024; v1 submitted 9 February, 2024;
originally announced February 2024.
-
POSSCORE: A Simple Yet Effective Evaluation of Conversational Search with Part of Speech Labelling
Authors:
Zeyang Liu,
Ke Zhou,
Jiaxin Mao,
Max L. Wilson
Abstract:
Conversational search systems, such as Google Assistant and Microsoft Cortana, provide a new search paradigm where users are allowed, via natural language dialogues, to communicate with search systems. Evaluating such systems is very challenging since search results are presented in the format of natural language sentences. Given the unlimited number of possible responses, collecting relevance ass…
▽ More
Conversational search systems, such as Google Assistant and Microsoft Cortana, provide a new search paradigm where users are allowed, via natural language dialogues, to communicate with search systems. Evaluating such systems is very challenging since search results are presented in the format of natural language sentences. Given the unlimited number of possible responses, collecting relevance assessments for all the possible responses is infeasible. In this paper, we propose POSSCORE, a simple yet effective automatic evaluation method for conversational search. The proposed embedding-based metric takes the influence of part of speech (POS) of the terms in the response into account. To the best knowledge, our work is the first to systematically demonstrate the importance of incorporating syntactic information, such as POS labels, for conversational search evaluation. Experimental results demonstrate that our metrics can correlate with human preference, achieving significant improvements over state-of-the-art baseline metrics.
△ Less
Submitted 7 September, 2021;
originally announced September 2021.
-
Meta-evaluation of Conversational Search Evaluation Metrics
Authors:
Zeyang Liu,
Ke Zhou,
Max L. Wilson
Abstract:
Conversational search systems, such as Google Assistant and Microsoft Cortana, enable users to interact with search systems in multiple rounds through natural language dialogues. Evaluating such systems is very challenging given that any natural language responses could be generated, and users commonly interact for multiple semantically coherent rounds to accomplish a search task. Although prior s…
▽ More
Conversational search systems, such as Google Assistant and Microsoft Cortana, enable users to interact with search systems in multiple rounds through natural language dialogues. Evaluating such systems is very challenging given that any natural language responses could be generated, and users commonly interact for multiple semantically coherent rounds to accomplish a search task. Although prior studies proposed many evaluation metrics, the extent of how those measures effectively capture user preference remains to be investigated. In this paper, we systematically meta-evaluate a variety of conversational search metrics. We specifically study three perspectives on those metrics: (1) reliability: the ability to detect "actual" performance differences as opposed to those observed by chance; (2) fidelity: the ability to agree with ultimate user preference; and (3) intuitiveness: the ability to capture any property deemed important: adequacy, informativeness, and fluency in the context of conversational search. By conducting experiments on two test collections, we find that the performance of different metrics varies significantly across different scenarios whereas consistent with prior studies, existing metrics only achieve a weak correlation with ultimate user preference and satisfaction. METEOR is, comparatively speaking, the best existing single-turn metric considering all three perspectives. We also demonstrate that adapted session-based evaluation metrics can be used to measure multi-turn conversational search, achieving moderate concordance with user satisfaction. To our knowledge, our work establishes the most comprehensive meta-evaluation for conversational search to date.
△ Less
Submitted 27 April, 2021;
originally announced April 2021.
-
Evaluating Collaborative Search Interfaces with Information Seeking Theory
Authors:
Max L. Wilson,
m. c. schraefel
Abstract:
Despite the many implicit references to the social aspects of search within Information Seeking and Retrieval research, there has been relatively little work that has specifically investigated the additional requirements for collaborative search software. In this paper we re-assess a recent evaluation framework, designed for individual information seeking experiences, to see a) how it could stil…
▽ More
Despite the many implicit references to the social aspects of search within Information Seeking and Retrieval research, there has been relatively little work that has specifically investigated the additional requirements for collaborative search software. In this paper we re-assess a recent evaluation framework, designed for individual information seeking experiences, to see a) how it could still be applied to collaborative search software; b) how it could produce additional requirements for collaborative search; and c) how it could be extended in future work to be even more appropriate for collaborative search evaluation. The position held after the assessment is that it can be used to evaluate collaborative search software, while providing new insights into their requirements. Finally, future work will validate the frameworks applicability to collaborative search and investigate roles within collaborative groups as a means to extend the framework.
△ Less
Submitted 5 August, 2009;
originally announced August 2009.
