ForTune: Running Offline Scenarios to Estimate Impact on Business Metrics
Authors:
Georges Dupret,
Konstantin Sozinov,
Carmen Barcena Gonzalez,
Ziggy Zacks,
Amber Yuan,
Benjamin Carterette,
Manuel Mai,
Shubham Bansal,
Gwo Liang,
Lien,
Andrey Gatash,
Roberto Sanchis Ojeda,
Mounia Lalmas
Abstract:
Making ideal decisions as a product leader in a web-facing company is extremely difficult. In addition to navigating the ambiguity of customer satisfaction and achieving business goals, one must also pave a path forward for ones' products and services to remain relevant, desirable, and profitable. Data and experimentation to test product hypotheses are key to informing product decisions. Online co…
▽ More
Making ideal decisions as a product leader in a web-facing company is extremely difficult. In addition to navigating the ambiguity of customer satisfaction and achieving business goals, one must also pave a path forward for ones' products and services to remain relevant, desirable, and profitable. Data and experimentation to test product hypotheses are key to informing product decisions. Online controlled experiments by A/B testing may provide the best data to support such decisions with high confidence, but can be time-consuming and expensive, especially when one wants to understand impact to key business metrics such as retention or long-term value. Offline experimentation allows one to rapidly iterate and test, but often cannot provide the same level of confidence, and cannot easily shine a light on impact on business metrics. We introduce a novel, lightweight, and flexible approach to investigating hypotheses, called scenario analysis, that aims to support product leaders' decisions using data about users and estimates of business metrics. Its strengths are that it can provide guidance on trade-offs that are incurred by growing or shifting consumption, estimate trends in long-term outcomes like retention and other important business metrics, and can generate hypotheses about relationships between metrics at scale.
△ Less
Submitted 29 February, 2024;
originally announced March 2024.
Beyond Cumulated Gain and Average Precision: Including Willingness and Expectation in the User Model
Authors:
Benjamin Piwowarski,
Georges Dupret,
Mounia Lalmas
Abstract:
In this paper, we define a new metric family based on two concepts: The definition of the stop** criterion and the notion of satisfaction, where the former depends on the willingness and expectation of a user exploring search results. Both concepts have been discussed so far in the IR literature, but we argue in this paper that defining a proper single valued metric depends on merging them into…
▽ More
In this paper, we define a new metric family based on two concepts: The definition of the stop** criterion and the notion of satisfaction, where the former depends on the willingness and expectation of a user exploring search results. Both concepts have been discussed so far in the IR literature, but we argue in this paper that defining a proper single valued metric depends on merging them into a single conceptual framework.
△ Less
Submitted 20 September, 2012;
originally announced September 2012.
Learning to Rank Query Recommendations by Semantic Similarities
Authors:
Sumio Fujita,
Georges Dupret,
Ricardo Baeza-Yates
Abstract:
Logs of the interactions with a search engine show that users often reformulate their queries. Examining these reformulations shows that recommendations that precise the focus of a query are helpful, like those based on expansions of the original queries. But it also shows that queries that express some topical shift with respect to the original query can help user access more rapidly the informat…
▽ More
Logs of the interactions with a search engine show that users often reformulate their queries. Examining these reformulations shows that recommendations that precise the focus of a query are helpful, like those based on expansions of the original queries. But it also shows that queries that express some topical shift with respect to the original query can help user access more rapidly the information they need. We propose a method to identify from the query logs of past users queries that either focus or shift the initial query topic. This method combines various click-based, topic-based and session based ranking strategies and uses supervised learning in order to maximize the semantic similarities between the query and the recommendations, while at the same diversifying them. We evaluate our method using the query/click logs of a Japanese web search engine and we show that the combination of the three methods proposed is significantly better than any of them taken individually.
△ Less
Submitted 12 April, 2012;
originally announced April 2012.
