-
Demo: Soccer Information Retrieval via Natural Queries using SoccerRAG
Authors:
Aleksander Theo Strand,
Sushant Gautam,
Cise Midoglu,
Pål Halvorsen
Abstract:
The rapid evolution of digital sports media necessitates sophisticated information retrieval systems that can efficiently parse extensive multimodal datasets. This paper demonstrates SoccerRAG, an innovative framework designed to harness the power of Retrieval Augmented Generation (RAG) and Large Language Models (LLMs) to extract soccer-related information through natural language queries. By leve…
▽ More
The rapid evolution of digital sports media necessitates sophisticated information retrieval systems that can efficiently parse extensive multimodal datasets. This paper demonstrates SoccerRAG, an innovative framework designed to harness the power of Retrieval Augmented Generation (RAG) and Large Language Models (LLMs) to extract soccer-related information through natural language queries. By leveraging a multimodal dataset, SoccerRAG supports dynamic querying and automatic data validation, enhancing user interaction and accessibility to sports archives. We present a novel interactive user interface (UI) based on the Chainlit framework which wraps around the core functionality, and enable users to interact with the SoccerRAG framework in a chatbot-like visual manner.
△ Less
Submitted 3 June, 2024;
originally announced June 2024.
-
SoccerRAG: Multimodal Soccer Information Retrieval via Natural Queries
Authors:
Aleksander Theo Strand,
Sushant Gautam,
Cise Midoglu,
Pål Halvorsen
Abstract:
The rapid evolution of digital sports media necessitates sophisticated information retrieval systems that can efficiently parse extensive multimodal datasets. This paper introduces SoccerRAG, an innovative framework designed to harness the power of Retrieval Augmented Generation (RAG) and Large Language Models (LLMs) to extract soccer-related information through natural language queries. By levera…
▽ More
The rapid evolution of digital sports media necessitates sophisticated information retrieval systems that can efficiently parse extensive multimodal datasets. This paper introduces SoccerRAG, an innovative framework designed to harness the power of Retrieval Augmented Generation (RAG) and Large Language Models (LLMs) to extract soccer-related information through natural language queries. By leveraging a multimodal dataset, SoccerRAG supports dynamic querying and automatic data validation, enhancing user interaction and accessibility to sports archives. Our evaluations indicate that SoccerRAG effectively handles complex queries, offering significant improvements over traditional retrieval systems in terms of accuracy and user engagement. The results underscore the potential of using RAG and LLMs in sports analytics, paving the way for future advancements in the accessibility and real-time processing of sports data.
△ Less
Submitted 3 June, 2024;
originally announced June 2024.
-
SoccerNet-Echoes: A Soccer Game Audio Commentary Dataset
Authors:
Sushant Gautam,
Mehdi Houshmand Sarkhoosh,
Jan Held,
Cise Midoglu,
Anthony Cioppa,
Silvio Giancola,
Vajira Thambawita,
Michael A. Riegler,
Pål Halvorsen,
Mubarak Shah
Abstract:
The application of Automatic Speech Recognition (ASR) technology in soccer offers numerous opportunities for sports analytics. Specifically, extracting audio commentaries with ASR provides valuable insights into the events of the game, and opens the door to several downstream applications such as automatic highlight generation. This paper presents SoccerNet-Echoes, an augmentation of the SoccerNet…
▽ More
The application of Automatic Speech Recognition (ASR) technology in soccer offers numerous opportunities for sports analytics. Specifically, extracting audio commentaries with ASR provides valuable insights into the events of the game, and opens the door to several downstream applications such as automatic highlight generation. This paper presents SoccerNet-Echoes, an augmentation of the SoccerNet dataset with automatically generated transcriptions of audio commentaries from soccer game broadcasts, enhancing video content with rich layers of textual information derived from the game audio using ASR. These textual commentaries, generated using the Whisper model and translated with Google Translate, extend the usefulness of the SoccerNet dataset in diverse applications such as enhanced action spotting, automatic caption generation, and game summarization. By incorporating textual data alongside visual and auditory content, SoccerNet-Echoes aims to serve as a comprehensive resource for the development of algorithms specialized in capturing the dynamics of soccer games. We detail the methods involved in the curation of this dataset and the integration of ASR. We also highlight the implications of a multimodal approach in sports analytics, and how the enriched dataset can support diverse applications, thus broadening the scope of research and development in the field of sports analytics.
△ Less
Submitted 12 May, 2024;
originally announced May 2024.
-
Soccer on Social Media
Authors:
Mehdi Houshmand Sarkhoosh,
Sayed Mohammad Majidi Dorcheh,
Sushant Gautam,
Cise Midoglu,
Saeed Shafiee Sabet,
Pål Halvorsen
Abstract:
In the era of digitalization, social media has become an integral part of our lives, serving as a significant hub for individuals and businesses to share information, communicate, and engage. This is also the case for professional sports, where leagues, clubs and players are using social media to reach out to their fans. In this respect, a huge amount of time is spent curating multimedia content f…
▽ More
In the era of digitalization, social media has become an integral part of our lives, serving as a significant hub for individuals and businesses to share information, communicate, and engage. This is also the case for professional sports, where leagues, clubs and players are using social media to reach out to their fans. In this respect, a huge amount of time is spent curating multimedia content for various social media platforms and their target users. With the emergence of Artificial Intelligence (AI), AI-based tools for automating content generation and enhancing user experiences on social media have become widely popular. However, to effectively utilize such tools, it is imperative to comprehend the demographics and preferences of users on different platforms, understand how content providers post information in these channels, and how different types of multimedia are consumed by audiences. This report presents an analysis of social media platforms, in terms of demographics, supported multimedia modalities, and distinct features and specifications for different modalities, followed by a comparative case study of select European soccer leagues and teams, in terms of their social media practices. Through this analysis, we demonstrate that social media, while being very important for and widely used by supporters from all ages, also requires a fine-tuned effort on the part of soccer professionals, in order to elevate fan experiences and foster engagement.
△ Less
Submitted 18 October, 2023;
originally announced October 2023.
-
Grand Challenge On Detecting Cheapfakes
Authors:
Duc-Tien Dang-Nguyen,
Sohail Ahmed Khan,
Cise Midoglu,
Michael Riegler,
Pål Halvorsen,
Minh-Son Dao
Abstract:
Cheapfake is a recently coined term that encompasses non-AI ("cheap") manipulations of multimedia content. Cheapfakes are known to be more prevalent than deepfakes. Cheapfake media can be created using editing software for image/video manipulations, or even without using any software, by simply altering the context of an image/video by sharing the media alongside misleading claims. This alteration…
▽ More
Cheapfake is a recently coined term that encompasses non-AI ("cheap") manipulations of multimedia content. Cheapfakes are known to be more prevalent than deepfakes. Cheapfake media can be created using editing software for image/video manipulations, or even without using any software, by simply altering the context of an image/video by sharing the media alongside misleading claims. This alteration of context is referred to as out-of-context (OOC) misuse of media. OOC media is much harder to detect than fake media, since the images and videos are not tampered. In this challenge, we focus on detecting OOC images, and more specifically the misuse of real photographs with conflicting image captions in news items. The aim of this challenge is to develop and benchmark models that can be used to detect whether given samples (news image and associated captions) are OOC, based on the recently compiled COSMOS dataset.
△ Less
Submitted 3 April, 2023;
originally announced April 2023.
-
ACM Multimedia Grand Challenge on Detecting Cheapfakes
Authors:
Shivangi Aneja,
Cise Midoglu,
Duc-Tien Dang-Nguyen,
Sohail Ahmed Khan,
Michael Riegler,
Pål Halvorsen,
Chris Bregler,
Balu Adsumilli
Abstract:
Cheapfake is a recently coined term that encompasses non-AI (``cheap'') manipulations of multimedia content. Cheapfakes are known to be more prevalent than deepfakes. Cheapfake media can be created using editing software for image/video manipulations, or even without using any software, by simply altering the context of an image/video by sharing the media alongside misleading claims. This alterati…
▽ More
Cheapfake is a recently coined term that encompasses non-AI (``cheap'') manipulations of multimedia content. Cheapfakes are known to be more prevalent than deepfakes. Cheapfake media can be created using editing software for image/video manipulations, or even without using any software, by simply altering the context of an image/video by sharing the media alongside misleading claims. This alteration of context is referred to as out-of-context (OOC) misuse of media. OOC media is much harder to detect than fake media, since the images and videos are not tampered. In this challenge, we focus on detecting OOC images, and more specifically the misuse of real photographs with conflicting image captions in news items. The aim of this challenge is to develop and benchmark models that can be used to detect whether given samples (news image and associated captions) are OOC, based on the recently compiled COSMOS dataset.
△ Less
Submitted 29 July, 2022;
originally announced July 2022.
-
Visual explanations for polyp detection: How medical doctors assess intrinsic versus extrinsic explanations
Authors:
Steven Hicks,
Andrea Storås,
Michael Riegler,
Cise Midoglu,
Malek Hammou,
Thomas de Lange,
Sravanthi Parasa,
Pål Halvorsen,
Inga Strümke
Abstract:
Deep learning has in recent years achieved immense success in all areas of computer vision and has the potential of assisting medical doctors in analyzing visual content for disease and other abnormalities. However, the current state of deep learning is very much a black box, making medical professionals highly skeptical about integrating these methods into clinical practice. Several methods have…
▽ More
Deep learning has in recent years achieved immense success in all areas of computer vision and has the potential of assisting medical doctors in analyzing visual content for disease and other abnormalities. However, the current state of deep learning is very much a black box, making medical professionals highly skeptical about integrating these methods into clinical practice. Several methods have been proposed in order to shine some light onto these black boxes, but there is no consensus on the opinion of the medical doctors that will consume these explanations. This paper presents a study asking medical doctors about their opinion of current state-of-the-art explainable artificial intelligence methods when applied to a gastrointestinal disease detection use case. We compare two different categories of explanation methods, intrinsic and extrinsic, and gauge their opinion of the current value of these explanations. The results indicate that intrinsic explanations are preferred and that explanation.
△ Less
Submitted 23 March, 2022;
originally announced April 2022.
-
MMSys'22 Grand Challenge on AI-based Video Production for Soccer
Authors:
Cise Midoglu,
Steven A. Hicks,
Vajira Thambawita,
Tomas Kupka,
Pål Halvorsen
Abstract:
Soccer has a considerable market share of the global sports industry, and the interest in viewing videos from soccer games continues to grow. In this respect, it is important to provide game summaries and highlights of the main game events. However, annotating and producing events and summaries often require expensive equipment and a lot of tedious, cumbersome, manual labor. Therefore, automating…
▽ More
Soccer has a considerable market share of the global sports industry, and the interest in viewing videos from soccer games continues to grow. In this respect, it is important to provide game summaries and highlights of the main game events. However, annotating and producing events and summaries often require expensive equipment and a lot of tedious, cumbersome, manual labor. Therefore, automating the video production pipeline providing fast game highlights at a much lower cost is seen as the "holy grail". In this context, recent developments in Artificial Intelligence (AI) technology have shown great potential. Still, state-of-the-art approaches are far from being adequate for practical scenarios that have demanding real-time requirements, as well as strict performance criteria (where at least the detection of official events such as goals and cards must be 100% accurate). In addition, event detection should be thoroughly enhanced by annotation and classification, proper clip**, generating short descriptions, selecting appropriate thumbnails for highlight clips, and finally, combining the event highlights into an overall game summary, similar to what is commonly aired during sports news. Even though the event tagging operation has by far received the most attention, an end-to-end video production pipeline also includes various other operations which serve the overall purpose of automated soccer analysis. This challenge aims to assist the automation of such a production pipeline using AI. In particular, we focus on the enhancement operations that take place after an event has been detected, namely event clip** (Task 1), thumbnail selection (Task 2), and game summarization (Task 3). Challenge website: https://mmsys2022.ie/authors/grand-challenge.
△ Less
Submitted 2 February, 2022;
originally announced February 2022.
-
MMSys'21 Grand Challenge on Detecting Cheapfakes
Authors:
Shivangi Aneja,
Cise Midoglu,
Duc-Tien Dang-Nguyen,
Michael Alexander Riegler,
Paal Halvorsen,
Matthias Niessner,
Balu Adsumilli,
Chris Bregler
Abstract:
Cheapfake is a recently coined term that encompasses non-AI ("cheap") manipulations of multimedia content. Cheapfakes are known to be more prevalent than deepfakes. Cheapfake media can be created using editing software for image/video manipulations, or even without using any software, by simply altering the context of an image/video by sharing the media alongside misleading claims. This alteration…
▽ More
Cheapfake is a recently coined term that encompasses non-AI ("cheap") manipulations of multimedia content. Cheapfakes are known to be more prevalent than deepfakes. Cheapfake media can be created using editing software for image/video manipulations, or even without using any software, by simply altering the context of an image/video by sharing the media alongside misleading claims. This alteration of context is referred to as out-of-context (OOC) misuse} of media. OOC media is much harder to detect than fake media, since the images and videos are not tampered. In this challenge, we focus on detecting OOC images, and more specifically the misuse of real photographs with conflicting image captions in news items. The aim of this challenge is to develop and benchmark models that can be used to detect whether given samples (news image and associated captions) are OOC, based on the recently compiled COSMOS dataset.
△ Less
Submitted 12 July, 2021;
originally announced July 2021.
-
MONROE-Nettest: A Configurable Tool for Dissecting Speed Measurements in Mobile Broadband Networks
Authors:
Cise Midoglu,
Leonhard Wimmer,
Andra Lutu,
Ozgu Alay,
Carsten Griwodz
Abstract:
As the demand for mobile connectivity continues to grow, there is a strong need to evaluate the performance of Mobile Broadband (MBB) networks. In the last years, mobile "speed", quantified most commonly by data rate, gained popularity as the widely accepted metric to describe their performance. However, there is a lack of consensus on how mobile speed should be measured. In this paper, we design…
▽ More
As the demand for mobile connectivity continues to grow, there is a strong need to evaluate the performance of Mobile Broadband (MBB) networks. In the last years, mobile "speed", quantified most commonly by data rate, gained popularity as the widely accepted metric to describe their performance. However, there is a lack of consensus on how mobile speed should be measured. In this paper, we design and implement MONROE-Nettest to dissect mobile speed measurements, and investigate the effect of different factors on speed measurements in the complex mobile ecosystem. MONROE-Nettest is built as an Experiment as a Service (EaaS) on top of the MONROE platform, an open dedicated platform for experimentation in operational MBB networks. Using MONROE-Nettest, we conduct a large scale measurement campaign and quantify the effects of measurement duration, number of TCP flows, and server location on measured downlink data rate in 6 operational MBB networks in Europe. Our results indicate that differences in parameter configuration can significantly affect the measurement results. We provide the complete MONROE-Nettest toolset as open source and our measurements as open data.
△ Less
Submitted 30 January, 2018; v1 submitted 21 October, 2017;
originally announced October 2017.
