-
DKPROMPT: Domain Knowledge Prompting Vision-Language Models for Open-World Planning
Authors:
Xiaohan Zhang,
Zainab Altaweel,
Yohei Hayamizu,
Yan Ding,
Saeid Amiri,
Hao Yang,
Andy Kaminski,
Chad Esselink,
Shiqi Zhang
Abstract:
Vision-language models (VLMs) have been applied to robot task planning problems, where the robot receives a task in natural language and generates plans based on visual inputs. While current VLMs have demonstrated strong vision-language understanding capabilities, their performance is still far from being satisfactory in planning tasks. At the same time, although classical task planners, such as P…
▽ More
Vision-language models (VLMs) have been applied to robot task planning problems, where the robot receives a task in natural language and generates plans based on visual inputs. While current VLMs have demonstrated strong vision-language understanding capabilities, their performance is still far from being satisfactory in planning tasks. At the same time, although classical task planners, such as PDDL-based, are strong in planning for long-horizon tasks, they do not work well in open worlds where unforeseen situations are common. In this paper, we propose a novel task planning and execution framework, called DKPROMPT, which automates VLM prompting using domain knowledge in PDDL for classical planning in open worlds. Results from quantitative experiments show that DKPROMPT outperforms classical planning, pure VLM-based and a few other competitive baselines in task completion rate.
△ Less
Submitted 25 June, 2024;
originally announced June 2024.
-
Deepfake tweets automatic detection
Authors:
Adam Frej,
Adrian Kaminski,
Piotr Marciniak,
Szymon Szmajdzinski,
Soveatin Kuntur,
Anna Wroblewska
Abstract:
This study addresses the critical challenge of detecting DeepFake tweets by leveraging advanced natural language processing (NLP) techniques to distinguish between genuine and AI-generated texts. Given the increasing prevalence of misinformation, our research utilizes the TweepFake dataset to train and evaluate various machine learning models. The objective is to identify effective strategies for…
▽ More
This study addresses the critical challenge of detecting DeepFake tweets by leveraging advanced natural language processing (NLP) techniques to distinguish between genuine and AI-generated texts. Given the increasing prevalence of misinformation, our research utilizes the TweepFake dataset to train and evaluate various machine learning models. The objective is to identify effective strategies for recognizing DeepFake content, thereby enhancing the integrity of digital communications. By develo** reliable methods for detecting AI-generated misinformation, this work contributes to a more trustworthy online information environment.
△ Less
Submitted 24 June, 2024;
originally announced June 2024.
-
Integrating Action Knowledge and LLMs for Task Planning and Situation Handling in Open Worlds
Authors:
Yan Ding,
Xiaohan Zhang,
Saeid Amiri,
Nieqing Cao,
Hao Yang,
Andy Kaminski,
Chad Esselink,
Shiqi Zhang
Abstract:
Task planning systems have been developed to help robots use human knowledge (about actions) to complete long-horizon tasks. Most of them have been developed for "closed worlds" while assuming the robot is provided with complete world knowledge. However, the real world is generally open, and the robots frequently encounter unforeseen situations that can potentially break the planner's completeness…
▽ More
Task planning systems have been developed to help robots use human knowledge (about actions) to complete long-horizon tasks. Most of them have been developed for "closed worlds" while assuming the robot is provided with complete world knowledge. However, the real world is generally open, and the robots frequently encounter unforeseen situations that can potentially break the planner's completeness. Could we leverage the recent advances on pre-trained Large Language Models (LLMs) to enable classical planning systems to deal with novel situations?
This paper introduces a novel framework, called COWP, for open-world task planning and situation handling. COWP dynamically augments the robot's action knowledge, including the preconditions and effects of actions, with task-oriented commonsense knowledge. COWP embraces the openness from LLMs, and is grounded to specific domains via action knowledge. For systematic evaluations, we collected a dataset that includes 1,085 execution-time situations. Each situation corresponds to a state instance wherein a robot is potentially unable to complete a task using a solution that normally works. Experimental results show that our approach outperforms competitive baselines from the literature in the success rate of service tasks. Additionally, we have demonstrated COWP using a mobile manipulator. Supplementary materials are available at: https://cowplanning.github.io/
△ Less
Submitted 5 October, 2023; v1 submitted 27 May, 2023;
originally announced May 2023.
-
Grounding Classical Task Planners via Vision-Language Models
Authors:
Xiaohan Zhang,
Yan Ding,
Saeid Amiri,
Hao Yang,
Andy Kaminski,
Chad Esselink,
Shiqi Zhang
Abstract:
Classical planning systems have shown great advances in utilizing rule-based human knowledge to compute accurate plans for service robots, but they face challenges due to the strong assumptions of perfect perception and action executions. To tackle these challenges, one solution is to connect the symbolic states and actions generated by classical planners to the robot's sensory observations, thus…
▽ More
Classical planning systems have shown great advances in utilizing rule-based human knowledge to compute accurate plans for service robots, but they face challenges due to the strong assumptions of perfect perception and action executions. To tackle these challenges, one solution is to connect the symbolic states and actions generated by classical planners to the robot's sensory observations, thus closing the perception-action loop. This research proposes a visually-grounded planning framework, named TPVQA, which leverages Vision-Language Models (VLMs) to detect action failures and verify action affordances towards enabling successful plan execution. Results from quantitative experiments show that TPVQA surpasses competitive baselines from previous studies in task completion rate.
△ Less
Submitted 19 June, 2023; v1 submitted 17 April, 2023;
originally announced April 2023.
