CaptainCook4D: A dataset for understanding errors in procedural activities
Authors:
Rohith Peddi,
Shivvrat Arya,
Bharath Challa,
Likhitha Pallapothula,
Akshay Vyas,
Jikai Wang,
Qifan Zhang,
Vasundhara Komaragiri,
Eric Ragan,
Nicholas Ruozzi,
Yu Xiang,
Vibhav Gogate
Abstract:
Following step-by-step procedures is an essential component of various activities carried out by individuals in their daily lives. These procedures serve as a guiding framework that helps to achieve goals efficiently, whether it is assembling furniture or preparing a recipe. However, the complexity and duration of procedural activities inherently increase the likelihood of making errors. Understan…
▽ More
Following step-by-step procedures is an essential component of various activities carried out by individuals in their daily lives. These procedures serve as a guiding framework that helps to achieve goals efficiently, whether it is assembling furniture or preparing a recipe. However, the complexity and duration of procedural activities inherently increase the likelihood of making errors. Understanding such procedural activities from a sequence of frames is a challenging task that demands an accurate interpretation of visual information and the ability to reason about the structure of the activity. To this end, we collect a new egocentric 4D dataset, CaptainCook4D, comprising 384 recordings (94.5 hours) of people performing recipes in real kitchen environments. This dataset consists of two distinct types of activity: one in which participants adhere to the provided recipe instructions and another in which they deviate and induce errors. We provide 5.3K step annotations and 10K fine-grained action annotations and benchmark the dataset for the following tasks: supervised error recognition, multistep localization, and procedure learning
△ Less
Submitted 22 December, 2023;
originally announced December 2023.
