Physion: Evaluating Physical Prediction from Vision in Humans and Machines
Authors:
Daniel M. Bear,
Elias Wang,
Damian Mrowca,
Felix J. Binder,
Hsiao-Yu Fish Tung,
R. T. Pramod,
Cameron Holdaway,
Sirui Tao,
Kevin Smith,
Fan-Yun Sun,
Li Fei-Fei,
Nancy Kanwisher,
Joshua B. Tenenbaum,
Daniel L. K. Yamins,
Judith E. Fan
Abstract:
While current vision algorithms excel at many challenging tasks, it is unclear how well they understand the physical dynamics of real-world environments. Here we introduce Physion, a dataset and benchmark for rigorously evaluating the ability to predict how physical scenarios will evolve over time. Our dataset features realistic simulations of a wide range of physical phenomena, including rigid an…
▽ More
While current vision algorithms excel at many challenging tasks, it is unclear how well they understand the physical dynamics of real-world environments. Here we introduce Physion, a dataset and benchmark for rigorously evaluating the ability to predict how physical scenarios will evolve over time. Our dataset features realistic simulations of a wide range of physical phenomena, including rigid and soft-body collisions, stable multi-object configurations, rolling, sliding, and projectile motion, thus providing a more comprehensive challenge than previous benchmarks. We used Physion to benchmark a suite of models varying in their architecture, learning objective, input-output structure, and training data. In parallel, we obtained precise measurements of human prediction behavior on the same set of scenarios, allowing us to directly evaluate how well any model could approximate human behavior. We found that vision algorithms that learn object-centric representations generally outperform those that do not, yet still fall far short of human performance. On the other hand, graph neural networks with direct access to physical state information both perform substantially better and make predictions that are more similar to those made by humans. These results suggest that extracting physical representations of scenes is the main bottleneck to achieving human-level and human-like physical understanding in vision algorithms. We have publicly released all data and code to facilitate the use of Physion to benchmark additional models in a fully reproducible manner, enabling systematic evaluation of progress towards vision algorithms that understand physical environments as robustly as people do.
△ Less
Submitted 20 June, 2022; v1 submitted 15 June, 2021;
originally announced June 2021.
Visual sco** operations for physical assembly
Authors:
Felix J Binder,
Marcelo M Mattar,
David Kirsh,
Judith E Fan
Abstract:
Planning is hard. The use of subgoals can make planning more tractable, but selecting these subgoals is computationally costly. What algorithms might enable us to reap the benefits of planning using subgoals while minimizing the computational overhead of selecting them? We propose visual sco**, a strategy that interleaves planning and acting by alternately defining a spatial region as the next s…
▽ More
Planning is hard. The use of subgoals can make planning more tractable, but selecting these subgoals is computationally costly. What algorithms might enable us to reap the benefits of planning using subgoals while minimizing the computational overhead of selecting them? We propose visual sco**, a strategy that interleaves planning and acting by alternately defining a spatial region as the next subgoal and selecting actions to achieve it. We evaluated our visual sco** algorithm on a variety of physical assembly problems against two baselines: planning all subgoals in advance and planning without subgoals. We found that visual sco** achieves comparable task performance to the subgoal planner while requiring only a fraction of the total computational cost. Together, these results contribute to our understanding of how humans might make efficient use of cognitive resources to solve complex planning problems.
△ Less
Submitted 4 August, 2021; v1 submitted 10 June, 2021;
originally announced June 2021.
