VideoPoet: A Large Language Model for Zero-Shot Video Generation
Authors:
Dan Kondratyuk,
Lijun Yu,
Xiuye Gu,
José Lezama,
Jonathan Huang,
Grant Schindler,
Rachel Hornung,
Vighnesh Birodkar,
Jimmy Yan,
Ming-Chang Chiu,
Krishna Somandepalli,
Hassan Akbari,
Yair Alon,
Yong Cheng,
Josh Dillon,
Agrim Gupta,
Meera Hahn,
Anja Hauth,
David Hendon,
Alonso Martinez,
David Minnen,
Mikhail Sirotenko,
Kihyuk Sohn,
Xuan Yang,
Hartwig Adam
, et al. (6 additional authors not shown)
Abstract:
We present VideoPoet, a language model capable of synthesizing high-quality video, with matching audio, from a large variety of conditioning signals. VideoPoet employs a decoder-only transformer architecture that processes multimodal inputs -- including images, videos, text, and audio. The training protocol follows that of Large Language Models (LLMs), consisting of two stages: pretraining and tas…
▽ More
We present VideoPoet, a language model capable of synthesizing high-quality video, with matching audio, from a large variety of conditioning signals. VideoPoet employs a decoder-only transformer architecture that processes multimodal inputs -- including images, videos, text, and audio. The training protocol follows that of Large Language Models (LLMs), consisting of two stages: pretraining and task-specific adaptation. During pretraining, VideoPoet incorporates a mixture of multimodal generative objectives within an autoregressive Transformer framework. The pretrained LLM serves as a foundation that can be adapted for a range of video generation tasks. We present empirical results demonstrating the model's state-of-the-art capabilities in zero-shot video generation, specifically highlighting VideoPoet's ability to generate high-fidelity motions. Project page: http://sites.research.google/videopoet/
△ Less
Submitted 4 June, 2024; v1 submitted 21 December, 2023;
originally announced December 2023.
Learning to Detect Touches on Cluttered Tables
Authors:
Norberto Adrian Goussies,
Kenji Hata,
Shruthi Prabhakara,
Abhishek Amit,
Tony Aube,
Carl Cepress,
Diana Chang,
Li-Te Cheng,
Horia Stefan Ciurdar,
Mike Cleron,
Chelsey Fleming,
Ashwin Ganti,
Divyansh Garg,
Niloofar Gheissari,
Petra Luna Grutzik,
David Hendon,
Daniel Iglesia,
** Kim,
Stuart Kyle,
Chris LaRosa,
Roman Lewkow,
Peter F McDermott,
Chris Melancon,
Paru Nackeeran,
Neal Norwitz
, et al. (6 additional authors not shown)
Abstract:
We present a novel self-contained camera-projector tabletop system with a lamp form-factor that brings digital intelligence to our tables. We propose a real-time, on-device, learning-based touch detection algorithm that makes any tabletop interactive. The top-down configuration and learning-based algorithm makes our method robust to the presence of clutter, a main limitation of existing camera-pro…
▽ More
We present a novel self-contained camera-projector tabletop system with a lamp form-factor that brings digital intelligence to our tables. We propose a real-time, on-device, learning-based touch detection algorithm that makes any tabletop interactive. The top-down configuration and learning-based algorithm makes our method robust to the presence of clutter, a main limitation of existing camera-projector tabletop systems. Our research prototype enables a set of experiences that combine hand interactions and objects present on the table. A video can be found at https://youtu.be/hElC_c25Fg8.
△ Less
Submitted 10 April, 2023;
originally announced April 2023.
