-
Multitask Learning to Improve Egocentric Action Recognition
Authors:
Georgios Kapidis,
Ronald Poppe,
Elsbeth van Dam,
Lucas Noldus,
Remco Veltkamp
Abstract:
In this work we employ multitask learning to capitalize on the structure that exists in related supervised tasks to train complex neural networks. It allows training a network for multiple objectives in parallel, in order to improve performance on at least one of them by capitalizing on a shared representation that is developed to accommodate more information than it otherwise would for a single t…
▽ More
In this work we employ multitask learning to capitalize on the structure that exists in related supervised tasks to train complex neural networks. It allows training a network for multiple objectives in parallel, in order to improve performance on at least one of them by capitalizing on a shared representation that is developed to accommodate more information than it otherwise would for a single task. We employ this idea to tackle action recognition in egocentric videos by introducing additional supervised tasks. We consider learning the verbs and nouns from which action labels consist of and predict coordinates that capture the hand locations and the gaze-based visual saliency for all the frames of the input video segments. This forces the network to explicitly focus on cues from secondary tasks that it might otherwise have missed resulting in improved inference. Our experiments on EPIC-Kitchens and EGTEA Gaze+ show consistent improvements when training with multiple tasks over the single-task baseline. Furthermore, in EGTEA Gaze+ we outperform the state-of-the-art in action recognition by 3.84%. Apart from actions, our method produces accurate hand and gaze estimations as side tasks, without requiring any additional input at test time other than the RGB video clips.
△ Less
Submitted 15 September, 2019;
originally announced September 2019.
-
Egocentric Hand Track and Object-based Human Action Recognition
Authors:
Georgios Kapidis,
Ronald Poppe,
Elsbeth van Dam,
Lucas P. J. J. Noldus,
Remco C. Veltkamp
Abstract:
Egocentric vision is an emerging field of computer vision that is characterized by the acquisition of images and video from the first person perspective. In this paper we address the challenge of egocentric human action recognition by utilizing the presence and position of detected regions of interest in the scene explicitly, without further use of visual features.
Initially, we recognize that h…
▽ More
Egocentric vision is an emerging field of computer vision that is characterized by the acquisition of images and video from the first person perspective. In this paper we address the challenge of egocentric human action recognition by utilizing the presence and position of detected regions of interest in the scene explicitly, without further use of visual features.
Initially, we recognize that human hands are essential in the execution of actions and focus on obtaining their movements as the principal cues that define actions. We employ object detection and region tracking techniques to locate hands and capture their movements. Prior knowledge about egocentric views facilitates hand identification between left and right. With regard to detection and tracking, we contribute a pipeline that successfully operates on unseen egocentric videos to find the camera wearer's hands and associate them through time. Moreover, we emphasize on the value of scene information for action recognition. We acknowledge that the presence of objects is significant for the execution of actions by humans and in general for the description of a scene. To acquire this information, we utilize object detection for specific classes that are relevant to the actions we want to recognize.
Our experiments are targeted on videos of kitchen activities from the Epic-Kitchens dataset. We model action recognition as a sequence learning problem of the detected spatial positions in the frames. Our results show that explicit hand and object detections with no other visual information can be relied upon to classify hand-related human actions. Testing against methods fully dependent on visual features, signals that for actions where hand motions are conceptually important, a region-of-interest-based description of a video contains equally expressive information with comparable classification performance.
△ Less
Submitted 2 May, 2019;
originally announced May 2019.
-
On bounding the bandwidth of graphs with symmetry
Authors:
Edwin R. van Dam,
Renata Sotirov
Abstract:
We derive a new lower bound for the bandwidth of a graph that is based on a new lower bound for the minimum cut problem. Our new semidefinite programming relaxation of the minimum cut problem is obtained by strengthening the known semidefinite programming relaxation for the quadratic assignment problem (or for the graph partition problem) by fixing two vertices in the graph; one on each side of th…
▽ More
We derive a new lower bound for the bandwidth of a graph that is based on a new lower bound for the minimum cut problem. Our new semidefinite programming relaxation of the minimum cut problem is obtained by strengthening the known semidefinite programming relaxation for the quadratic assignment problem (or for the graph partition problem) by fixing two vertices in the graph; one on each side of the cut. This fixing results in several smaller subproblems that need to be solved to obtain the new bound. In order to efficiently solve these subproblems we exploit symmetry in the data; that is, both symmetry in the min-cut problem and symmetry in the graphs. To obtain upper bounds for the bandwidth of graphs with symmetry, we develop a heuristic approach based on the well-known reverse Cuthill-McKee algorithm, and that improves significantly its performance on the tested graphs. Our approaches result in the best known lower and upper bounds for the bandwidth of all graphs under consideration, i.e., Hamming graphs, 3-dimensional generalized Hamming graphs, Johnson graphs, and Kneser graphs, with up to 216 vertices.
△ Less
Submitted 2 January, 2015; v1 submitted 4 December, 2012;
originally announced December 2012.
