When the Ground Truth is not True: Modelling Human Biases in Temporal Annotations
Authors:
Taku Yamagata,
Emma L. Tonkin,
Benjamin Arana Sanchez,
Ian Craddock,
Miquel Perello Nieto,
Raul Santos-Rodriguez,
Weisong Yang,
Peter Flach
Abstract:
In supervised learning, low quality annotations lead to poorly performing classification and detection models, while also rendering evaluation unreliable. This is particularly apparent on temporal data, where annotation quality is affected by multiple factors. For example, in the post-hoc self-reporting of daily activities, cognitive biases are one of the most common ingredients. In particular, re…
▽ More
In supervised learning, low quality annotations lead to poorly performing classification and detection models, while also rendering evaluation unreliable. This is particularly apparent on temporal data, where annotation quality is affected by multiple factors. For example, in the post-hoc self-reporting of daily activities, cognitive biases are one of the most common ingredients. In particular, reporting the start and duration of an activity after its finalisation may incorporate biases introduced by personal time perceptions, as well as the imprecision and lack of granularity due to time rounding. Here we propose a method to model human biases on temporal annotations and argue for the use of soft labels. Experimental results in synthetic data show that soft labels provide a better approximation of the ground truth for several metrics. We showcase the method on a real dataset of daily activities.
△ Less
Submitted 6 February, 2023;
originally announced February 2023.
