-
Materials In Paintings (MIP): An interdisciplinary dataset for perception, art history, and computer vision
Authors:
Mitchell J. P. van Zuijlen,
Hubert Lin,
Kavita Bala,
Sylvia C. Pont,
Maarten W. A. Wijntjes
Abstract:
A painter is free to modify how components of a natural scene are depicted, which can lead to a perceptually convincing image of the distal world. This signals a major difference between photos and paintings: paintings are explicitly created for human perception. Studying these painterly depictions could be beneficial to a multidisciplinary audience. In this paper, we capture and explore the paint…
▽ More
A painter is free to modify how components of a natural scene are depicted, which can lead to a perceptually convincing image of the distal world. This signals a major difference between photos and paintings: paintings are explicitly created for human perception. Studying these painterly depictions could be beneficial to a multidisciplinary audience. In this paper, we capture and explore the painterly depictions of materials to enable the study of depiction and perception of materials through the artists' eye. We annotated a dataset of 19k paintings with 200k+ bounding boxes from which polygon segments were automatically extracted. Each bounding box was assigned a coarse label (e.g., fabric) and a fine-grained label (e.g., velvety, silky). We demonstrate the cross-disciplinary utility of our dataset by presenting novel findings across art history, human perception, and computer vision. Our experiments include analyzing the distribution of materials depicted in paintings, showing how painters create convincing depictions using a stylized approach, and demonstrating how paintings can be used to build more robust computer vision models. We conclude that our dataset of painterly material depictions is a rich source for gaining insights into the depiction and perception of materials across multiple disciplines. The MIP dataset is freely accessible at https://materialsinpaintings.tudelft.nl
△ Less
Submitted 10 December, 2020; v1 submitted 5 December, 2020;
originally announced December 2020.
-
What Can Style Transfer and Paintings Do For Model Robustness?
Authors:
Hubert Lin,
Mitchell van Zuijlen,
Sylvia C. Pont,
Maarten W. A. Wijntjes,
Kavita Bala
Abstract:
A common strategy for improving model robustness is through data augmentations. Data augmentations encourage models to learn desired invariances, such as invariance to horizontal flip** or small changes in color. Recent work has shown that arbitrary style transfer can be used as a form of data augmentation to encourage invariance to textures by creating painting-like images from photographs. How…
▽ More
A common strategy for improving model robustness is through data augmentations. Data augmentations encourage models to learn desired invariances, such as invariance to horizontal flip** or small changes in color. Recent work has shown that arbitrary style transfer can be used as a form of data augmentation to encourage invariance to textures by creating painting-like images from photographs. However, a stylized photograph is not quite the same as an artist-created painting. Artists depict perceptually meaningful cues in paintings so that humans can recognize salient components in scenes, an emphasis which is not enforced in style transfer. Therefore, we study how style transfer and paintings differ in their impact on model robustness. First, we investigate the role of paintings as style images for stylization-based data augmentation. We find that style transfer functions well even without paintings as style images. Second, we show that learning from paintings as a form of perceptual data augmentation can improve model robustness. Finally, we investigate the invariances learned from stylization and from paintings, and show that models learn different invariances from these differing forms of data. Our results provide insights into how stylization improves model robustness, and provide evidence that artist-created paintings can be a valuable source of data for model robustness.
△ Less
Submitted 27 May, 2021; v1 submitted 29 November, 2020;
originally announced November 2020.
-
Insights From A Large-Scale Database of Material Depictions In Paintings
Authors:
Hubert Lin,
Mitchell Van Zuijlen,
Maarten W. A. Wijntjes,
Sylvia C. Pont,
Kavita Bala
Abstract:
Deep learning has paved the way for strong recognition systems which are often both trained on and applied to natural images. In this paper, we examine the give-and-take relationship between such visual recognition systems and the rich information available in the fine arts. First, we find that visual recognition systems designed for natural images can work surprisingly well on paintings. In parti…
▽ More
Deep learning has paved the way for strong recognition systems which are often both trained on and applied to natural images. In this paper, we examine the give-and-take relationship between such visual recognition systems and the rich information available in the fine arts. First, we find that visual recognition systems designed for natural images can work surprisingly well on paintings. In particular, we find that interactive segmentation tools can be used to cleanly annotate polygonal segments within paintings, a task which is time consuming to undertake by hand. We also find that FasterRCNN, a model which has been designed for object recognition in natural scenes, can be quickly repurposed for detection of materials in paintings. Second, we show that learning from paintings can be beneficial for neural networks that are intended to be used on natural images. We find that training on paintings instead of natural images can improve the quality of learned features and we further find that a large number of paintings can be a valuable source of test data for evaluating domain adaptation algorithms. Our experiments are based on a novel large-scale annotated database of material depictions in paintings which we detail in a separate manuscript.
△ Less
Submitted 24 November, 2020;
originally announced November 2020.
-
Sketch-and-test: picture-centered research with p5.js assisted crowdsourcing
Authors:
Maarten W. A. Wijntjes,
Mitchell van Zuijlen
Abstract:
Relating human judgements to pictures is central to a wide variety of scientific disciplines. Pictures are used to evoke and study faculties of the human mind, while human input is used to label, understand and model pictorial representations. Human input is often collected through online crowdsourcing experiments. This paper discusses the usage of crowdsourcing in two major branches of picture-ce…
▽ More
Relating human judgements to pictures is central to a wide variety of scientific disciplines. Pictures are used to evoke and study faculties of the human mind, while human input is used to label, understand and model pictorial representations. Human input is often collected through online crowdsourcing experiments. This paper discusses the usage of crowdsourcing in two major branches of picture-centered research, human and computer vision, and identifies novel directions such as art history and design. We demonstrate that a wide variety of experiments can be conducted by using p5.js, a library originally intended to facilitate visual creation. We report five complementary experimental paradigms to illustrated the accessibility and versatility of p5.js: Change blindness, BubbleView, 3D shape perception, Composition, and Perspective reconstruction. Results reveal that literature findings can be reproduced and novel insights can easily be achieved with the p5.js library. The creative freedom of p5.js combined with low threshold access to crowdsourcing seems like a powerful combination for all picture-centred research areas: perception, design, art history, communication, and beyond.
△ Less
Submitted 8 March, 2021; v1 submitted 17 April, 2020;
originally announced April 2020.
-
Annotating shadows, highlights and faces: the contribution of a 'human in the loop' for digital art history
Authors:
Maarten W. A. Wijntjes
Abstract:
While automatic computational techniques appear to reveal novel insights in digital art history, a complementary approach seems to get less attention: that of human annotation. We argue and exemplify that a 'human in the loop' can reveal insights that may be difficult to detect automatically. Specifically, we focussed on perceptual aspects within pictorial art. Using rather simple annotation tasks…
▽ More
While automatic computational techniques appear to reveal novel insights in digital art history, a complementary approach seems to get less attention: that of human annotation. We argue and exemplify that a 'human in the loop' can reveal insights that may be difficult to detect automatically. Specifically, we focussed on perceptual aspects within pictorial art. Using rather simple annotation tasks (e.g. delineate human lengths, indicate highlights and classify gaze direction) we could both replicate earlier findings and reveal novel insights into pictorial conventions. We found that Canaletto depicted human figures in rather accurate perspective, varied viewpoint elevation between approximately 3 and 9 meters and highly preferred light directions parallel to the projection plane. Furthermore, we found that taking the averaged images of leftward looking faces reveals a woman, and for rightward looking faces showed a male, confirming earlier accounts on lateral gender bias in pictorial art. Lastly, we confirmed and refined the well-known light-from-the-left bias. Together, the annotations, analyses and results exemplify how human annotation can contribute and complement to technical and digital art history.
△ Less
Submitted 10 September, 2018;
originally announced September 2018.
