-
ProtoExplorer: Interpretable Forensic Analysis of Deepfake Videos using Prototype Exploration and Refinement
Authors:
Merel de Leeuw den Bouter,
Javier Lloret Pardo,
Zeno Geradts,
Marcel Worring
Abstract:
In high-stakes settings, Machine Learning models that can provide predictions that are interpretable for humans are crucial. This is even more true with the advent of complex deep learning based models with a huge number of tunable parameters. Recently, prototype-based methods have emerged as a promising approach to make deep learning interpretable. We particularly focus on the analysis of deepfak…
▽ More
In high-stakes settings, Machine Learning models that can provide predictions that are interpretable for humans are crucial. This is even more true with the advent of complex deep learning based models with a huge number of tunable parameters. Recently, prototype-based methods have emerged as a promising approach to make deep learning interpretable. We particularly focus on the analysis of deepfake videos in a forensics context. Although prototype-based methods have been introduced for the detection of deepfake videos, their use in real-world scenarios still presents major challenges, in that prototypes tend to be overly similar and interpretability varies between prototypes. This paper proposes a Visual Analytics process model for prototype learning, and, based on this, presents ProtoExplorer, a Visual Analytics system for the exploration and refinement of prototype-based deepfake detection models. ProtoExplorer offers tools for visualizing and temporally filtering prototype-based predictions when working with video data. It disentangles the complexity of working with spatio-temporal prototypes, facilitating their visualization. It further enables the refinement of models by interactively deleting and replacing prototypes with the aim to achieve more interpretable and less biased predictions while preserving detection accuracy. The system was designed with forensic experts and evaluated in a number of rounds based on both open-ended think aloud evaluation and interviews. These sessions have confirmed the strength of our prototype based exploration of deepfake videos while they provided the feedback needed to continuously improve the system.
△ Less
Submitted 20 September, 2023;
originally announced September 2023.
-
Forensic Video Steganalysis in Spatial Domain by Noise Residual Convolutional Neural Network
Authors:
Mart Keizer,
Zeno Geradts,
Meike Kombrink
Abstract:
This research evaluates a convolutional neural network (CNN) based approach to forensic video steganalysis. A video steganography dataset is created to train a CNN to conduct forensic steganalysis in the spatial domain. We use a noise residual convolutional neural network to detect embedded secrets since a steganographic embedding process will always result in the modification of pixel values in v…
▽ More
This research evaluates a convolutional neural network (CNN) based approach to forensic video steganalysis. A video steganography dataset is created to train a CNN to conduct forensic steganalysis in the spatial domain. We use a noise residual convolutional neural network to detect embedded secrets since a steganographic embedding process will always result in the modification of pixel values in video frames. Experimental results show that the CNN-based approach can be an effective method for forensic video steganalysis and can reach a detection rate of 99.96%. Keywords: Forensic, Steganalysis, Deep Steganography, MSU StegoVideo, Convolutional Neural Networks
△ Less
Submitted 29 May, 2023;
originally announced May 2023.
-
Verification system based on long-range iris and Graph Siamese Neural Networks
Authors:
Francesco Zola,
Jose Alvaro Fernandez-Carrasco,
Jan Lukas Bruse,
Mikel Galar,
Zeno Geradts
Abstract:
Biometric systems represent valid solutions in tasks like user authentication and verification, since they are able to analyze physical and behavioural features with high precision. However, especially when physical biometrics are used, as is the case of iris recognition, they require specific hardware such as retina scanners, sensors, or HD cameras to achieve relevant results. At the same time, t…
▽ More
Biometric systems represent valid solutions in tasks like user authentication and verification, since they are able to analyze physical and behavioural features with high precision. However, especially when physical biometrics are used, as is the case of iris recognition, they require specific hardware such as retina scanners, sensors, or HD cameras to achieve relevant results. At the same time, they require the users to be very close to the camera to extract high-resolution information. For this reason, in this work, we propose a novel approach that uses long-range (LR) distance images for implementing an iris verification system. More specifically, we present a novel methodology for converting LR iris images into graphs and then use Graph Siamese Neural Networks (GSNN) to predict whether two graphs belong to the same person. In this study, we not only describe this methodology but also evaluate how the spectral components of these images can be used for improving the graph extraction and the final classification task. Results demonstrate the suitability of this approach, encouraging the community to explore graph application in biometric systems.
△ Less
Submitted 28 July, 2022;
originally announced August 2022.
-
Camera identification by grou** images from database, based on shared noise patterns
Authors:
Teun Baar,
Wiger van Houten,
Zeno Geradts
Abstract:
Previous research showed that camera specific noise patterns, so-called PRNU-patterns, are extracted from images and related images could be found. In this particular research the focus is on grou** images from a database, based on a shared noise pattern as an identification method for cameras. Using the method as described in this article, groups of images, created using the same camera, could…
▽ More
Previous research showed that camera specific noise patterns, so-called PRNU-patterns, are extracted from images and related images could be found. In this particular research the focus is on grou** images from a database, based on a shared noise pattern as an identification method for cameras. Using the method as described in this article, groups of images, created using the same camera, could be linked from a large database of images. Using MATLAB programming, relevant image noise patterns are extracted from images much quicker than common methods by the use of faster noise extraction filters and improvements to reduce the calculation costs. Relating noise patterns, with a correlation above a certain threshold value, can quickly be matched. Hereby, from a database of images, groups of relating images could be linked and the method could be used to scan a large number of images for suspect noise patterns.
△ Less
Submitted 12 July, 2012; v1 submitted 11 July, 2012;
originally announced July 2012.
