Innocence Project is one of the 69 organizations that are part of the Innocence Network, that work to free innocent people from jail and to prevent wrongful convictions. This work can be with legal services or investigation of the case presented.  [12]
In the United States, the Innocence Project reported over 375 DNA exonerations [13]. The causes of wrongful conviction that the organization are aware of are: eyewitness identification, false confessions, forensic science errors, poor defense, among others. [12]
Innocence Project Argentina is part of the Innocence Network, and it works in Argentinian cases. This organization is the one that got in touch with Laboratorio de Sueño y Memoria from Instituto Tecnológico de Buenos Aires (ITBA).
The Lab does research in neuroscience. Some of the areas they are working on are: activation of memories while slee**, sleep paralysis, wave processing for the brain and false memories formation [14].

In 2019, Zlotnik and Vansintjan defined memory as "the capacity to store and retrieve information" [15]. The process of formation of memories have different phases.
The first of these phases is the acquisition. In this phase, the sensory stimuli are encoded into neurochemical representations. The second phase is the consolidation, that is a period in which the memory is stabilized in order to subsist through time. The retrieval phase is where the information in a memory can be recovered. A consolidated memory can go through a re-consolidation phase, in which its information can be modified and it is re-stabilized. [16]
This last phase is the one that has a key role for the experiments of the Lab. There are some theories about how false memories are created. Between them, there is the activation-monitoring theory, which explains the creation of false memories using two principles. The first came as a conclusion of an experiment that consists in giving the subject a list of words and later ask if a certain word was in the list or not. It was discovered that if the words in the list were related somehow, and the word asked later followed the same topic, the subject would think the word was in the list, even if it that was not the case. The second principle states that if the person cannot remember the source of the information of interest, that person may create a false memory. [17]

A Generative Adversarial Network (GAN) is a deep-learning-based model introduced by Goodfellow [18], that allows to create synthetic data that is indistinguishable from real data. The model has two neural networks (a generator and a discriminator), and a training data set.
The generator’s objective is to create data as similar as possible of the training data set. The discriminator’s objective is to say which data comes from the generator and which data comes from the training data set just having the data as input. This way, the generator and discriminator competes against each other to obtain better results.
If the discriminator fails to tell the source of the data, it means the generator was able to fool the discriminator. In this case, the discriminator will be the one learning from its mistake. If the discriminator is able to tell correctly the source of the data, the generator is the one that will be learning. When the training is complete, the generator network is the one used to keep generating data.

In our case, the data are images of faces. In the training, the generator will be learning the structure of these images, projecting them to a $\mathbb{R}^{n}$ space called latent space. This way, the generator takes as an input a $\mathbb{R}^{n}$ point, usually called latent code or $z$, and the output is the image in that point of the latent space.

In 2020, Erik Härkönen, Aaron Hertzmann, et al. released a paper titled GANSpace: Discovering Interpretable GAN Controls [28] showing how to find latent directions allowing to modify the properties of the objects in the generated image by the neural network. This was mainly based on Principal Component Analysis (PCA), which provides a much faster way to extract meaningful latent directions in comparison to manual supervision or expensive optimization. The results of this paper can be applied to StyleGAN’s 2 architecture, paving the way to modify facial attributes of a generated image within the neural network itself.

Another paper was released in 2022 focusing in face editing with GANs moving across a direction in the latent space. Face editing with GAN’s – A Review [29], written by Parthak Mehta, Sarthak Mishra, et al, shows how can other classification models can be used, such as logistic regression or SVM, to find the directions that represent a feature in the trained StyleGAN model. Not only that, but they also found a way to find a latent vector which generated image is similar to a target one. This way, a real portrait can be used as an input to find a similar face generated by the neural network, which attributes could then be modified using the directions that represent a feature.

This project aims to continue the work of Jimena Lozano and Maite Herrán Oyhanarte on the subject of GANs to create and manipulate facial images [1]. This tool was intended as a software to generate realistic facial images with the ability to manipulate their main characteristics, such as eye size and separation, nose size, etc. The resulting software is able to generate new random, artificial facial images from scratch and generate transitions between two generated pictures. StyleGAN 2 neural network was the core of this software to provide those features, paired with an API and a Web application built in Python and React respectively. All of these made possible an initial version of a software that enabled the Lab to improve their preparation workflow for their experiments.

There is no standardized way of defining a face-frame. We define it as the set of characteristics of a face that gives context for the internal characteristics of such face. A face-frame includes the contour of the face, hair, ears. A face-frame does not include eyes, mouth, nose or eyebrows. For example, Figure 3.1 shows two faces with the same frame. The hypothesis of the Lab is that this two faces may be mistaken with each other because they share the same frame.

Having this in mind, we measure the variation of the face-frame between two images based on the position of the face and hair in the image.

As mentioned in section 4.1, StyleGAN2 uses a metric [31] to estimate perceptual similarity to the target image. Albeit this method works pretty well, it does not take into account the face-frame variation. To try to obtain results with the least variation of face-frame as possible, a post-processing algorithm (henceforth referred to as face-frame correction) has been designed to pick the best image, which has the lower variation of them all, using the function defined in 3.1.2 to measure this metric.
The correction takes the target image and the latent code of the projection as an input. First, it calculates the standard deviation of 10000 random latent codes that create realistic images, which allows us to apply noise to the original latent code in a way that we can be certain it will yield a realistic image. The $f(targetimage,G(initiallatentcode))$ is also calculated and stored in this first step. The noise strength in each iteration is the result of applying the following formula:

Being $i$ the current iteration number, $l_{std}$ the standard deviation of the latent codes previously mentioned, $noise_{0}$ a constant representing the initial noise factor, and $nrl$ a constant representing the noise ramp length. In these experiments, the values are $noise_{0}=0.005$ and $nrl=0.75$, which follow what StyleGAN2 uses [32]
Throughout every of the $n$ iterations, the algorithm introduces a Gaussian noise multiplied by $strength(i)$ to the latent code. Depending on the constants $noise_{0}$ and $nrl$, it can be expected that the output image differs too little from the original one so that, if this new latent code generates an image with less face-frame variation than the previous latent code, then this new one is stored and used as the current latent code. If that is not the case, the original is preserved. In Figure 3.3, we can see an example of a target face being projected into the latent space as it outputs the neural network and another one after applying the correction algorithm.

Having set fixed values of $noise_{0}=0.005$ and $nrl=0.75$ taken from StyleGAN2 [32], the optimal value of $n$ is going to be studied having in mind the reduction of the face-frame variation of the target image and the projected one as much as possible.

The image to latent space projection operation StyleGAN2 provides, takes advantage of the fact that the latent space is semantically smooth. This means that small changes in the input vector, will result to small changes in the resulting image. A random input vector is used to start with. This vector is slightly modified and the resulting image is compared to the target image. It uses a standard LPIPS metric to estimate perceptual similarity between an image and the target image [31]. If the modification resulted in a better similarity to the target image, it is maintained. This process of slight modifications and measurements is done a thousand times.
The target image’s frame and the resulting image’s frame are compared.

We can take advantage of the fact that there are multiple latent directions we can move through, modifying certain attributes of an image. Some of this latent directions are of interest as they can be used by the Lab to edit an image right in the neural network itself. We define an operation done on an image which originates by a latent code as an addition or subtraction of a vector in such a way that a property of the face is modified.
Having this in mind, we will study the face-frame variation when applying some of these operations. Let $O$ be the operation to be studied. The face-frames to compare are the ones from $I_{1}$ and $O(I_{1})=I_{2}$. As the results may vary from image to image, the measurement is taken as the average measurement of a hundred different images. The same hundred images are used in each operation.

In order to be examine the effectiveness of the face-frame correction algorithm proposed in Section 4.1, we first need to take a look at how well StyleGAN2 does when projecting a target image to the latent space in terms of face-frame variation. To measure this, we project 10 images of different faces to the latent space without using the correction algorithm, and we measure the variation using the function defined in Section 3.1.2. These results are then processed into a table allowing us to have a baseline of how much the variation is using the default StyleGAN2’s function.

After examining the variation that StyleGAN2 introduces, we can then study how well our proposed algorithm corrects this variation. In order to test this, we set $n=2000$ and we run our algorithm for each of the 10 output images of the previous Section 5.1. We then compare the variation in each iteration, allowing us to study the optimal $n$ in terms of computational time and reduced variation.

We also want to take a look at the face-frame variation for the 10 images when transforming those moving across the aforementioned latent directions. This allows us to rank each operation in respect of the variation they introduce on the generated image, making them more or less elegible for image modification if face-frame stabilization is of the essence.

Table 6.1 shows the face-frame variation that the image to latent space projection achieved with each image. The maximum variation is at 3.192% while the minimum at 0.747%. The mean variation between these images is of 1.812%. This results suggest that images are a decent starting point when trying to generate similar faces with minimum face-frame variation.

Figures 6.2 and 6.3 illustrate the improvement in the face-frame variation through the iterations of the correction. The curves of variation significantly decrease in the first 750 iterations. The last 1250 iterations takes account for the last 10% of the improvement. It is notable that the improvement in the variation is always superior to the 20% of the initial value, and in some of the cases, it goes over the 50%.

We also check the face-frame variation for the 10 images when transforming those moving across the aforementioned latent directions. In Figure  6.5, we can see the chosen images. They are labeled as 1 to 10 from left to right.

When an image moves a great magnitude through a direction, they tend to give unexpected results, deforming the base structure of the face and giving unrealistic images. This is the reason the range measured in each direction changes in each case. For the set of chosen images, we decided the following ranges:

Figures 6.8 and 6.9 both show that modifying the horizontal and vertical face perspective in the image greatly worsens the face-frame variation. Figure 6.6 shows that gender modification is another operation that significantly increases the face-frame variation. These operations are some of the worst ones when considering this variation as can be seen in 6.13, together with smile modification (6.12) and age modification (6.6).

However, figures 6.11 and 6.10 both show that modifying the openness of the face’s mouth or the eyes are the two operations that least increase the face-frame variation.

Another interesting result illustrated in 6.13 is that all the tested operations have a linear impact on the face-frame variation.

Latent directions are the key to modify faces inside the latent space. However, only some of the known latent directions have been studied. New directions can be studied to understand how the latent space is arranged. To find new directions, an unsupervised discovery method can be used, for example the one Voynov and Babenko propose [33].
If one of the found directions is interpretable in a similar way than other known, it is possible to compare them by their face-frame variation to choose between them for a better option.
This will allow further face’s characteristics modification within the net itself in such a way that it could be possible to help the Lab disprove the belief that police lineups are helpful when searching for a crime perpetrator.

The face-frame variation function used for this project described in section 3.1.2, is a first version that was created with this project in mind and isn’t flawless: sometimes it confuses facial hair, such as beard, as being part of the face and sometimes it labels it as part of the background. The neck is also sometimes considered part of the face, which in reality isn’t. There may be a more precise way to measure the face-frame variation between two images that we are not aware of and which can solve this issues.
One approach could be taking advantage of facial landmark detection algorithms [34] to also avoid losing facial characteristics, as our algorithm only considers the face-frame, but it doesn’t take into consideration all of the facial characteristics.

The face-frame correction algorithm used is only applied and tested when projecting images to the latent space. However, it may also be used to reduce face-frame variation when performing some operations to a facial image, such as modifying its eye-opening.

This project was done using StyleGAN2 [9] as it is the state of art of face images generation at the moment. However, the same experiment can be done using different networks. Text-to-image networks are starting to be popular, it may exist a way that tools such us DALL-E [35] or Stable diffusion [36], can be used to modify faces with similar results as the ones obtained with StyleGAN2. In fact, DALL-E 2[37] allows for replacing parts of an image using a text description entered in a prompt, in a process called inpainting. This allows for photo realistic modification of images, which yields a credible image in a fast and accurate way.