AI-Generated Faces in the Real World:
A Large-Scale Case Study of Twitter Profile Images

To estimate the prevalence of generated images on a social network, it is important to obtain a mostly unconditional sample. In the case of Twitter, this can be achieved by using the API endpoint that provides real-time access to a random $1\text{\,}\mathrm{\char 37\relax}$ subset of all publicly posted tweets. We download each author’s profile image together with their profile metadata (cf. Section A.1 for an overview). Note that this approach only enables us to obtain profile images from users who write posts during the data collection period. Additionally, we omit users who have not set a profile image, that is, who are using Twitter’s default profile image. From March 7 to March 15 2023, we collected $14\,989\,385$ profile images.

We continue with labeled datasets of fake and real images which can be used to train a detector. As discussed in Section 1, existing observations suggest that the vast majority of generated profile images on Twitter are taken from TPDNE, which generates images with StyleGAN2 \parencitekarrasAnalyzingImprovingImage2020 trained on the FFHQ \parencitekarrasStylebasedGeneratorArchitecture2019 dataset¹¹1When published in 2019, TPDNE used the original StyleGAN \parencitekarrasStylebasedGeneratorArchitecture2019, but switched to StyleGAN2 \parencitekarrasAnalyzingImprovingImage2020 shortly after its release.. We therefore decide to focus on this specific kind of fake faces and use $10\,000$ images from TPDNE as our fake-labeled dataset (denoted by $\mathcal{D}_{F}$) and correspondingly $10\,000$ images from FFHQ as our real-labeled dataset (denoted by $\mathcal{D}_{R}$). We discuss this limitation of focusing on TPDNE in Section 8. As prior work shows that processing operations like resizing and compression can affect the detection \parenciteparmarOnAliasedResizing2022, mandelliTrainingCNNsPresence2020, we consider two dataset variations:

• •

  $\mathcal{D}_{R}^{\mathbb{X}}$ and $\mathcal{D}_{F}^{\mathbb{X}}$. To obtain profile images with the social network’s processing steps, we adapt the approach from \textciteboatoTrueFaceDatasetDetection2022. We upload both $\mathcal{D}_{R}$ and $\mathcal{D}_{F}$ to Twitter, set each image as profile image, and then download all images again. We denote these processed images by $\mathcal{D}_{R}^{\mathbb{X}}$ and $\mathcal{D}_{F}^{\mathbb{X}}$.

• •

  $\mathcal{D}_{R}^{\mathbb{X}^{\prime}}$ and $\mathcal{D}_{F}^{\mathbb{X}^{\prime}}$. We additionally simulate a user which zooms into the profile image during the upload, as it is common for social media platforms. We denote these images by $\mathcal{D}_{R}^{\mathbb{X}^{\prime}}$ and $\mathcal{D}_{F}^{\mathbb{X}^{\prime}}$, respectively.

We confirm in Section 7.2 that considering the preprocessing indeed improves the detection performance under realistic conditions.

Social media platforms often have very popular users with a lot of followers. These popular users are rather unlikely to use deceptive fake images. Hence, we can build a proxy-labeled dataset with presumably real images. In particular, we select $10\,000$ profile images from the accounts in $\mathcal{D}_{W}^{\mathbb{X}}$ with the highest numbers of followers that also pass our pre-filter (which is presented in the next section). We denote the so-created proxy-labeled dataset of real profile images by $\mathcal{D}_{P}^{\mathbb{X}}$.

Finally, there are documented cases of generated profile images that were discovered manually. For example, blog posts regularly report such images when analyzing inauthentic Twitter accounts \parencitenortenoConspiradorNortenoSubstack2024. These cases can be used to build a labeled dataset of fake images in the wild, which we denote by $\mathcal{D}_{D}^{\mathbb{X}}$. Such a dataset is not free of bias, but provides a good means to finally check the performance of our classifier on an independent source. For our study, we use a dataset of $1\,353$ generated Twitter profile images that were manually collected between November 2022 and May 2023 \parenciteyangCharacteristicsPrevalenceFake2024.

We start with a pre-filter $\phi$ to discard irrelevant samples. In our case, we can discard images without any face or where the face is too small. We use the efficient BlazeFace \parencitebazarevskyBlazeFace2019 face detector to detect faces and locate facial landmarks. An image passes $\phi$ if at least one face is detected and the Euclidean distance between the coordinates of both eyes is greater or equal to $0.1$. The pre-filter serves two purposes: First, the overall computational complexity decreases by reducing the number of analyzed candidates in the subsequent, more demanding detection stage. Second, the detection stage is trained on facial images, so that other types of profile images, such as logos or monochrome images, could be wrongly classified as fake. Filtering irrelevant images can therefore decrease the false positive rate (FPR).

To automatically label a profile image as real or fake, we use a state-of-the-art CNN detector based on ResNet-50 \parenciteheDeepResidualLearning2016. Previous work \parencitewangCNNgeneratedImagesAre2020, mandelliTrainingCNNsPresence2020, cozzolinoSpoCSpoofingCamera2021, cozzolinoUniversalGANImage2021, gragnanielloAreGANGenerated2021 has demonstrated that this model is able to effectively distinguish real from generated images and that it provides good generalization capabilities. We initially attempted to use pre-trained fake image detectors, however, we found that the heavy pre-processing performed by Twitter makes it necessary to train our own detector (cf. Section 7.1). In particular, we train on the combination of $\mathcal{D}_{R}^{\mathbb{X}}$ and $\mathcal{D}_{P}^{\mathbb{X}}$ for real images, and $\mathcal{D}_{F}^{\mathbb{X}}$ for fake images. The resulting final classifier is denoted by $\mathcal{C}_{R^{\mathbb{X}},P^{\mathbb{X}}/F^{\mathbb{X}}}$. Note that we experiment with using other dataset variations to train a classifier in our ablation study in Section 7.2. Yet, using processed real, fake, and proxy-labeled real images provides the highest performance for processed and zoomed inputs.

Faces generated by StyleGAN2 \parencitekarrasAnalyzingImprovingImage2020 are characterized by being almost perfectly aligned with respect to their facial landmarks, caused by the alignment of the training dataset FFHQ. By superimposing multiple images, this characteristic has been leveraged to visually identify clusters of fake accounts in social networks \parencitenimmoOperationNavalGazing2020, nimmoIRAAgainUnlucky2020, graphikateamStepMyParler2020, graphikateamFakeClusterBoosts2021, stanfordinternetobservatoryReplyguysGoHunting2020, strickAnalysisProchinaPropaganda2021, goldsteinResearchNoteThis2022. We automate this manual process by extracting facial landmarks with BlazeFace \parencitebazarevskyBlazeFace2019 and computing the deviation from a reference. For each landmark $L_{1},\dotsc,L_{12}$ (x- and y-coordinates of eyes, ears, mouth, and nose), we compute its mean $\mu_{i}$ and standard deviation $\sigma_{i}$ over a reference dataset. In our study, we use the training subset of $\mathcal{D}_{F}^{\mathbb{X}}$ as reference. We define an image $x$ as being aligned, if the condition

holds, where $L_{i}(x)$ are the landmarks extracted from the image $x$ and $k\in\mathbb{Z}$ controls the maximum deviation from the reference. We set $k=7$. During our evaluation in Section 4, we find that this is the lowest value at which all generated images in the validation set of $\mathcal{D}_{F}$ are aligned. While a close alignment hints towards a face generated by StyleGAN2, it is ineffective if the image has been cropped or geometrically transformed.

Additionally, we leverage GAN inversion \parencitexiaGANInversionSurvey2023 as an assistance tool. For a given input image, this method finds the latent code which reconstructs the original input when passed through the generator. We use the provided implementation by \textcitekarrasAnalyzingImprovingImage2020 to invert images using StyleGAN2. Previous work has shown that generated images can be reconstructed more successfully than real images \parencitealbrightSourceGeneratorAttribution2019,karrasAnalyzingImprovingImage2020,pasquiniIdentifyingSyntheticFaces2023 (we provide a visual example in Appendix B). Note that inversion also relies on facial alignment. If an adversary uses a cropped version of a fake face, the inversion result will be distorted. We therefore only use inversion as labeling assistance if the image is aligned.

On Twitter, social interactions are primarily measured in the number of followers an account has and the number of other accounts it follows. Figures 5(a) and 5(b) visualize the distribution of these metrics for real- and fake-image accounts at the time of data collection. We find that fake-image accounts have fewer followers (mean: 393.35, median: 60) compared to real-image accounts (mean: $5\,086.38$, median: 165) in our dataset. $1\,997$ ($25.86\text{\,}\mathrm{\char 37\relax}$) of all fake-image accounts have $9$ or fewer followers and $1\,063$ ($13.76\text{\,}\mathrm{\char 37\relax}$) have exactly zero followers. We notice that $1\,996$ fake-image accounts ($25.84\text{\,}\mathrm{\char 37\relax}$) have exactly 106 followers. Our content analysis in Section 6.2 reveals that these accounts belong to a large cluster of fake accounts involved in coordinated inauthentic behavior.

We find that fake-image accounts also follow fewer other accounts (mean: 283.18, median: 21) compared to real-image accounts (mean: 759.83, median: 262). Interestingly, $2\,175$ fake-image accounts ($28.16\text{\,}\mathrm{\char 37\relax}$) follow exactly two other accounts. In contrast to the number of followers, a relatively small number of fake-image accounts ($163$, $2.11\text{\,}\mathrm{\char 37\relax}$) follows exactly zero other accounts.

Figure 5(c) shows that fake-image accounts do participate in Twitter based on the number of tweets. Yet, they are overall less active than real-image accounts. On average, fake-image accounts posted $3\,158.9$ (median: 112) tweets, as opposed to $17\,096.39$ (median: 3450) tweets from real-image accounts. $1\,948$ ($25.22\text{\,}\mathrm{\char 37\relax}$) of all fake-image accounts have 10 or fewer tweets. In addition, Figure 5(d) shows the average number of tweets per day, calculated by dividing the total number of tweets by the number of days the account exists. Based on the median, fake-image accounts are still less active than real-image accounts (0.95 vs. 3.7 tweets per day). However, a large fraction of fake-image accounts posts exceptionally many tweets per day, causing a higher mean (19.96 vs. 13.56 tweets per day). In particular, there are 266 fake-image accounts ($3.44\text{\,}\mathrm{\char 37\relax}$) that submitted more than 100 tweets per day.

Figure 6(a) compares the times of account creation. Fake-image accounts are considerably “younger”, with more than half of them ($52.38\text{\,}\mathrm{\char 37\relax}$) being created in 2023 (note that our data collection happened in March 2023). In contrast, only $6.22\text{\,}\mathrm{\char 37\relax}$ of real-image accounts have been created in this period.

In addition to the creation date, we also examine the account status after a certain period of time. We checked the status of all $7\,723$ fake-image accounts nine months after data collection by querying the respective profile page. As a reference, we did the same for an equal number of randomly sampled real-image accounts. Accounts can be either alive, deactivated (by the user), or suspended (by Twitter). Figure 7 illustrates that more than half of the fake-image accounts ($52.07\text{\,}\mathrm{\char 37\relax}$) have been suspended. In contrast, only $5.01\text{\,}\mathrm{\char 37\relax}$ of real-image accounts in the reference set have been suspended. The high number of suspended fake-image accounts suggests that they were violating Twitter’s rules.

In Figures 6(b) and 6(c), we analyze the account creation of fake-image accounts given their status. We observe various suspended accounts that were created in bulk just shortly before our data collection, especially in the middle of February. Note that we do not know when these accounts were suspended, so that we cannot determine the effective lifetime of these accounts.

Our analysis shows that real and fake-image accounts notably differ. Fake-image accounts have fewer social interactions, both regarding the number of followers and the number of accounts they follow. While these metrics are distributed evenly for real-image accounts, we observe patterns with fake-image accounts. There are large groups with identical values, indicating an orchestrated network of inauthentic users. Moreover, fake-image accounts are not passive, they considerably participate in Twitter based on the number of tweets. Although they are in general less active than real-image accounts, there are several fake-image accounts that post very frequently, hinting towards spamming attacks. Finally, fake-image accounts have a more limited lifetime. They are usually created more recently than real-image accounts, and they are also disproportionately often suspended by Twitter. This suggests inauthentic behavior. Moreover, a substantial number was created in bulk just before our data collection period started. This bulk creation (or batch creation) is a common pattern for inauthentic behavior, used, for example, to amplify messages or to participate in spamming or trolling activities \parencitegurajalaFakeTwitterAccounts2015,ferraraTwitterSpamFalse2022.

The clustering for English content posted by users that are not on the platform anymore reveals a notable pattern: we observe a single, extremely large cluster that encompasses $49.67\text{\,}\mathrm{\char 37\relax}$ of all unavailable English tweets. Despite the variability of the actual content, these tweets all share a common structure. Each tweet begins by mentioning a specific Twitter user, followed by a short sequence of English terms. Interestingly, these sequences do not form logical sentences, so they are neither semantically nor syntactically correct. Each of these sentences is then followed by a specific Chinese hashtag that can be translated to: “This is really useful”. Unfortunately, we can only speculate about their purpose. Our hypothesis is that the embedded hyperlinks within the tweets may have directed users to malicious external websites. As the links are no longer functional, we cannot verify this hypothesis.

The accounts’ metadata corroborate the assumption that the $1\,579$ accounts within this cluster were part of an organized network. All but three were created between February 16 and February 20, which is consistent with our observations in Figure 6(c). Up to 754 accounts were created on a single day. We also find that this cluster contributes to the large number of accounts with identical social connections (cf. Figure 5(a) and Figure 5(b)). $94.93\text{\,}\mathrm{\char 37\relax}$ have exactly 106 followers and $95.31\text{\,}\mathrm{\char 37\relax}$ follow exactly two other accounts. The usernames (Twitter handles) appear to be constructed from a list of German-sounding first names and last names (or initials), and optionally one or multiple digits (e.g., @GuntherForstner86). $67.44\text{\,}\mathrm{\char 37\relax}$ of all accounts have the same display name that can be translated to “Noon Namshi Sivvi discount code is strong and effective” (Noon and Namshi are e-commerce platforms operating in the Arabic region). These accounts also have their location set to “KSA” (Kingdom of Saudi Arabia). Moreover, the accounts contain nonsense descriptions like “Personal west service street laugh small.”. We hypothesize that these were automatically generated or translated. Again, we can only speculate about the reason, especially about the mixed use of English, German, Arabic, and Chinese language. We also notice that $99.18\text{\,}\mathrm{\char 37\relax}$ of all accounts in this cluster use profile images that are duplicates within our dataset of fake-image accounts. Appendix D elaborates our method for identifying duplicate images.

The remaining clusters mainly focus on giveaways, often related to cryptocurrencies, with tweets like

$50 (2 winners x $25) 24 hours - like, follow)
i will #giveaway 100 usdt worth of $loop as we cele-00brate our 10k milestone

or the promotion of illegal content such as links to broadcasting streams of soccer matches, e.g.,

live stream arouca vs benfica live [link]

Another trend is the distribution of links to websites and Telegram groups containing explicit content, e.g.,

follow for more [link]

The clustering of tweets from users who were still active after nine months reveals similarities and differences. Figure 9 depicts a visual representation of the top clusters with their representative text tokens. A significant portion of all clusters is again related to various forms of cryptocurrency, stocks, and giveaways, e.g.,

drop your #tezos #nft if you need it sold!
15000$ in $eth — 5 lucky winners!

Additionally, we find a significant share of adult content/porn related clusters, actively advertising explicit content, also through dedicated patterns like

beautiful/charming/etc. [profile of porn actress] [link]

Compared to inactive users, we observe that available accounts also engage in discussions on contentious or political issues. These include, for example, the war in Ukraine, election-related discourse, and debates on COVID and vaccinations:

welcome to nazi ukraine #russia
desantis racks up wins while trump, potential 2024 00opponents take swipes at florida governor
albos crocodile tears: watch this video, that the main-00stream media refuses to show.
someone needs to find an antidote for the vaxxx

For the Turkish accounts, we restrict our analysis to content posted by users who have been removed, since this is the majority of the dataset. Our findings indicate that all of this content is related to pornography or escort services. The primary distinction among the clusters are the cities mentioned within the posts. Most tweets also contain links to other websites, which are no longer functional. Upon examining the metadata of all 932 accounts, we again identify the systematic pattern for usernames that we already observed in the large cluster of English tweets. However, first and last names appear to be of Turkish descent. Moreover, almost all accounts have their location set to a real Turkish city. $46.78\text{\,}\mathrm{\char 37\relax}$ of all accounts again use duplicate profile images and $85.52\text{\,}\mathrm{\char 37\relax}$ were created within one month. These findings again indicate that at least some systematic approach (automatic or semi-automatic) is used to generate the accounts.

For Arabic tweets, we again only consider accounts that have been suspended. All clustered tweets appear to be related to literature, with individual clusters being characterized by mentions of certain authors, countries, or topics—all related to the Arabic region. These tweets make up $72.34\text{\,}\mathrm{\char 37\relax}$ of all unavailable Arabic tweets. Surprisingly, the tweets share a common structure with those from the large cluster of English tweets: they contain the specific Chinese hashtag, an external link, and an incoherent sentence. Our metadata analysis suggests that the $1\,806$ accounts indeed belong to the same cluster, despite the different language. Almost all accounts were created between February 16 and February 20, with 892 being created on a single day. We observe the same anomalies regarding the (German) usernames, locations, descriptions, and social connections. Given the book-related content and the frequently occurring username that promotes a discount code, we hypothesize that the external links might have referred to the respective shop** platforms.

Our content analysis reveals that English, Turkish, and Arabic are the dominant languages used by the fake-image accounts in our collection. We identify large networks of fake-image accounts that were probably automatically created and that participated in large-scale spamming attacks. We observe recurring patterns as part of the automation. Accounts are created in bulk. Tweets, usernames, locations, descriptions, and social connections follow a systematic pattern. Multiple accounts within a network share the same profile image. Furthermore, our analysis shows that frequently occurring topics are cryptocurrencies, giveaways, and content related to pornography and escort services. Fake-image accounts also participate in controversial political discussions. These findings align with prior analyses of inauthentic content on Twitter \parenciteratkiewiczDetectingTrackingPolitical2011,cresciDecadeSocialBot2020,nizzoliChartingLandscapeOnline2020,pfefferJustAnotherDay2023.

We test three existing pre-trained classifiers: $\mathcal{C}_{\text{Wang}}$ \parencitewangCNNgeneratedImagesAre2020 (which is the basis of our classifier), $\mathcal{C}_{\text{Grag}}$ \parencitegragnanielloAreGANGenerated2021, and $\mathcal{C}_{\text{Ojha}}$ \parenciteojhaUniversalFakeImage2023. Appendix C provides more details on these three classifiers. We evaluate four conditions (see Table 3). The conditions (a)-(c) correspond to those in Figure 1 and all use images processed by Twitter. We additionally test the pre-trained detectors on unprocessed images in condition (d).

Table 3 shows the AUCs of the three classifiers compared to $\mathcal{C}_{R^{\mathbb{X}},P^{\mathbb{X}}/F^{\mathbb{X}}}$. Our trained detector significantly outperforms the pre-trained classifiers under the Twitter conditions (a)-(c). The fact that the latter perform better under the clean condition (d) demonstrates the strong effect of Twitter’s processing. It is therefore not possible to use a pre-trained detector for our study of in-the-wild profile images.

We consider three classifier variants and analyze their performance under the three conditions from Figure 1, respectively. The classifier $\mathcal{C}_{R/F}$ is trained on $\mathcal{D}_{R}$ and $\mathcal{D}_{F}$ and represents the most straightforward option. The images are not processed by Twitter, but we resize them to $400\times 400$ pixels to match the resolution of actual profile images. The second classifier, $\mathcal{C}_{R^{\mathbb{X}}/F^{\mathbb{X}}}$, is trained on the same images but with Twitter’s processing. Finally, $\mathcal{C}_{R^{\mathbb{X}},P^{\mathbb{X}}/F^{\mathbb{X}}}$ is additionally trained on $\mathcal{D}_{P}^{\mathbb{X}}$ as real images.

Table 4 shows the AUCs of the three detector variants under the different conditions. Our finally chosen classifier, $\mathcal{C}_{R^{\mathbb{X}},P^{\mathbb{X}}/F^{\mathbb{X}}}$, has the highest performance in all conditions. The classifier $\mathcal{C}_{R/F}$ trained on unprocessed images performs worse than both variants trained on processed images, confirming our findings from Section 7.1. Note that, while an AUC $>0.99$ is still very high, the small difference can cause a significant increase in false positives, given the size of $\mathcal{D}_{W}^{\mathbb{X}}$. Overall, our results provide two insights. First, the classifier $\mathcal{C}$ should be trained on images that are processed similarly to the target images. Second, including proxy-labeled real images from the target distribution ($\mathcal{D}_{P}^{\mathbb{X}}$) improves the detection performance. A closer look shows that this causes a better separation of the classifier scores, shifting scores towards either end of the output range. This motivates our choice of $\mathcal{C}_{R^{\mathbb{X}},P^{\mathbb{X}}/F^{\mathbb{X}}}$ for our study.