Fingerprinting web servers through Transformer-encoded HTTP response headers

The reason why web server fingerprinting is possible in the first place, Lee et al. (2002) argue, is because of the nature of the documents that describe the protocol. They have some ambiguity in how they should be interpreted, and leave flexibility in how they should be implemented.

Lee et al. describe some of the characteristics that distinguish implementations from each other:

• •

  Lexical: e.g., the response code message and capitalization, header wording and capitalization, and line terminators;

• •

  Syntactic: e.g., the header ordering, ordering of lists in header values, and the formatting of values;

• •

  Semantic: e.g., the presence or absence of specific headers, and the response given to malformed or undefined requests.

All that is required to craft a fingerprint for a web server is provoking it with a multitude of different requests, and recording its response to each. It should be noted that Lee et al. argued that some of these characteristics could not be uniquely assigned to one of the three linguistic domains. In this work, we share their stance that, for the purpose of identifying characteristics for web server fingerprinting, we do not need to spend too much time debating which test case belongs to which domain.

The topic of web server fingerprinting was revisited by Yang et al. (2010), who found that masking software⁴⁴4Such as ModSecurity and ServerMask. was increasingly being used to protect against fingerprinting. As those tools are unable to completely mask all responses, the web server still leaks a certain set of characteristics, which become an indicator that masking software is used. They argue that the only comprehensive solution against fingerprinting is by making sure that there is no difference in how the HTTP specification is implemented by software. Huang et al. (2015) described some additional fingerprinting characteristics

and concluded that detecting individual software versions was challenging from HTTP status codes alone. Book et al. (2013) provide the inspiration for this work. They crafted 10 different HTTP requests, and sent them to 120.000 domains in total. These were the highest-ranking 110.000 and the lowest ranking 10.000 sites from the Alexa Top 1 Million Sites dataset - the most popular websites on the Internet by descending order.

As features, they used the returned HTTP status code for each test. The target label required for supervised machine learning was extracted from the Server: header, which was reported by the majority of servers. They found that server type prediction was possible, although their methodology was not powerful enough to distinguish individual versions. Their dataset was also highly unbalanced, as a single type constituted almost half of the data. For future work, they suggested that a more comprehensive suite of test cases should be designed, as they relied only on their informed judgement. They also recommended the use of more data as well as fingerprinting server software other than web servers.

When the experiment by Book et al. (2013) was performed, the use of transport-layer encryption was uncommon. It was only in 2016, after the introduction of Let’s Encrypt⁵⁵5https://letsencrypt.org/2016/04/12/leaving-beta-new-sponsors.html and the Automatic Certificate Management Environment (ACME) protocol by Barnes et al. (2019), that TLS certificates became free of charge. In fact, there is no mention of HTTPS at all in the paper by Book et al. (2013). Concretely, this means that not only do we have to make requests to the default, unencrypted HTTP port 80, but also to HTTPS port 443 to get a complete picture of a typical, general-purpose web server.

Additionally, a new version of the HTTP protocol was officially introduced by Belshe et al. (2015). HTTP/2 is limited to technical, low-level improvements, and does not introduce any changes in semantics. These low-level changes, however, may result in more diverse protocol noncompliance by server software, as the implementation and interpretation of the protocol is up to the developers. HTTP/1.1 is still typically used for unencrypted connections, and HTTP/2 is typically only used for encrypted connections.

Modern web browsers generally negotiate HTTP/2 support in a backwards-compatible manner (i.e., servers are always given the opportunity to downgrade to a lower version of the protocol), and only attempt it over an encrypted connection. Web servers may thus vary in how well they support HTTP/2 under atypical conditions. These include not negotiating in a backwards-compatible way, or attempting to use HTTP/2 over an unencrypted connection.

In the reverse context, which is to say detecting clients from their requests as opposed to servers from their responses, much more research has been done. This is an active and popular research topic, as spam and malware continues to evolve. Whereas server fingerprinting is mostly rule-based, machine learning models trained on HTTP data from client requests have been extremely successful in detecting malicious or anomalous clients, generally achieving between 92% and 99% accuracy (Yu et al., 2018; Yuan et al., 2018a; Niu et al., 2019; Laughter et al., 2021). It should be noted that almost all of these experiments constitute a form of binary classification, which is generally easier than a multiclass problem such as detecting specific types or versions.

A notable innovation was developed by Mizuno et al. (2018). They built a system called BotDetector, to detect malware-infected clients from their HTTP traffic. Their central idea was to use a template generation algorithm based on DBSCAN clustering. Their algorithm clusters the keys and values of HTTP headers, resulting in fewer unique values, and thus fewer feature dimensions. The precision and recall of their models were high, indicating that this is a promising approach to reduce feature dimensionality. Niu et al. (2019) used a similar technique, but on a different dataset, and attempted to optimize XGBoost to perform classification. They found that XGBoost slightly outperformed the models tested by Mizuno et al. (2018), and agreed that template generation is effective. Template generation is thus a potential method to reduce the large amount of target classes in our experiment.

A number of experiments concentrated on creating (neural) embeddings and representations of HTTP headers instead of performing manual feature engineering (Li et al., 2020; Yuan et al., 2018b; Yu et al., 2018). For example, by using tf-idf, word2vec, GLoVe, or combinations of those. The most recent of these experiments was performed by Gniewkowski et al. (2021). They used a RoBERTa model and BPE tokenizer to generate a representation model for HTTP headers, for the purpose of detecting anomalous requests.

They trained their model only on unlabeled normal traffic, and found that simple classification algorithms could distinguish normal and anomalous traffic from the embeddings alone.

Given the performance of embeddings, neural architectures, and machine learning in general in many of the studies in this field, the main takeaway for this work is that it seems to be feasible to treat HTTP header data as a natural language, with its own grammar and vocabulary⁶⁶6One might conclude that humanity’s greatest technical infrastructure is fundamentally backed by the same thing as our social infrastructure: language.. As almost all papers in the field highlight that ordering information, in particular when it concerns the representation of URLs and headers, is an important predictor, any strategy for HTTP representation would do well to model sequences as wel as occurrences.

Niakanlahiji et al. (2018) created a classical machine learning framework for detecting phishing web pages. They performed extensive feature engineering, and found that the presence or absence of certain HTTP header fields, the number of header fields, and the number of non-standard header fields were important indicators of whether a website was malicious or not.

Al-Hakimi and Bax (2021) developed two feature extraction methods to detect malicious web servers from HTTP response headers. They concluded that no good feature extraction methods existed, and that these are prerequisites for doing effective machine learning experiments. They noted that their research was closely related to web server fingerprinting. They sought to perform binary classification (malicious/non-malicious), and only relied on passive data (i.e., a single HTTP request)⁷⁷7As mentioned in the first section, software type and version fingerprinting typically requires actively provoking HTTP responses.. They proposed a uniqueness value for each encountered header, and recording the indexed order in which headers occur for each sample.

In their paper, they dismissed natural language processing techniques as a good way to tackle the problem. They argued that the sparsity and size of a tf-idf Bag-of-Words vector, along with the inability to capture ordering information, make it unsuitable.

Vaswani et al. (2017) introduced the Transformer model. It dispenses completely with the previous recurrent and convolutional approaches to sequence representation, and only uses multi-head attention. This architecture requires significantly less training time, as it is simpler and more parallelizable in nature. The Transformer model is extremely popular because of this, and has a wide range of applications including, and not limited to, machine translation, question-answering systems, classification, and named entity recognition. Gniewkowski et al. (2021) have demonstrated their viability in representing sequence-aware HTTP headers, and dispenses with the need to use a bag-of-words approach, which Al-Hakimi and Bax (2021) found unsuitable.

Most modern of-the-shelf NLP frameworks are centered around the idea that a single sample is a single text document, but this is not the case for our experiment. Our samples are lists of individual documents, of which each document is - in principle - a complete HTTP response header with a typical length of ~100-1500 characters of plain text. The maximum token length supported by popular Transformer models such as BERT is 512, meaning that longer texts must be split up or truncated. Ideally, we would like to represent our samples as one continuous document split by control tokens, but that is computationally infeasible as it would result in over 10k tokens. This is not possible even with long-sequence Transformer models such as the Longformer (Beltagy et al., 2020) and Big Bird (Zaheer et al., 2020). Furthermore, we do not necessarily need to retain sequence information between test cases in the first place, as the HTTP requests are more or less independent from each other. It is likely sufficient to encode each test case separately, and then concatenate the resulting vectors to represent a single sample.

Transformer models are usually fine-tuned on a downstream task such as classification by attaching a different head to the last hidden layer. Given that we must encode our test cases separately, this approach will not work. Instead, we use the final hidden state of the Transformer as input for a separate feed forward neural network. Reimers and Gurevych (2019) note, however, that the raw, pooled final hidden state may not be a good representation of a document as a whole. In ordinary Transformer classifiers, this is not a problem as the encoder is immediately fine-tuned on a downstream task.

When using the output of a Transformer encoder directly as an embedding, Reimers and Gurevych (2019) suggest fine-tuning it on a semantic textual similarity task to generate superior semantic embeddings. This unfortunately requires an annotated corpus. There have been recent advances in unsupervised sentence embedding learning as well, for example by Wang et al. (2021).

The test cases we use to construct fingerprints from web servers have been designed based on strategies discussed in the literature by Lee et al. (2002), Book et al. (2013), Huang et al. (2015), Yang et al. (2010), and those employed by existing rule-based fingerprinting tools. We tried to find a good balance between exhaustively trying all thinkable interesting scenarios, and minimizing bandwidth, storage, and model complexity. We list the specifics of each test in appendix A.5.

In almost all circumstances, the Host: and Accept: headers are expected as part of a valid HTTP request. Many web servers host multiple websites, and the Host: header is used to disambiguate which specific website is requested. The Accept: header indicates what type of content is acceptable for the client. These two headers, along with User-Agent:, are always sent with standard HTTP-request tools such as curl and wget, and we therefore believe it is reasonable to include those headers by default. The HEAD method is identical to a GET request, but instructs that the server should not actually send any actual website content. The HTTP/1.1 and HTTP/2 specifications indicate that the HEAD method must be supported by compliant web servers. Using HEAD instead of GET is bandwidth-efficient, and we use it as the default request method for that reason.

We only perform requests that are marked as safe and idempotent by the HTTP specification, and ones that will plausibly not leak private information. Concretely, this means that we will limit ourselves to the GET, HEAD, and OPTIONS methods, as Book et al. (2013) did. We will also not attempt a path traversal attack by making relative requests, even though this was suggested as a fingerprinting strategy by Lee et al. (2002) and implemented by Book et al. (2013), as it may leak private information. In Figure 1 an example of an HTTP HEAD request can be seen. This example is equivalent to the baseline (test case 0) directed at the www.rug.nl domain.

Notably missing from the test suite are (malformed) absolute URLs in the first line of the request (Lee et al., 2002), and requests of a large size (Lee et al., 2002; Huang et al., 2015). We chose not to include these as we would have to test multiple absolute URLs to get meaningful differences for the first scenario. For the second scenario, we would have to try at least four tests with arbitrarily chosen URL lengths according to Lee et al. (2002) and Huang et al. (2015). We therefore consider these two scenarios to be too time and storage expensive.

In this section, we describe the algorithm that we devised in order to gather our raw data. As input to our algorithm, we use domains from the Tranco list by Pochat et al. (2019). This is a robust ranked meta-list of millions of domains, not unlike the Alexa dataset which Book et al. (2013) used. We were able to gather data for the top 4.8 million domains in that list, over the span of two days.

We use two types of target labels: major and minor. The major label is the web server type, such as nginx, and the minor label is the web server type and version combined, such as nginx/1.18.0. As mentioned in the introduction, selecting and creating such target labels is non-trivial. Some of the issues are:

• •

  the number of different web servers in existence;

• •

  the differences in formatting (e.g., capitalization; use of special characters; spacing);

• •

  the Zipfian distribution of web servers (i.e., the dataset is highly unbalanced, with the tail consisting of web servers with only a handful of examples).

Below, we discuss some of the strategies to tackle these issues.

In Table 1 we make a comparison between the distribution of server types, as reported by different organisations. The distribution of our target labels appears to be similar to Book et al. (2013), and general surveys done by Netcraft⁹⁹9https://news.netcraft.com/archives/2022/11/10/november-2022-web-server-survey.html and w3techs¹⁰¹⁰10https://w3techs.com/technologies/overview/web_server. As the methodologies of the sources vary, direct comparisons cannot be made and the table should be considered to contain a rough approximation only.

For this experiment, we have chosen an uncomplicated approach to sample and label filtering. The five major server types we use in this experiment are popular, publicly available, general purpose origin servers which make up approximately between 40% and 60% of the World Wide Web. As caches, content delivery networks, load balancers, and proxies may unpredictably affect the result, we have chosen to omit those entirely¹¹¹¹11This is based on self-advertised Server: headers. It is possible some samples are behind one of those systems but do not advertise it.. This causes our dataset to be unrepresentative of the true distribution of the World Wide Web, as a part of our raw data is removed. For example, the Cloudflare CDN constitutes a sizable 13% of the data. After removing poor samples and filtering the target classes, approximately 1.96 million out of 4.77 million samples remain. 24% of these contain a minor label. The filtering of classes and samples are explained below.

We used the RoBERTa model by Liu et al. (2019) as implemented by Wolf et al. (2019), and its corresponding byte-level BPE tokenizer. All settings were left at their defaults¹⁵¹⁵15See the transformers 4.23.1 package and roberta-base configuration ., except for the vocabulary size, which was set to 2048 instead of 30522. This seemed appropriate as the vocabulary of HTTP headers is significantly smaller than human language. RoBERTa was chosen as it achieves greater performance than BERT in most benchmarks, and was used by Gniewkowski et al. (2021).

The model and tokenizer were trained from scratch on the unique sentence list described in the previous section. The training objective for the model was masked language modelling, with a masking rate of 0.15. MLM was introduced by Devlin et al. (2019) and is an unsupervised auto-encoding method to pre-train Transformer architectures with context-aware linguistic information, after which the model can be used in a downstream task. Individual tokens in the input documents are randomly masked, and the model is then trained to fill in the missing token. Our model did this for 30 epochs, after which it had a loss of 1.17. The validation plot indicated that we had reached a point of diminishing returns. Considering the simplicity of the language (HTTP status lines only) and the risk of overfitting, we found further training not to be necessary for this particular experiment.

Having trained the Transformer encoder, we used it to generate a vector-space representation for a single document (i.e., status line from a single test case). This document is tokenized with padding and truncation, and fed through the RoBERTa model. For this we disable gradient calculation and apply mean pooling on the final hidden layer as implemented, and recommended by, the SentenceTransformers project¹⁶¹⁶16https://www.sbert.net/examples/applications/computing-embeddings/README.html#sentence-embeddings-with-transformers (Reimers and Gurevych, 2019). This results in a raw, 768-dimensional sentence-level embedding.

We do this for all lines in the unique sentence list (section 3.5.3) and fit a 64-component principal component analysis model on them. We chose to implement dimensionality reduction through PCA, as it was suggested by the SentenceTransformers project¹⁷¹⁷17https://www.sbert.net/examples/training/distillation/README.html#dimensionality-reduction. They found that this has a very limited impact on subsequent model performance on semantic textual similarity tasks, while significantly reducing computational and storage complexity.

Then, for each sample in the dataset:

• •

  we generate pooled embeddings for all 32 test case responses individually;

• •

  we apply a PCA transform on those 768-dimensional vectors to reduce them to 64 dimensions;

• •

  we concatenate those embeddings into a single 2048-dimensional vector.

A ready-to-use dataset is then created. Each sample contains the major class label, minor class label, and a 2048-dimensional float32 array. This dataset is randomized, and split into a train set (80%), validation set (10%), and test set (10%).

See Figure 3 for a visualization of the dataset. These are the Transformer-encoded, PCA-reduced, concatenated representations of individual web servers. There are clearly visible clusters discernible to the human eye, indicating that the test case response data and generated embeddings are appropriate for distinguishing between web servers.

In Tables 2 and 3 we show the classification reports for both classifiers. In these tables, support is the amount of samples in the test set which have that class as the true label. In appendix A.6 we list Figures 8, 9, 10, and 11, which are the confusion matrices for the respective classifiers.

We establish our baseline scores by evaluating a dummy classifier, which infers the most frequent class from the train set and then always predicts that class on the test set. This represents a naive but clever guessing strategy. Any classifier must be able to beat this baseline for it to be considered "better than random".

We consider macro f1-score to be a good evaluation metric, as it compensates for the class imbalance and dominance of nginx and Apache. The baseline macro f1-score for this task is 0.12. Book et al. (2013) did not report any of the common classification metrics, and tested their Bayesian model against a portion of their training data, instead of a reserved test set. Although not explicitly described as such¹⁸¹⁸18They described their accuracy somewhat confusingly as: ”our algorithm produced different results from the server type returned by the site for approximately 38% of sites”. The most frequent type occurred in 41% of the cases, which makes their result better than majority guessing, but not enormously so., it appears that they achieved 62% accuracy with 8 classes.

Overall, both the FNN and RF classifier perform beyond all expectations, achieving 0.96 and 0.94 macro f1-score respectively. This suggests that all five web servers have a distinct set of fingerprints, which are apparent from status lines alone. As a simple RF classifier is able to discern this signal with remarkable accuracy, it is likely that these are surface-level fingerprints and are not necessarily learned from subtle, weakly defined relationships between test cases.

Both classifiers struggle with detecting Microsoft-IIS and openresty, relatively speaking. This matches with what is seen in Figure 3: these two classes are less well-defined than Apache, nginx, and LiteSpeed. It is, however, unknown whether this is caused by the limited sample size of those classes or because they are inherently more ambiguous.

Furthermore, it is possible that our performance is the ceiling of what is achievable given the possibility of a reported target label being forged. Forging may have been done manually, or through masking software. If a target label is forged, it cannot be expected that a system can reliably classify it, given that it is trained on the true signal that belongs to that class.

The minor version classification task was only performed by the FNN. The corresponding classification report can be seen in Table LABEL:tab:classification-report-ffnn-minortypes in appendix A.7. Note that due to the rarity of some target classes, they may not appear in all three splits, and may not have been learned by the classifier. Additionally, very rare classes may be forged¹⁹¹⁹19For example, microsoft-iis/99.99 has not been released by Microsoft but is present in the list of classes.. For these reasons, we consider the weighted f1-score to be a better metric than the macro f1-score for this task. The macro f1-score compensates for class imbalance by taking the unweighted mean of each label’s f1-score, and thus treats each class equally. The weighted f1-score, on the other hand, weighs the f1-scores by the number of true instances for each label, and thus assigns a greater importance to more frequent labels. In this case, more frequently occurring labels are more likely to be authentic.

The baseline weighted f1-score on this challenging 347-class classification task is 0.02. Given that only the first line of the HTTP header is used in each test case, a weighted f1-score of 0.55 for a FNN is impressive. This is a non-trivial improvement in classification performance.

Our hypothesis was that it would not be possible to discriminate minor versions from status lines alone. Clearly, this is wrong. It is plausible that, given complete headers, more data, and longer training time for the Transformer encoder and feed forward classifier, better performance can be achieved. There is no apparent reason why our proposed method would not scale to more classes and more data.

In Figure 5 in appendix A.8, the calculated Gini impurity can be seen. The Random Forest classifier provides this metric on a per-feature basis. We divided the 2048 features by 64, rounded them down to the nearest integer, and took the mean of the resulting values to construct an interpretable average per test case.

Tests run on HTTPS/443 appear to be more important than those run on HTTP/80 overall, although the relative ranking of tests appear to roughly align. This matches with what is seen in the data: many HTTP/80 responses are 3xx location redirects that point towards HTTPS/443. This is also reflected in the discrepancy in importance between protocols for test cases 12-15. These tests depend on whether actual content is served: location redirects do not have those.

The most powerful predictor is test case 7. Looking at the data, Apache almost always causes an <ERROR> token, whereas the other web servers generally return 404 Bad Request. The second most powerful predictor, test case 2, appears to fingerprint Microsoft-IIS. This web server never supports HTTP/2 on port 80: this target class is thus ruled out if any response at all is given. Test cases 8 (invalid HTTP version) and 9 (invalid method) were suggested as fingerprints by almost all previous papers. The high Gini impurity of these features thus matches the literature. Explicit support for HTTP/1.0 as tested by cases 4 and 5 is also an important predictor. Furthermore, test case 0 is the lowest ranked test for both protocols. This matches what is expected from a baseline, standard HTTP request. Test case 15 appears to hold no predictive power on both protocols. It is possible that it is too dependent on the actual content of a website to be a meaningful discriminator of web servers.

Given that the learned feature importances match our intuitions and the general patterns in the data, it is likely that our data collection, processing, and embedding were done correctly. There are no odd or otherwise unexpected results.

Although the five selected target classes make up roughly half of the World Wide Web, they are only a small subset of all web server types in existence. It remains unknown whether precision and recall of rare server types will be similar. The proposed system should scale to many classes and work with entire headers, but this is yet to be demonstrated in practice.

It is important to note that there are some fundamental issues with our automatic data collection. These were also identified by Book et al. (2013), except for a population sampling issue they did not consider. We have also not performed a thorough statistical analysis of our data, and thus cannot draw conclusions about the fundamental correctness and integrity of our dataset. Overcoming the limitations of the data is possible, but requires careful selection, annotation and verification of its correctness. The amount of manual work required for that is significant, and makes us wonder whether it doesn’t defeat the original purpose of a machine learning approach requiring less effort than a rule-based system.

Besides analyzing and solving the aforementioned limitations of this work, we have identified potentially other interesting topics for future research.