OpsEval: A Comprehensive IT Operations Benchmark Suite
for Large Language Models

Our benchmark questions have been collected from various sources; we summarize them into four categories: company materials, certification exams, Ops textbooks, and automated generation. Each source is highly esteemed globally and reviewed by our collaborators.

Company Materials include production environment materials like Ops tickets and error logs, as well as internal documents and tests for Ops staff training. We have established cooperative relationships with 10 companies and received materials from them. The companies and enterprises we collaborate with cover various sectors, including Internet, telecommunications, finance, and Ops service/tool providers. Each company has designated experts to work with us to select questions. Information about the companies and experts can be found in the appendix.

Certification Exams include knowledge assessments necessary for becoming an Ops staff and are naturally in the form of multiple-choice and short-answer questions. We obtained the relevant study guidebooks for these certification exams from public book websites and extracted sample questions from them as one of the sources for Ops questions.

Operations Textbooks. We first constructed a seeding keyword list for the Ops field and searched for related books. The textbooks contain relatively complete knowledge content, which can provide experts with materials for question creation, and some books themselves also include a certain number of exercises at the end of the chapters.

Besides the sources described above, to enhance the diversity and depth of our test set, we also source question-answering questions from authoritative books covering a range of Ops domains by extracting textbook contents and providing them to GPT4. In Sec 6, we discussed the methodology and challenges in Automated Generation, ho** to provide some experience in Ops benchmark creation.

By the end of the data collection process, we have collected 21250 questions in total, which are to be further selected and curated.

We systematically carried out the processing of our original test set in the following stages:

Deduplication: Any repeated or highly similar questions are identified and removed to avoid redundancy in the test set. We calculate the cosine similarity of the question stems to detect duplicate questions and identify pairs of questions with a similarity above a certain threshold (th=0.7). These pairs are later manually reviewed to confirm if they are duplicates.

Dependance Filtering: Since OpsEval primarily focuses on the ability of large language models to understand Ops knowledge, we have filtered out questions that rely on external images or document content to ensure the completeness of the question content itself.

Question Categorization: In the complex landscape of Ops, recognizing the multidimensional nature of tasks is essential. We devise a categorization that captures many tasks that professionals confront in practical applications. The categorization process consists of two steps: automated screening and manual review.

We first use GPT-4 for topic modeling to gain rough insights about the dataset, leveraging its capabilities to determine the relevance of each question to Ops. The topic modeling resulted in more than 20 tasks but had an imbalanced distribution. The prompt we used to let GPT-4 do the categorization is shown in Figure 2(a).

Secondly, during the manual review process (as we will introduce this step later in this section), we involved dozens of experts in manual screening. By merging similar tasks and labeling different ability levels, we categorize the questions into eight tasks and three ability levels, respectively.

• •

  Task Categorization: Our eight distinct Ops tasks are formulated by industry relevance, task frequency, and the significance of each area in Ops. Details of the eight tasks can be found in Appendix A.2.

• •

  Ability Categorization: Based on which ability is required to answer them, questions are classified into three categories: Knowledge Recall, Analytical thinking, and Practical Application, reflecting the challenges professionals might encounter in real-world scenarios. The three abilities are described in Appendix A.3.

The distribution of the questions across these eight tasks and three ability levels is depicted in Table 1.

Manual Review: In the manual review step, we asked Ops experts from the industry to inspect the results of the previous three automated steps, including confirming duplicate and invalid questions and examining the classification results of GPT-4. Experts were also asked to drop the questions unrelated to the Ops field. We split the dataset by n-folds and ensure each fold has at least two experts to review. By involving experts’ consensus, we ensure the reliability and authority of the dataset after automated processing.

As listed in Table 2, this quality enhancement process resulted in a refined test set of approximately 7,000 multi-choice and 2000 question-answering questions.

After quality enhancement of the collected questions, we perform question formatting for the upcoming evaluation. Each question is structured to include the context, answer, task, and ability. Figure 2 shows three examples of the formatted questions. This structured approach facilitates easy evaluation and analysis.

In our evaluation of LLMs within the Ops domain, we categorize our assessment questions as multi-choice and question-answering.

Multi-choice questions offer a structured approach with definitive answers. These questions are straightforward and provide a clear metric for assessment. However, given the intricacies of advanced models, they may be influenced by the options provided, leading to their responses being driven more by pattern recognition rather than a proper understanding of the content.

We use accuracy as the metric. The output of LLM may contain additional strings besides the chosen option, even if we make explicit requests and use few-shot in the prompt. Therefore, we use a choice-extracting function based on regular expressions to extract the predicted answer of LLMs. Then, we calculate the accuracy based on the extracted answer and the ground-truth labels.

Question-answering questions do not come with predefined options. It necessitates the model to rely more on its comprehension and knowledge base, offering a clearer insight into its cognitive capabilities. Such questions can better assess LLMs’ ability to generate coherent and contextually relevant responses.

We use two metrics for question-answering questions: one is based on word overlaps, and the other is based on semantic similarity. For the first type, we use ROUGE (Lin, 2004) and BLEU (Papineni et al., 2002), widely used in NLP tasks, especially in the translation task. For the second type, we use GPT-4 and experts to obtain the output score of LLMs, called GPT4-Score and Expert-Evaluation, designed explicitly in OpsEval. We provide a more detailed description of these metrics in Appendix A.5.

For GPT4-Score, we provide GPT-4 with the question and the reference answer, followed by one anonymous model’s output. Figure 2(b) shows the prompt we used to ask GPT-4 to score.

We design expert evaluation for manually scoring LLMs’ outputs based on three criteria highly related to Ops’ needs. The three criteria in consideration are as follows:

• •

  Fluency. Assessment of the linguistic fluency in the model’s output and compliance with the question-answering question’s answering requirements.

• •

  Accuracy. Evaluation of the precision and correctness of the model’s output, including whether it adequately covers key points of the ground-truth answer.

• •

  Evidence. Examine whether the model’s output contains sufficient argumentation and evidential support to ensure the credibility and reliability of the answer.

For each output to a question of an LLM, we asked experts to score it between 0 and 3 for each criterion. During the scoring, the raw question, the detailed answer and its key points, and the output of an anonymous model are given at each iteration. Since avoiding bias towards specific models is important, no information about the model is given.

Fairness Consideration We are currently making 20% of the data available to the public for Ops community contribution and research purposes, yet for fairness of the evaluation, the complete version of OpsEval dataset is kept private and not disclosed. To evaluate a new model, users can submit a Docker image with an initialization script when starting a container based on it. We will run the evaluation automatically and obtain the result on the OpsEval website. Users can choose to disclose their results on the leaderboard of OpsEval or not based on their preference.

As shown in Table 3, we evaluate popular LLMs covering different weights from different organizations. The detailed information of all LLMs in Table 3 can be found in Appendix B.1.

Besides, we evaluate LLaMA-2-70B with multiple quantization parameters to get an overview of the effect of different quantization parameters. Specifically, we use GPTQ (Frantar et al., 2022) models with 3-bit and 4-bit quantization parameters.⁴⁴4The two quantization models are downloaded from https://huggingface.co/TheBloke/Llama-2-70B-chat-GPTQ. The size of LLaMA-2-70B is calculated based on 70B parameters. The two GPTQ models evaluated in our experiments are calibrated on wikitext (Merity et al., 2016), a language modeling dataset extracted from the set of verified Good and Featured articles on Wikipedia. GPTQ is a post-training quantization (PTQ) method to make the model smaller with a calibration dataset. It is a one-shot weight quantization method based on approximate, highly accurate, and efficient second-order information. The details of GPTQ models can be found in Table 4.

To get a comprehensive overview of the performance of popular LLMs on OpsEval, we use various settings to perform the evaluation. We evaluate LLMs in zero and few-shot settings (3-shot in our implementation). For each setting, we evaluate LLMs in four sub-settings of prompt engineering, that is, naive answers (Naive), self-consistency (SC (Wang et al., 2023b)), chain-of-thought (CoT (Wei et al., 2023a)), self-consistency with chain-of-thought (CoT+SC). We design prompts for both English and Chinese languages.

Naive. The Naive setting is to expect the LLMs to generate the answer without any other explanations. Since we have the task type of each question, we integrate the task into the prompt.

SC. Self-consistency is selecting the most consistent answer among several queries on LLMs. Although it aims “to replace the naive greedy decoding used in chain-of-thought prompting,” it can generate naive answers as it may generate different answers with the same prompts. We set the number of queries in SC to 5.

CoT. The CoT setting aims to enable LLMs to obtain complex reasoning capabilities through intermediate reasoning steps. We construct specific prompts for CoT setting in both zero and few-shot evaluations. Details of the prompt construction can be found in Appendix B.2.

CoT+SC. We combine CoT and SC to boost the performance of CoT prompting. By SC, We choose the consistent reasoning path and answer several of the same queries. Like the SC setting, we set the number of queries in CoT+SC to 5.

As for question-answering problems, we combine each question’s task type and ability and the question as the prompt for LLMs. Figure 9 in Appendix A.4 shows an example of constructing the prompt.

As a benchmark suited for LLM, it is necessary to ensure that the test set has not been included in the model’s pretraining process. Otherwise, the potential bias may damage the fairness of the benchmark. We adapted the methodology from Skywork (Wei et al., 2023b) to perform a leakage test on OpsEval’s dataset. We evaluate the LLM loss on samples (a sample is a concatenation of question and answer) from different datasets for several foundation models and calculate the average loss. For each dataset, we compare LM loss on the test split ($L_{test}$) and a specially curated reference set ($L_{ref}$ ) generated by GPT-4, designed to mimic the testing dataset. We define a key metric: $\Delta L=L_{test}-L_{ref}$, serving as an indicator of potential test data leakage during the training of the LLM, i.e., a lower value suggests possible leakage. Notice that while Skywork (Wei et al., 2023b) only asked GPT-4 to generate similar questions to GSM8K, we require GPT-4 to rewrite the question while preserving its original meaning and accuracy. By preserving the original meaning, if the $\Delta L$ of the test set is a lower value, it suggests that the LLM’s lower $L_{test}$ originates from over-fitting the distribution of tokens in the test set rather than internalizing the knowledge behind the questions, thereby suggesting a leakage in the test set.

Table 5 shows the results of leakage measurement. In addition to the two standard evaluation benchmarks (CEval (Huang et al., 2023) and MMLU (Hendrycks et al., 2021)), we conducted the same experiments on the alpaca dataset (Taori et al., 2023) and the Alpaca-GPT4 dataset (Peng et al., 2023), which is likely used in the pre-training of large models, using its $\Delta L$ as reference. The corpora likely involved in training show a significantly smaller $\Delta L$, whereas the loss for the OpsEval dataset remains at a relatively small positive value. This demonstrates the unbiased nature and non-leakage of the OpsEval test set. The models we used in the leakage test are listed in Appendix B.1.

The results of the few-shot evaluation with four settings on the Wired Network Operation test set are shown in Table 6. Results of the other scenarios and settings are shown in Appendix B.3. ⁵⁵5Due to the consideration of time, cost, and API rate limits, for GPT-4, we only make the 3-shot evaluation with the CoT setting to serve as an upper bound of all LLMs to provide a reference. From the overall performance results, we can come to several findings.

On both English and Chinese questions, GPT-4 consistently outperforms all other models, surpassing the best performances of all other LLMs. LLM with larger parameter size like Qwen-72B-Chat, ERNIE-Bot-4.0, and LLaMA-2-70B, when employing the Self-Consistency and Chain of Thought (CoT) prompt methods, approach the performance of ChatGPT. Smaller models, such as DevOps-Model-14B-Chat and Qwen-14B-Chat, exhibit competitive performance in multi-choice questions, approaching the capabilities of models with 70B parameters, thanks to their fine-tuning process and the quality of their training data.

Furthermore, the effectiveness of the four prompt settings varies across different LLMs. We examine LLMs’ zero-shot and few-shot performances under four settings mentioned earlier in Sec. 4 for both English and Chinese test sets. From the evaluation results, we can conclude the following observations:

1. (1)

   For most models, the performance improves from the setting of Naive to SC, CoT, and SC+CoT. Notably, few-shot performance is better than zero-shot performance.

2. (2)

   Among these settings, CoT prompts yield the most significant improvement in LLMs’ answering capability. SC prompts result in relatively minor improvements. LLMs’ responses are consistent across repeated questions, aligning with the desired outcome in operational tasks where reliability and consistency are essential.

3. (3)

   LLMs fine-tuned specifically for Chinese exhibit better performance on English and Chinese test sets than LLMs without Chinese fine-tuning. We discuss further insights into these observations in Appendix. B.5.

4. (4)

   In a few cases, more advanced evaluation methods surprisingly lead to poorer results. The detailed analysis for this can be found in Appendix B.6.1.

To investigate how LLMs perform in each Ops task and to what extent they possess the abilities of Knowledge Recall, Analytical Thinking, and Practical Application, we summarize the result of different parameter-size We group LLMs’ performance based on the task and ability classification mentioned in Sec. 3.2 and plot them on two radar charts in Figure 4.

In terms of the eight tasks we tested, LLMs generally yield higher accuracy in General Knowledge tasks, while their performance drops and varies drastically in highly specialized tasks like Automation Scripts and Network Configuration, reflecting the impact of specialized corpus and domain knowledge on the performance of LLMs.

Among the three abilities, LLMs perform the best in Practical Application, followed by Knowledge Recall. It is foreseen that LLMs perform poorly in Analytical Thinking questions, as accurately deducing conclusions from existing facts for LLMs remains a challenging research topic. LLMs perform best in Practical Application because the LLMs we test are trained on the corpus where best Ops practices are involved, familiarizing the LLMs with solutions to many real-world tasks.

By grou** LLMs by their parameter size, we find that although LLMs with 10B-20B parameters have higher accuracy in their best cases compared with LLMs with no more than 7B parameters, different 10B-20B LLMs’ performance varies drastically, sometimes even lower than that of 7B. LLMs with no more than 7B parameters, on the other hand, have a more stable performance range within the group.

Table 7 presents the evaluation results of 200 question-answering English questions across four metrics: ROUGE, BLEU, GPT4-Score, and Expert-Evaluation, sorted by GPT4-Score results. We have conducted question-answering tests on GPT-4 to verify its capabilities before using it for scoring. Since GPT-4 is currently the most capable among large models, its performance on question-answering questions surpasses other models. However, we did not include it in the table to avoid bias and misunderstanding.

The rankings based on ROUGE and BLEU scores do not align well with GPT4-Score and Expert-Evaluation, as shown in table 8. LLMs with poor performance may generate keywords, resulting in higher ROUGE and BLEU scores. In contrast, LLMs with good performance might receive lower ROUGE/BLEU scores due to differences in wording compared to the standard answers.

Regarding GPT4-Score, the rankings closely resemble those based on Expert-Evaluation. In table 9, we calculate the correlation coefficients between GPT4-Score and different sub-metrics of Expert-Evaluation to gain more insights. Among the three metrics, rankings of GPT4-Score align most closely with the Accuracy metric, suggesting that GPT4 is most reliable on the factuality with its vast knowledge base. The format and length of the generated content also heavily influence GPT4-Score, as suggested by the high positive correlation between GPT4-Score and Fluency. On the other hand, there are more discrepancies in rankings concerning the Evidence metric, indicating that GPT4-Score needs to consider the role of arguments and evidence in cases where answers are ambiguous.

In the Expert-Evaluation, where Evidence is a significant criterion, LLMs with more elaborate arguments can outperform GPT-3.5-turbo in total scores even when their Accuracy scores are much lower than the latter.

Figure 6 shows the accuracy of LLaMA-2-70B of different quantization parameters on English multi-choice questions. We do both few-shot and zero-shot evaluations with the naive setting. Using quantization during inference brings a performance degradation.

LLaMA2-70B-Int4 can achieve an accuracy close to LLaMA-2-70B without quantization. Specifically, on English multi-choice questions, the accuracy of the GPTQ model with 4-bit quantization parameters is 3.50% lower in zero-shot evaluation and 0.27% in few-shot evaluation compared to LLaMA-2-70B. As for Chinese questions, the accuracy of LLaMA2-70B-Int4 is 3.67% lower in zero-shot evaluation and 5.18% in few-shot evaluation compared to LLaMA-2-70B.

However, LLaMA2-70B-Int3 has a performance degradation that cannot be ignored, as shown in Figure 6. On average, the accuracy of LLaMA2-70B-Int3 has a 12.46% degradation compared to LLaMA-2-70B and a 9.30% degradation compared to LLaMA2-70B-Int4. The reason may be that the information of the full-sized model is lost too much in 3-bit quantization.

We summarize our key observations from the evaluation here:

• •

  Few-shot and CoT can significantly increase performance if the model is tuned to adapt to these techniques, while SC may have little influence on highly consistent LLMs.

• •

  LLMs perform best in general knowledge while deducing specialized conclusions remains challenging.

• •

  GPT4-Score is suitable as an automatic metric in large-scale qualitative evaluations.

• •

  Quantization with more than 3 bits can be an effective way to reduce computation and memory costs while preserving performance.

Overall, LLMs yield evaluation results on the OpsEval benchmark generally lower than those in general domains like MMLU (Hendrycks et al., 2021) and CEval (Huang et al., 2023). This comparison highlights the necessity of explicitly fine-tuning OpsLLM for the Ops field.

Our tests on the model with various prompt engineering techniques indicate that prompt engineering significantly impacts the effectiveness of eliciting operations knowledge from the model. This suggests that further research into prompt engineering is needed to enhance the performance of large models in this specific vertical domain.

During the data collection process, we have considered automating question-answer generation. First, we sampled the question-answer pairs and manually assessed their accuracy and domain relevance. Later, we used typical manual evaluation examples for few-shot learning, enabling GPT to evaluate question-answer pairs based on our evaluation criteria automatically.

Directly generated question-answers tend to be simple judgment or concept questions rather than reasoning questions that better demonstrate the model’s capabilities and knowledge density. Our goal is to ensure that while the topics of the questions remain relevant to the seed questions, their specific content is distinct from the original questions. By maintaining the overarching framework in the Ops domain, we can expand the number and types of questions, enabling a more comprehensive evaluation of model capabilities. Additionally, we can incorporate external knowledge during the data generation, continually enhancing our ability to evaluate new content.

Recognizing the necessity for LLMs to alleviate the hallucination problem, we explore the corresponding evaluation methodology.

Our approach involves both multi-choice and question-answering evaluations. Multi-choice questions assess the LLMs based on their knowledge and diverse Ops capabilities. However, to effectively uncover hallucinations, we rely on question-answering evaluation. This involves expert review, where outputs are scrutinized against three main criteria: Fluency, Accuracy, and Evidence. The Evidence criterion, in particular, is pivotal in highlighting hallucination issues. It underscores the significance of LLMs providing evidence or a reasoning process to support their outputs.

The intersection of hallucination problems and evidence-based evaluation is a focal point in our research. By emphasizing evidence, we aim to mitigate the issue of hallucinations, ensuring that LLM outputs are not only plausible but also verifiable. We plan to refine our multi-choice question design. This will enable us to automate the evaluation of model robustness against hallucinations more effectively. This adjustment is anticipated to enhance further our understanding of LLMs’ capabilities and limitations in dealing with hallucinations, thereby contributing to develo** more reliable and trustworthy LLMs in the Ops field.