WeaverBird: Empowering Financial Decision-Making with
Large Language Model, Knowledge Base, and Search Engine

The system firstly first ranks all the $2K$ search results (or $K$ results if the internet search has been skipped as discussed in section 2.1) based on their cosine similarities with the query (i.e., the second step in Figure 1).

After receiving the ranking results from the previous step, we plug the top $J$ (where $J<K$) results into the context part of the predefined prompt template. Since the knowledge of finance domain is dynamically evolving, shown in LABEL:lst:en_template, we ground each retrieved result and question with temporal facts (e.g., published date and question date) to facilitate the language model to adapt to time-evolved knowledge, possibly making the response more meaningful and accurate (** et al., 2023b).

Having temporal metadata for questions and retrieved results allows us to ask questions with relative time specifications (e.g., “3 months ago”), which are often used in finance queries. Moreover, it enables us to ingest new knowledge periodically, in a streaming setup, and evaluate on questions asked during that period (Liška et al., 2022).

The large language model organizes the information to form a fluent and informed response, corresponding to the third step in Figure 1. The response also includes accurate references to the sources of the information. As an option, our system can further provide a list of citations for the remaining $(K-J)$ paragraphs at the end of the response, to complement the main response and provide users with additional recommended reading materials.

Our large language model has undergone adaptation to the finance domain, thus being able to skillfully process the input information and generate appropriate responses. We started with a pre-trained GLM (Du et al., 2022; Zeng et al., 2022) that has been pre-trained using extensive English and Chinese corpora. Then we fine-tuned this model using our collected Chinese and English financial documents as well as expert-annotated financial query-response pairs, which we introduce in section 3.

For each document, we can construct a query-response pair that tunes the language model to perform content generation. To create the query, we fill the template “Please write a [TYPE] on [TOPIC] with the title [TITLE]” where TYPE, TOPIC, and TITLE can be found in the corresponding text file. The SUMMARY part of the file is used as the response. An example of query is “Please write a review report on China economics with the title CHINA POLICY: POLITBURO MAINTAINS PRO-GROWTH STANCE” and its response is “The politburo indicates continuous pro-growth policy without big stimulus…”

The reverse of a content generation pair can be used as a content summarization pair. Now the query will be created by filling the template “Please summarize the following [TYPE]: [SUMMARY]” and the response is the TITLE. An example of query is “Please summarize the following report: NIO’s 4Q22 earnings missed our prior forcast…” and its response is “NIO INC. (NIO.US): 4Q22 miss”.

Given a document, we would like the model to answer any questions that could be answered by directly identifying the relevant information from the document. This is the focus of the close-book QA research (Roberts et al., 2020). To achieve this goal, we need a specific kind of query-response pairs: query is the question and the document; response is the answer. We created such pairs using the question generation method proposed by Rajpurkar et al. (2016). Precisely, given a document, we used a pre-trained question generation model to identify some interesting information in the document and then generate a question whose answer should be that piece of information.

We would also like the system to answer open questions: unlike close-book QA, the evidence is not available in the context. We created this kind of Chinese question-answer (i.e., query-response) pairs by scrap** well-known online QA forums such as Zhihu and SinaFinance. On such forums, users post questions, tag questions, answer questions, and upvote or downvote answers. We identified 6,000 highly recognized questions with the “finance” tag; with each question being the query, we took the answer with the most upvotes to be its response. We then randomly sampled 4,000 Enligsh open QA questions from the English finance QA dataset FinQA (Bharti, 2023).

The system is based on the pre-trained ChatGLM2-6B (Du et al., 2022; Zeng et al., 2022), and we fine-tune it with the data and training strategy specified below.

We use the collected query-response pairs clarified in section 3.2 for training and evaluation. The whole set of pairs are divided into sets of 17,000 training pairs, 3,000 development pairs, and 4,100 test pairs.

We apply the LoRA (Hu et al., 2021) strategy to fine-tune the base model to adapt to the finance domain. The LoRA layer details are as follows: the LoRA rank is 16, the LoRA alpha is 32, the dropout is 0.1, and other settings are default.

The key and query encoders $f_{\text{key}}$ and $f_{\text{query}}$ are initialized as Multilingual-E5-base (ME5) model architecture (Wang et al., 2022).

Recall from section 2.2 that we need query-paragraph pairs to train the key and query encoders $f_{\text{key}}$ and $f_{\text{query}}$. We use the collected query-response pairs clarified in section 2.3 for training and evaluation. The chosen pairs were then divided into sets of 700 training pairs, 100 development pairs, and 200 test pairs.

We pass query-response pairs to the model to yield a scalar score for each of the pair and maximize the scores for the positive pairs while minimizing the scores for the negative pairs with the cross entropy loss.

For the implementation of LangChain, we use the code from the public GitHub repository at https://github.com/imClumsyPanda/langchain-ChatGLM (LangChain-ChatGLM, 2023) without any License. For the implementation of WebUI, we use the code from the public GitHub repository at https://github.com/THUDM/WebGLM (Liu et al., 2023) with Apache 2.0 License.

All the experiments were conducted on a server with 128G RAM, a 64 logical cores CPU (Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz) and 8 NVIDIA A100 80G Tensor Core GPUs for acceleration. During the fine-tuning process on the finance dataset, we set the train batch size on each device equal to $8$, the gradient accumulation step equal to 100, the max step equal to $10,000$ and the learning rate equal to 0.0001, and the running time of each step is close 72 seconds, the total running time is close to $200$ hours.

Our system is deployed on a server on Alibaba Cloud with 30G RAM, a 8 logical cores (Intel Xeon(Ice Lake) Platinum 8369B), and a NVIDIA A100 80G Tensor Core GPU. The inference speed depends search speed on the website, searches in local knowledge only take 1 second, and the search on the website takes 10 seconds on average.

We use the test set specified in section 5.2 for evaluation. When evaluating a method (i.e., a specific model paired with a specific similarity metric), we take the top $K=5$ highest-ranked retrievals for each given query, and evaluate the mean average precision (MAP) and mean average recall (MAR) on the entire test set of query-paragraph pairs. Intuitively, MAP measures the fraction of the predictions that are correct, while MAR measures the fraction of the actual paragraphs that are covered by the predictions. Higher is better.

The results are in Figure 3. As we can see, trained encoders work significantly better than pretrained encoders, indicating the effectiveness of our training method. Moreover, our ME5-based encoders significantly outperform the Contriver-based encoders: it is perhaps because a large portion of our test cases are in Chinese and ME5 is a bilingual model. Interestingly, cosine similarity turns out to work the best among all the similarity metrics, even though our training objective uses dot product.

. We use the test set specified in section 5.1 for evaluation. Since automatic evaluation for complicated natural language QA still remains an open research question, we rate the response by both human expert and GPT-4 (OpenAI, 2023) due to its efficiency and consistency (Hackl et al., 2023): each response is given a score in $[0,100]$ by a financial expert and GPT-4, respectively and the final score is an average of the two. Higher is better. We compare WeaverBird with WebGLM, FinGPT and FinChat. BloombergGPT (Wu et al., 2023) is non open-sourced so it is excluded from the baselines. Due to the budget constraint, we evaluate FinChat, a commercial chatbot in finance, on a subset of 300 questions sampled from the test set.

The strongest results of our WeaverBird system is summarized in Figure 3. Although WebGLM is also bilingual and empowered with search engine, WeaverBird has superior performance mainly due to its domain-finetuned base model, trained encoders on financial documents as well as optimized retrieval on knowledge base and searched results. FinGPT has lower score than WeaverBird and WebGLM because it has not used multi-source knowledge. FinChat performs poorly because it is not billingual and its knowledge base covers a restricted list of stocks.

Firstly, we remove the knowledge base (‘wv-wo-kb’ in Figure 3) and this strategy causes a moderate drop in performance. Secondly, we remove the search engine (‘wv-wo-se’ in Figure 3) and the performance declines with a notable drop, indicating the web search ability, which introduces the ‘global knowledge base’ on the internet, greatly improves the accuracy of responses. To conclude, the searched results contribute most to the response quality while the local knowledge base contributes moderately.

We randomly pick up $500$ examples of query-response pairs generated by WeaverBird and group the responses by the accuracies of retrieved documents (annotated by the experts). Not surprisingly, Figure 4 shows that higher accuracy of retrievals brings significant improvement on response qualities.

We manually analyze $50$ examples of finance-related query-response pairs annotated by the experts. Of these $50$, $70\%$ cases arrived at reasonable responses. However, we also found some flawed output, most of which can be corrected with one of the following classes of modification:

• •

  Increase $K$ to have more searched results; increase $J$ to have more input for LLM; or increase both $K$ and $J$.

• •

  The searched results usually consist of a number of relevant paragraphs instead of the full content of the web page. If needed, by using the functionality provided by our system, the user can choose to scrape the site and save the content to the local knowledge base. Given the full text, more relevant paragraphs are retrieved and better responses are possibly generated.

In this section, we present two illustrative examples. The first example poses a challenging question regarding the holdings of a renowned fund manager. Obviously, WeaverBird outperforms the competitors by delivering a reasonably accurate response, accompanied by citations to credible sources.

The second example is in Chinese. As we can see, our WeaverBird system exhibits superior performance in answering Chinese finance-related questions, which neither WebGLM nor FinChat could answer well.