Real-time Speech Summarization for Medical Conversations

RTSS system proposed by [1] has 3 major additional components: flexible recognizer of utterance units, utterance lookahead-er, and information overrider. Flexible recognizer of utterance units automatically split real-time Automatic Speech Recognition (ASR) transcript into segments with random lengths, while utterance lookahead-er seeks additional context from subsequent words generated by ASR, and information overrider continuously updates the summary in response to the latest contextual changes. We conducted surveys and gathered feedback from engineers and found that these extra components not only extend both inference and training time but also complicate RTSS systems, making it challenging for engineers to deploy and maintain them effectively. Furthermore, the summary is constantly updated after each utterance generated by the ASR system. This results in increased computational costs when compared to a scenario where a solid summary is generated after a set of utterances. The continuously updated summary creates confusion as users are unable to keep track, given the uncertainty of when a summary is completed.

In contrast, our approach is much simpler. Our design generates a local summary after every N utterances of speech within a conversation and a global summary after the end of a conversation. Every local summary is generated using the corresponding local context of N utterances, without the need to continuously update the new context generated by real-time ASR utterances. Meanwhile, the global summary serves as an "overrider" using the context of the entire conversation.

RTSS system by [1] generates a summary with a delay of one utterance. A large number of delayed utterances results in longer waiting time for users to receive the generated summary. Conversely, a low number of delayed utterances means that the context necessary for accurate summaries is missing, making summarization unnecessary. After analyzing the context within the VietMed corpus [4] and conducting our internal user survey, we found that setting N = {4, 5} (or a maximum of around 30 seconds) strikes a suitable balance. This ensures that each summary includes an adequate amount of context without kee** users waiting excessively.

We used GPT-3.5 Turbo³³3https://platform.openai.com/docs/models/gpt-3-5-turbo (or ChatGPT) to generate summaries for every transcript in our dataset, which we refer to as GPT-annotated summaries. We then split the dataset into two subsets: Gold standard (GOLD) set and the Synthetic (SYN) set. On the GOLD set, we performed human editing where the human annotator edit the GPT summary according to the annotation guideline, while on the SYN set, we did not. More information on GPT annotation is on section 4.

Our GOLD set contains gold standard data from REAL and SIM, while SYN contains GPT summaries from the extra medical text mentioned above.

Details are in Data Annotation section in the Appendix.

Table 1 shows the statistics of our dataset. To construct our VietMed-Sum dataset, we keep the original split of REAL by [4] as 5-5-6 hours for the corresponding train-dev-test set. We split our SIM set with a ratio of 8:1:1 for the corresponding train-dev-test, while the entire SYN is used for training.

Acquisition and annotation of medical dataset is challenging and costly, resulting in medical summarization datasets typically being smaller compared to those in the general domain. Compared to other public medical written text summarization dataset, such as MeQSum corpus of summarized consumer health questions [5], our dataset has 23 times more summaries. Besides, compared to the Chinese medical text summarization dataset by [3], ours is half the size.

We also introduce VietMed-Sum-en, the English version of VietMed-Sum which was translated using Google Translate⁵⁵5https://translate.google.com/. Results are in the Appendix.

To the best of our knowledge, existing prominent Vietnamese summarization datasets, such as VietNews [6] and FAQSum [7], utilize their titles and abstracts as summaries. Our dataset, however, lacks these pre-made summaries which would traditionally require human annotation. However, finding high-quality annotators for either low-resource languages like Vietnamese or medical domain is hard [8].

In recent year, there has been an increasing focus on utilizing Large Language Models (LLMs) for annotation [9, 10, 11]. Experimental results from [12] showed that fully GPT-3 labeling can ourperform fully human labeling in low-budget settings. GPT has shown to have adequate medical knowledge [13]. Furthermore, it achieves reasonable performance as an annotator in sequence generation tasks in Vietnamese [10].

We used GPT-3.5 Turbo to generate GPT summaries. Full setup details are in the Appendix.

The experimental results from the $2.5-budget setting in Table 2 demonstrate the importance of human annotation in the Vietnamese medical summarization task. Since summaries are heavily influenced by the annotators’ medical knowledge and writing style, we hypothesize that GPT summaries still provide useful medical knowledge but require additional training on human summaries for writing style transfer.

To verify our hypothesis, we devise the the $5-budget setting where we fine-tuned ViT5 on GPT summaries, then on human summaries, and compared this two-step process with fine-tuning only on human summaries. We also doubled the number of human summaries for fair comparison. Experimental results show that the two-step process achieves slightly better results while costing significantly less time. We hypothesize that fine-tuning on GPT summaries helps provide medical knowledge and fine-tuning on human summaries helps the model aligns its output’s text style.

Table 3 further support our argument. When we perform two-step finetuning on ViT5, the performance gap between human summaries and GPT summaries is significantly closed. As such, our labeling strategy involves creating the SYN set and refining the GOLD set to balance between time-consumption, cost and performance.

We further investigate the characteristics of the GPT summaries empirically. In this experiment, we sample 50 arbitrary transcripts of varying lengths from VietMed-Sum and let two annotators independently summarize them, one with GPT summaries as references (human editing) and one without.

We use ROUGE [14], a metric commonly used for summarization, to evaluate our models. More details are in the Appendix.

We employed $\text{BARTpho}_{\text{syllable}}$ and $\text{BARTpho}_{\text{word}}$ [15], ViT5, ViT5-vietnews [16] (ViT5 fine-tuned on Vietnews summarization dataset [17]), and ViPubmedT5 [18] models in our experiments. More information about the models is in the Appendix.

Table 4 shows the ROUGE scores on our GOLD test set for the baseline models fine-tuned on the combination of GOLD local and global summaries. ViT5, ViPubmedT5, and ViT5-vietnews consistently outperforms the BARTpho variants.

Results from Table 5 shows the models have a noticeable drop in performance. On the local summaries, the ROUGE scores from the BARTpho variants are much lower than that of the ViT5 variants. Conversely, when fine-tuned on the global summaries, both variants of BARTpho performs much better than the ViT5 variants except ViT5-vietnews, probably because ViT5-vietnews was previously fine-tuned on other Vietnamese abstractive summarization dataset.

Table 6 shows the ROUGE scores of each model fine-tuned on SYN-only and with GOLD. We found that fine-tuning only on SYN data did not improve the models’ performance. However, when we incorporated GOLD into fine-tuning the models either by concatenating the GOLD and SYN data or by performing two-step fine-tuning SYN $\rightarrow$ GOLD, the performance of the models drastically improved compared to training only on GOLD on all metrics. This is consistent with our observations from Subsection 4.3.

We report the results of our baseline models on the ASR transcripts on Table 7. The performance of the models is worse than that on VietMed-Sum, which we attribute to the noisy nature of the text generated by the ASR models. Nevertheless, the ROUGE scores remain fairly reasonable, which is proof to our model’s robustness.

While ROUGE is commonly used to evaluate the performance of the models, does not measure the fluency and factual alignment of the summaries. As such, we adopt the human evaluation methodology from [19, 20] and report the results in Table 8. Details of experiments are in the Appendix.