-
Fine-Tuning Language Models for Scientific Writing Support
Authors:
Justin Mücke,
Daria Waldow,
Luise Metzger,
Philipp Schauz,
Marcel Hoffman,
Nicolas Lell,
Ansgar Scherp
Abstract:
We support scientific writers in determining whether a written sentence is scientific, to which section it belongs, and suggest paraphrasings to improve the sentence. Firstly, we propose a regression model trained on a corpus of scientific sentences extracted from peer-reviewed scientific papers and non-scientific text to assign a score that indicates the scientificness of a sentence. We investiga…
▽ More
We support scientific writers in determining whether a written sentence is scientific, to which section it belongs, and suggest paraphrasings to improve the sentence. Firstly, we propose a regression model trained on a corpus of scientific sentences extracted from peer-reviewed scientific papers and non-scientific text to assign a score that indicates the scientificness of a sentence. We investigate the effect of equations and citations on this score to test the model for potential biases. Secondly, we create a map** of section titles to a standard paper layout in AI and machine learning to classify a sentence to its most likely section. We study the impact of context, i.e., surrounding sentences, on the section classification performance. Finally, we propose a paraphraser, which suggests an alternative for a given sentence that includes word substitutions, additions to the sentence, and structural changes to improve the writing style. We train various large language models on sentences extracted from arXiv papers that were peer reviewed and published at A*, A, B, and C ranked conferences. On the scientificness task, all models achieve an MSE smaller than $2\%$. For the section classification, BERT outperforms WideMLP and SciBERT in most cases. We demonstrate that using context enhances the classification of a sentence, achieving up to a $90\%$ F1-score. Although the paraphrasing models make comparatively few alterations, they produce output sentences close to the gold standard. Large fine-tuned models such as T5 Large perform best in experiments considering various measures of difference between input sentence and gold standard. Code is provided under https://github.com/JustinMuecke/SciSen.
△ Less
Submitted 21 June, 2023; v1 submitted 19 June, 2023;
originally announced June 2023.
-
Correlates of Programmer Efficacy and Their Link to Experience: A Combined EEG and Eye-Tracking Study
Authors:
Norman Peitek,
Annabelle Bergum,
Maurice Rekrut,
Jonas Mucke,
Matthias Nadig,
Chris Parnin,
Janet Siegmund,
Sven Apel
Abstract:
Background: Despite similar education and background, programmers can exhibit vast differences in efficacy. While research has identified some potential factors, such as programming experience and domain knowledge, the effect of these factors on programmers' efficacy is not well understood.
Aims: We aim at unraveling the relationship between efficacy (speed and correctness) and measures of progr…
▽ More
Background: Despite similar education and background, programmers can exhibit vast differences in efficacy. While research has identified some potential factors, such as programming experience and domain knowledge, the effect of these factors on programmers' efficacy is not well understood.
Aims: We aim at unraveling the relationship between efficacy (speed and correctness) and measures of programming experience. We further investigate the correlates of programmer efficacy in terms of reading behavior and cognitive load.
Method: For this purpose, we conducted a controlled experiment with 37~participants using electroencephalography (EEG) and eye tracking. We asked participants to comprehend up to 32 Java source-code snippets and observed their eye gaze and neural correlates of cognitive load. We analyzed the correlation of participants' efficacy with popular programming experience measures.
Results: We found that programmers with high efficacy read source code more targeted and with lower cognitive load. Commonly used experience levels do not predict programmer efficacy well, but self-estimation and indicators of learning eagerness are fairly accurate.
Implications: The identified correlates of programmer efficacy can be used for future research and practice (e.g., hiring). Future research should also consider efficacy as a group sampling method, rather than using simple experience measures.
△ Less
Submitted 13 March, 2023;
originally announced March 2023.
-
On the Interplay between TLS Certificates and QUIC Performance
Authors:
Marcin Nawrocki,
Pouyan Fotouhi Tehrani,
Raphael Hiesgen,
Jonas Mücke,
Thomas C. Schmidt,
Matthias Wählisch
Abstract:
In this paper, we revisit the performance of the QUIC connection setup and relate the design choices for fast and secure connections to common Web deployments. We analyze over 1M Web domains with 272k QUIC-enabled services and find two worrying results. First, current practices of creating, providing, and fetching Web certificates undermine reduced round trip times during the connection setup sinc…
▽ More
In this paper, we revisit the performance of the QUIC connection setup and relate the design choices for fast and secure connections to common Web deployments. We analyze over 1M Web domains with 272k QUIC-enabled services and find two worrying results. First, current practices of creating, providing, and fetching Web certificates undermine reduced round trip times during the connection setup since sizes of 35% of server certificates exceed the amplification limit. Second, non-standard server implementations lead to larger amplification factors than QUIC permits, which increase even further in IP spoofing scenarios. We present guidance for all involved stakeholders to improve the situation.
△ Less
Submitted 4 November, 2022;
originally announced November 2022.
-
Waiting for QUIC: On the Opportunities of Passive Measurements to Understand QUIC Deployments
Authors:
Jonas Mücke,
Marcin Nawrocki,
Raphael Hiesgen,
Patrick Sattler,
Johannes Zirngibl,
Georg Carle,
Thomas C. Schmidt,
Matthias Wählisch
Abstract:
In this paper, we study the potentials of passive measurements to gain advanced knowledge about QUIC deployments. By analyzing one month backscatter traffic of the /9 CAIDA network telescope, we are able to make the following observations. First, we can identify different off-net deployments of hypergiants, using packet features such as QUIC source connection IDs (SCID), packet coalescence, and pa…
▽ More
In this paper, we study the potentials of passive measurements to gain advanced knowledge about QUIC deployments. By analyzing one month backscatter traffic of the /9 CAIDA network telescope, we are able to make the following observations. First, we can identify different off-net deployments of hypergiants, using packet features such as QUIC source connection IDs (SCID), packet coalescence, and packet lengths. Second, Facebook and Google configure significantly different retransmission timeouts and maximum number of retransmissions. Third, SCIDs allow further insights into load balancer deployments such as number of servers per load balancer. We bolster our results by active measurements.
△ Less
Submitted 2 September, 2022;
originally announced September 2022.
