Showing 1–2 of 2 results for author: Chambers, B

The Utility of Large Language Models and Generative AI for Education Research

Authors: Andrew Katz, Umair Shakir, Ben Chambers

Abstract: The use of natural language processing (NLP) techniques in engineering education can provide valuable insights into the underlying processes involved in generating text. While accessing these insights can be labor-intensive if done manually, recent advances in NLP and large language models have made it a realistic option for individuals. This study explores and evaluates a combination of clusterin… ▽ More The use of natural language processing (NLP) techniques in engineering education can provide valuable insights into the underlying processes involved in generating text. While accessing these insights can be labor-intensive if done manually, recent advances in NLP and large language models have made it a realistic option for individuals. This study explores and evaluates a combination of clustering, summarization, and prompting techniques to analyze over 1,000 student essays in which students discussed their career interests. The specific assignment prompted students to define and explain their career goals as engineers. Using text embedding representations of student responses, we clustered the responses together to identify thematically similar statements from students. The clustered responses were then summarized to quickly identify career interest themes. We also used a set of a priori codes about career satisfaction and sectors to demonstrate an alternative approach to using these generative text models to analyze student writing. The results of this study demonstrate the feasibility and usefulness of NLP techniques in engineering education research. By automating the initial analysis of student essays, researchers and educators can more efficiently and accurately identify key themes and patterns in student writing. The methods presented in this paper have broader applications for engineering education and research purposes beyond analyzing student essays. By explaining these methods to the engineering education community, readers can utilize them in their own contexts. △ Less

Submitted 29 May, 2023; originally announced May 2023.

Comments: 3 figures, 10 tables

Semantic maps and metrics for science Semantic maps and metrics for science using deep transformer encoders

Authors: Brendan Chambers, James Evans

Abstract: The growing deluge of scientific publications demands text analysis tools that can help scientists and policy-makers navigate, forecast and beneficially guide scientific research. Recent advances in natural language understanding driven by deep transformer networks offer new possibilities for map** science. Because the same surface text can take on multiple and sometimes contradictory specialize… ▽ More The growing deluge of scientific publications demands text analysis tools that can help scientists and policy-makers navigate, forecast and beneficially guide scientific research. Recent advances in natural language understanding driven by deep transformer networks offer new possibilities for map** science. Because the same surface text can take on multiple and sometimes contradictory specialized senses across distinct research communities, sensitivity to context is critical for infometric applications. Transformer embedding models such as BERT capture shades of association and connotation that vary across the different linguistic contexts of any particular word or span of text. Here we report a procedure for encoding scientific documents with these tools, measuring their improvement over static word embeddings in a nearest-neighbor retrieval task. We find discriminability of contextual representations is strongly influenced by choice of pooling strategy for summarizing the high-dimensional network activations. Importantly, we note that fundamentals such as domain-matched training data are more important than state-of-the-art NLP tools. Yet state-of-the-art models did offer significant gains. The best approach we investigated combined domain-matched pretraining, sound pooling, and state-of-the-art deep transformer network encoders. Finally, with the goal of leveraging contextual representations from deep encoders, we present a range of measurements for understanding and forecasting research communities in science. △ Less

Submitted 13 April, 2021; originally announced April 2021.