-
CTBench: A Comprehensive Benchmark for Evaluating Language Model Capabilities in Clinical Trial Design
Authors:
Nafis Neehal,
Bowen Wang,
Shayom Debopadhaya,
Soham Dan,
Keerthiram Murugesan,
Vibha Anand,
Kristin P. Bennett
Abstract:
CTBench is introduced as a benchmark to assess language models (LMs) in aiding clinical study design. Given study-specific metadata, CTBench evaluates AI models' ability to determine the baseline features of a clinical trial (CT), which include demographic and relevant features collected at the trial's start from all participants. These baseline features, typically presented in CT publications (of…
▽ More
CTBench is introduced as a benchmark to assess language models (LMs) in aiding clinical study design. Given study-specific metadata, CTBench evaluates AI models' ability to determine the baseline features of a clinical trial (CT), which include demographic and relevant features collected at the trial's start from all participants. These baseline features, typically presented in CT publications (often as Table 1), are crucial for characterizing study cohorts and validating results. Baseline features, including confounders and covariates, are also necessary for accurate treatment effect estimation in studies involving observational data. CTBench consists of two datasets: "CT-Repo," containing baseline features from 1,690 clinical trials sourced from clinicaltrials.gov, and "CT-Pub," a subset of 100 trials with more comprehensive baseline features gathered from relevant publications. Two LM-based evaluation methods are developed to compare the actual baseline feature lists against LM-generated responses. "ListMatch-LM" and "ListMatch-BERT" use GPT-4o and BERT scores (at various thresholds), respectively, for evaluation. To establish baseline results, advanced prompt engineering techniques using LLaMa3-70B-Instruct and GPT-4o in zero-shot and three-shot learning settings are applied to generate potential baseline features. The performance of GPT-4o as an evaluator is validated through human-in-the-loop evaluations on the CT-Pub dataset, where clinical experts confirm matches between actual and LM-generated features. The results highlight a promising direction with significant potential for improvement, positioning CTBench as a useful tool for advancing research on AI in CT design and potentially enhancing the efficacy and robustness of CTs.
△ Less
Submitted 25 June, 2024;
originally announced June 2024.
-
Incept-N: A Convolutional Neural Network based Classification Approach for Predicting Nationality from Facial Features
Authors:
Masum Shah Junayed,
Afsana Ahsan Jeny,
Nafis Neehal
Abstract:
The nationality of a human being is a well-known identifying characteristic used for every major authentication purpose in every country. Albeit advances in the application of Artificial Intelligence and Computer Vision in different aspects, its contribution to this specific security procedure is yet to be cultivated. With a goal to successfully applying computer vision techniques to predict the n…
▽ More
The nationality of a human being is a well-known identifying characteristic used for every major authentication purpose in every country. Albeit advances in the application of Artificial Intelligence and Computer Vision in different aspects, its contribution to this specific security procedure is yet to be cultivated. With a goal to successfully applying computer vision techniques to predict the nationality of a person based on his facial features, we have proposed this novel method and have achieved an average of 93.6% accuracy with very low misclassification rate.
△ Less
Submitted 18 May, 2018;
originally announced May 2018.
-
Runtime Optimization of Identification Event in ECG Based Biometric Authentication
Authors:
Nafis Neehal,
Dewan Ziaul Karim,
Sejuti Banik,
Tasfia Anika
Abstract:
Biometric Authentication has become a very popular method for different state-of-the-art security architectures. Albeit the ubiquitous acceptance and constant development of trivial biometric authentication methods such as fingerprint, palm-print, retinal scan etc., the possibility of producing a highly competitive performance from somewhat less-popular methods still remains. Electrocardiogram (EC…
▽ More
Biometric Authentication has become a very popular method for different state-of-the-art security architectures. Albeit the ubiquitous acceptance and constant development of trivial biometric authentication methods such as fingerprint, palm-print, retinal scan etc., the possibility of producing a highly competitive performance from somewhat less-popular methods still remains. Electrocardiogram (ECG) based biometric authentication is such a method, which, despite its limited appearance in earlier research works, are currently being observed as equivalently high-performing as other trivial popular methods. In this paper, we have proposed a model to optimize the runtime of identification event in ECG based biometric authentication and we have achieved a maximum of 79.26% time reduction with 100% accuracy.
△ Less
Submitted 15 May, 2018;
originally announced May 2018.
-
Crick-net: A Convolutional Neural Network based Classification Approach for Detecting Waist High No Balls in Cricket
Authors:
Md. Harun-Ur-Rashid,
Shekina Khatun,
Mehe Zabin Trisha,
Nafis Neehal,
Md. Zahid Hasan
Abstract:
Cricket is undoubtedly one of the most popular games in this modern era. As human beings are prone to error, there remains a constant need for automated analysis and decision making of different events in this game. Simultaneously, with advent and advances in Artificial Intelligence and Computer Vision, application of these two in different domains has become an emerging trend. Applying several co…
▽ More
Cricket is undoubtedly one of the most popular games in this modern era. As human beings are prone to error, there remains a constant need for automated analysis and decision making of different events in this game. Simultaneously, with advent and advances in Artificial Intelligence and Computer Vision, application of these two in different domains has become an emerging trend. Applying several computer vision techniques in analyzing different Cricket events and automatically coming into decisions has become popular in recent days. In this paper, we have deployed a CNN based classification method with Inception V3 in order to automatically detect and differentiate waist high no balls with fair balls. Our approach achieves an overall average accuracy of 88% with a fairly low cross-entropy value.
△ Less
Submitted 30 January, 2021; v1 submitted 15 May, 2018;
originally announced May 2018.
-
InceptB: A CNN Based Classification Approach for Recognizing Traditional Bengali Games
Authors:
Mohammad Shakirul Islam,
Ferdouse Ahmed Foysal,
Nafis Neehal,
Enamul Karim,
Syed Akhter Hossain
Abstract:
Sports activities are an integral part of our day to day life. Introducing autonomous decision making and predictive models to recognize and analyze different sports events and activities has become an emerging trend in computer vision arena. Albeit the advances and vivid applications of artificial intelligence and computer vision in recognizing different popular western games, there remains a ver…
▽ More
Sports activities are an integral part of our day to day life. Introducing autonomous decision making and predictive models to recognize and analyze different sports events and activities has become an emerging trend in computer vision arena. Albeit the advances and vivid applications of artificial intelligence and computer vision in recognizing different popular western games, there remains a very minimal amount of efforts in the application of computer vision in recognizing traditional Bangladeshi games. We, in this paper, have described a novel Deep Learning based approach for recognizing traditional Bengali games. We have retrained the final layer of the renowned Inception V3 architecture developed by Google for our classification approach. Our approach shows promising results with an average accuracy of 80% approximately in correctly recognizing among 5 traditional Bangladeshi sports events.
△ Less
Submitted 16 September, 2018; v1 submitted 3 May, 2018;
originally announced May 2018.
