-
Bayesian Inference for Evidence Accumulation Models with Regressors
Authors:
Viet Hung Dao,
David Gunawan,
Robert Kohn,
Minh-Ngoc Tran,
Guy E. Hawkins,
Scott D. Brown
Abstract:
Evidence accumulation models (EAMs) are an important class of cognitive models used to analyze both response time and response choice data recorded from decision-making tasks. Developments in estimation procedures have helped EAMs become important both in basic scientific applications and solution-focussed applied work. Hierarchical Bayesian estimation frameworks for the linear ballistic accumulat…
▽ More
Evidence accumulation models (EAMs) are an important class of cognitive models used to analyze both response time and response choice data recorded from decision-making tasks. Developments in estimation procedures have helped EAMs become important both in basic scientific applications and solution-focussed applied work. Hierarchical Bayesian estimation frameworks for the linear ballistic accumulator model (LBA) and the diffusion decision model (DDM) have been widely used, but still suffer from some key limitations, particularly for large sample sizes, for models with many parameters, and when linking decision-relevant covariates to model parameters. We extend upon previous work with methods for estimating the LBA and DDM in hierarchical Bayesian frameworks that include random effects which are correlated between people, and include regression-model links between decision-relevant covariates and model parameters. Our methods work equally well in cases where the covariates are measured once per person (e.g., personality traits or psychological tests) or once per decision (e.g., neural or physiological data). We provide methods for exact Bayesian inference, using particle-based MCMC, and also approximate methods based on variational Bayesian (VB) inference. The VB methods are sufficiently fast and efficient that they can address large-scale estimation problems, such as with very large data sets. We evaluate the performance of these methods in applications to data from three existing experiments. Detailed algorithmic implementations and code are freely available for all methods.
△ Less
Submitted 31 May, 2023; v1 submitted 20 February, 2023;
originally announced February 2023.
-
Efficient Selection Between Hierarchical Cognitive Models: Cross-validation With Variational Bayes
Authors:
Viet-Hung Dao,
David Gunawan,
Minh-Ngoc Tran,
Robert Kohn,
Guy E. Hawkins,
Scott D. Brown
Abstract:
Model comparison is the cornerstone of theoretical progress in psychological research. Common practice overwhelmingly relies on tools that evaluate competing models by balancing in-sample descriptive adequacy against model flexibility, with modern approaches advocating the use of marginal likelihood for hierarchical cognitive models. Cross-validation is another popular approach but its implementat…
▽ More
Model comparison is the cornerstone of theoretical progress in psychological research. Common practice overwhelmingly relies on tools that evaluate competing models by balancing in-sample descriptive adequacy against model flexibility, with modern approaches advocating the use of marginal likelihood for hierarchical cognitive models. Cross-validation is another popular approach but its implementation has remained out of reach for cognitive models evaluated in a Bayesian hierarchical framework, with the major hurdle being prohibitive computational cost. To address this issue, we develop novel algorithms that make variational Bayes (VB) inference for hierarchical models feasible and computationally efficient for complex cognitive models of substantive theoretical interest. It is well known that VB produces good estimates of the first moments of the parameters which gives good predictive densities estimates. We thus develop a novel VB algorithm with Bayesian prediction as a tool to perform model comparison by cross-validation, which we refer to as CVVB. In particular, the CVVB can be used as a model screening device that quickly identifies bad models. We demonstrate the utility of CVVB by revisiting a classic question in decision making research: what latent components of processing drive the ubiquitous speed-accuracy tradeoff? We demonstrate that CVVB strongly agrees with model comparison via marginal likelihood yet achieves the outcome in much less time. Our approach brings cross-validation within reach of theoretically important psychological models, and makes it feasible to compare much larger families of hierarchically specified cognitive models than has previously been possible.
△ Less
Submitted 8 October, 2021; v1 submitted 12 February, 2021;
originally announced February 2021.
-
Identifying relationships between cognitive processes across tasks, contexts, and time
Authors:
Laura Wall,
David Gunawan,
Scott D. Brown,
Minh-Ngoc Tran,
Robert Kohn,
Guy E. Hawkins
Abstract:
It is commonly assumed that a specific testing occasion (task, design, procedure, etc.) provides insights that generalise beyond that occasion. This assumption is infrequently carefully tested in data. We develop a statistically principled method to directly estimate the correlation between latent components of cognitive processing across tasks, contexts, and time. This method simultaneously estim…
▽ More
It is commonly assumed that a specific testing occasion (task, design, procedure, etc.) provides insights that generalise beyond that occasion. This assumption is infrequently carefully tested in data. We develop a statistically principled method to directly estimate the correlation between latent components of cognitive processing across tasks, contexts, and time. This method simultaneously estimates individual-participant parameters of a cognitive model at each testing occasion, group-level parameters representing across-participant parameter averages and variances, and across-task correlations. The approach provides a natural way to "borrow" strength across testing occasions, which can increase the precision of parameter estimates across all testing occasions. Two example applications demonstrate that the method is practical in standard designs. The examples, and a simulation study, also provide evidence about the reliability and validity of parameter estimates from the linear ballistic accumulator model. We conclude by highlighting the potential of the parameter-correlation method to provide an "assumption-light" tool for estimating the relatedness of cognitive processes across tasks, contexts, and time.
△ Less
Submitted 26 March, 2020; v1 submitted 16 October, 2019;
originally announced October 2019.
-
Time-evolving psychological processes over repeated decisions
Authors:
David Gunawan,
Guy E. Hawkins,
Robert Kohn,
Minh-Ngoc Tran,
Scott D. Brown
Abstract:
Many psychological experiments have subjects repeat a task to gain the statistical precision required to test quantitative theories of psychological performance. In such experiments, time-on-task can have sizable effects on performance, changing the psychological processes under investigation. Most research has either ignored these changes, treating the underlying process as static, or sacrificed…
▽ More
Many psychological experiments have subjects repeat a task to gain the statistical precision required to test quantitative theories of psychological performance. In such experiments, time-on-task can have sizable effects on performance, changing the psychological processes under investigation. Most research has either ignored these changes, treating the underlying process as static, or sacrificed some psychological content of the models for statistical simplicity. We use particle Markov chain Monte-Carlo methods to study psychologically plausible time-varying changes in model parameters. Using data from three highly-cited experiments we find strong evidence in favor of a hidden Markov switching process as an explanation of time-varying effects. This embodies the psychological assumption of "regime switching", with subjects alternating between different cognitive states representing different modes of decision-making. The switching model explains key long- and short-term dynamic effects in the data. The central idea of our approach can be applied quite generally to quantitative psychological theories, beyond the models and data sets that we investigate.
△ Less
Submitted 3 November, 2021; v1 submitted 26 June, 2019;
originally announced June 2019.
-
Robustly estimating the marginal likelihood for cognitive models via importance sampling
Authors:
Minh-Ngoc Tran,
Marcel Scharth,
David Gunawan,
Robert Kohn,
Scott D. Brown,
Guy E. Hawkins
Abstract:
Recent advances in Markov chain Monte Carlo (MCMC) extend the scope of Bayesian inference to models for which the likelihood function is intractable. Although these developments allow us to estimate model parameters, other basic problems such as estimating the marginal likelihood, a fundamental tool in Bayesian model selection, remain challenging. This is an important scientific limitation because…
▽ More
Recent advances in Markov chain Monte Carlo (MCMC) extend the scope of Bayesian inference to models for which the likelihood function is intractable. Although these developments allow us to estimate model parameters, other basic problems such as estimating the marginal likelihood, a fundamental tool in Bayesian model selection, remain challenging. This is an important scientific limitation because testing psychological hypotheses with hierarchical models has proven difficult with current model selection methods. We propose an efficient method for estimating the marginal likelihood for models where the likelihood is intractable, but can be estimated unbiasedly. It is based on first running a sampling method such as MCMC to obtain samples for the model parameters, and then using these samples to construct the proposal density in an importance sampling (IS) framework with an unbiased estimate of the likelihood. Our method has several attractive properties: it generates an unbiased estimate of the marginal likelihood, it is robust to the quality and target of the sampling method used to form the IS proposals, and it is computationally cheap to estimate the variance of the marginal likelihood estimator. We also obtain the convergence properties of the method and provide guidelines on maximizing computational efficiency. The method is illustrated in two challenging cases involving hierarchical models: identifying the form of individual differences in an applied choice scenario, and evaluating the best parameterization of a cognitive model in a speeded decision making context. Freely available code to implement the methods is provided. Extensions to posterior moment estimation and parallelization are also discussed.
△ Less
Submitted 11 December, 2019; v1 submitted 14 June, 2019;
originally announced June 2019.
-
Uncharted Forest a Technique for Exploratory Data Analysis
Authors:
Casey Kneale,
Steven D. Brown
Abstract:
Exploratory data analysis is crucial for develo** and understanding classification models from high-dimensional datasets. We explore the utility of a new unsupervised tree ensemble called uncharted forest for visualizing class associations, sample-sample associations, class heterogeneity, and uninformative classes for provenance studies. The uncharted forest algorithm can be used to partition da…
▽ More
Exploratory data analysis is crucial for develo** and understanding classification models from high-dimensional datasets. We explore the utility of a new unsupervised tree ensemble called uncharted forest for visualizing class associations, sample-sample associations, class heterogeneity, and uninformative classes for provenance studies. The uncharted forest algorithm can be used to partition data using random selections of variables and metrics based on statistical spread. After each tree is grown, a tally of the samples that arrive at every terminal node is maintained. Those tallies are stored in single sample association matrix and a likelihood measure for each sample being partitioned with one another can be made. That matrix may be readily viewed as a heat map, and the probabilities can be quantified via new metrics that account for class or cluster membership. We display the advantages and limitations of using this technique by applying it to two classification datasets and three provenance study datasets. Two of the metrics presented in this paper are also compared with widely used metrics from two algorithms that have variance-based clustering mechanisms.
△ Less
Submitted 30 June, 2018; v1 submitted 11 February, 2018;
originally announced February 2018.
-
Band Target Entropy Minimization and Target Partial Least Squares for Spectral Recovery and Calibration
Authors:
Casey Kneale,
Steven D. Brown
Abstract:
The resolution and calibration of pure spectra of minority components in measurements of chemical mixtures without prior knowledge of the mixture is a challenging problem. In this work, a combination of band target entropy minimization (BTEM) and target partial least squares (T-PLS) was used to obtain estimates for single pure component spectra and to calibrate those estimates in a true, one-at-a-…
▽ More
The resolution and calibration of pure spectra of minority components in measurements of chemical mixtures without prior knowledge of the mixture is a challenging problem. In this work, a combination of band target entropy minimization (BTEM) and target partial least squares (T-PLS) was used to obtain estimates for single pure component spectra and to calibrate those estimates in a true, one-at-a-time fashion. This approach allows for minor components to be targeted and their relative amounts estimated in the presence of other varying components in spectral data. The use of T-PLS estimation is an improvement to the BTEM method because it overcomes the need to identify all of the pure components prior to estimation. Estimated amounts from this combination were found to be similar to those obtained from a standard method, multivariate curve resolution-alternating least squares (MCR-ALS), on a simple, three component mixture dataset. Studies from two experimental datasets demonstrate where the combination of BTEM and T-PLS could model the pure component spectra and obtain concentration profiles of minor components but MCR-ALS could not.
△ Less
Submitted 27 March, 2018; v1 submitted 11 February, 2018;
originally announced February 2018.
-
Small Moving Window Calibration Models for Soft Sensing Processes with Limited History
Authors:
Casey Kneale,
Steven D. Brown
Abstract:
Five simple soft sensor methodologies with two update conditions were compared on two experimentally-obtained datasets and one simulated dataset. The soft sensors investigated were moving window partial least squares regression (and a recursive variant), moving window random forest regression, the mean moving window of $y$, and a novel random forest partial least squares regression ensemble (RF-PL…
▽ More
Five simple soft sensor methodologies with two update conditions were compared on two experimentally-obtained datasets and one simulated dataset. The soft sensors investigated were moving window partial least squares regression (and a recursive variant), moving window random forest regression, the mean moving window of $y$, and a novel random forest partial least squares regression ensemble (RF-PLS), all of which can be used with small sample sizes so that they can be rapidly placed online. It was found that, on two of the datasets studied, small window sizes led to the lowest prediction errors for all of the moving window methods studied. On the majority of datasets studied, the RF-PLS calibration method offered the lowest one-step-ahead prediction errors compared to those of the other methods, and it demonstrated greater predictive stability at larger time delays than moving window PLS alone. It was found that both the random forest and RF-PLS methods most adequately modeled the datasets that did not feature purely monotonic increases in property values, but that both methods performed more poorly than moving window PLS models on one dataset with purely monotonic property values. Other data dependent findings are presented and discussed.
△ Less
Submitted 13 March, 2018; v1 submitted 31 October, 2017;
originally announced October 2017.
