B03 – Gaze dynamics and reinforcement learning during Pavlovian conditioning

During the first two funding periods, we studied Bayesian approaches to data assimilation for dynamical cognitive models of gaze control, including model comparisons and efficient algorithms. Based on these advances, our process-based, dynamical models could reproduce and explain inter-individual differences in eye-movement control between human participants. In the third funding period, we will apply our models of eye-movement control to Pavlovian conditioning, where model parameters change over time due to reinforcement learning. While the mathematical theory of reinforcement learning contributed to our understanding of behaviours in animals and humans, our new approach will be an innovative combination of reinforcement learning with explicit, process-based models of gaze control. Thus, our work will provide an understanding of how reinforcement learning interacts with dynamical cognitive processes of gaze control such as attention allocation or working memory, and how these processes are altered in psychological disorders.
In Pavlovian conditioning, a conditioned stimulus (CS/cue, e.g., an initially neutral picture) precedes presentation of an unconditioned stimulus (US/outcome; e.g., monetary win) such that individuals learn to predict the outcome from the cue. After learning, participants respond to the cue, either focusing attention on the predictive cue (sign-trackers) or on the outcome-location (goal-trackers). We and others have demonstrated, that the long-dominant theory that dopaminergic reward prediction errors underlie Pavlovian conditioning, surprisingly is valid only for sign-trackers, while goal-trackers seem to rely on a different mechanism. Specifically, this research provided evidence that sign-trackers use model-free reinforcement learning algorithms that rely on past encounters of wins and losses via reward prediction errors and that goal-trackers use model-based reinforcement learning algorithms that involve cognitive anticipations of potential future states.
In our project, we focus on mechanisms of gaze-control in human sign-/goal-trackers. To define individuals as sign-/goal-trackers, we can assess whether a participant looked more at a visually presented cue or at the unconditioned stimulus location. Previous work on human sign-/goal-tracking has treated gaze as a summary statistic (mean looking times at relevant stimuli). However, gaze behaviour is highly dynamic and evolves sequentially, with 3 to 4 saccades (fast eye  movements) occurring per second. Using process-based modelling, we will improve our understanding of control of gaze in sign- versus goal-trackers, which is essential also for clinical applications. Our resulting integrated model of gaze control and Pavlovian conditioning will predict dynamical behaviour on two time scales, (i) eye movements within trials (millisecond scale) and (ii) changes due to learning across trials (over minutes). Our mathematical models of gaze control from the previous funding periods, will be the starting point for the work in the third funding phase.
First, we will develop a computational model of individual differences in reinforcement learning across trials. Rodent data show that sign-tracker rats approach the conditioned stimulus even if this behaviour is punished via omission of the reward (a phenomenon termed negative auto-maintenance). We plan to model individual differences in such effects using Pavlovian forms of reinforcement learning based on evolutionary biases, where appetitive stimuli drive approach towards them although this behaviour is never reinforced. Second, we will perform empirical tests of key model predictions in human participants, by conducting two experimental studies on sign-/goal-tracking, involving an investigation of negative auto-maintenance and a manipulation of working memory load. Third, to explain how cues guide gaze responses, we plan to develop a novel integrated computational model for gaze dynamics during Pavlovian conditioning. We will expand our model of gaze dynamics developed in the previous funding period to capture dynamics of value learning via a reinforcement learning model. Model evaluation will be achieved by data assimilation of the novel integrated model using existing data sets and new experimental data; our approach will include hierarchical Bayesian parameter inference and model comparisons between competing models or model implementations. Resulting individual estimates of model parameters will be correlated with clinical parameters, e.g., in addiction.