Substantia Nigra and Reinforcement Learning

Robb Rutledge and I have a piece just out in the Journal of Neuroscience. It’s a Journal Club article – a ‘short, critical, review’ of a recent paper published by Ramayya et al in Michael Kahana’s group at University of Pennsylvania. The paper concerns a simple learning experiment performed in a group of people who are very interesting to neuroscientists: patients with severe Parkinson’s disease.

Patients for whom drug treatment has proved ineffective regularly undergo a remarkable treatment called Deep Brain Stimulation (DBS). This technique was pioneered by surgerons such as Tipu Aziz, in research that relied heavily upon non-human primate experiemntation. In my view, this offers some of the strongest evidence of how indispensible animal research is to the furtherance of science and medicine; there’s no way we could have developed the technique, which has profoundly impacted hundreds (thousands?) of Parkinson’s patients, without animal research. DBS involves implanting an electrode deep into the brain, typically into an area named the SubThalamic Nucleus. Switching the electrode on probably inhibits activity in the immediate surroundinga, but the effects are very widespread, with lots of activity changes in parts of the brain concerned with movement (which you can read about in a nice paper by my colleague Joshua Kahan).

Electrodes implanted into the brains of people with Parkinson’s Disease can alleviate motor symptoms

In order to implant the electrode in the right spot (not easy; the target is about 5 x 5 x 4mm), patients are awake during surgery. This allows surgeons to use microstimulation – small bursts of stimulation – to test the function of the brain area that the electrode is currently in. Ramayya et al this performed stimulation whilst the electrode was in an area of the brain called the substantia nigra, which contains lots of neurons containing a famous chemical called dopamine; it’s actually these neurons that die during Parkinson’s Disease. Dopamine neurons are believed to be very important for learning and decision-making, and are involved in processing feedback from decisions; they fire when you’ve made a choice worth repeating. Ramayya et al showed that stimulating these neurons had a quite specific effect: it made people more likely to repeat actions but not choices.

They showed this in a task in which people were choosing between two pictures (say an apple and an orange) which were presented on the left or right of the screen (image below). After each choice they were told whether they had won money on that trial. Crucially, whether they won or lost depended solely on the picture they chose; whether that picture was on the right or left was actually irrelevant. Nevertheless, when patients were stimulated in the substantia nigra, they became much more likely to repeat actions – choosing left or right- that had been associated with stimulation. This suggests that separate dopaminergic systems exist in the brains to reinforce actions and stimulus-choices (like those between apples and oranges). This actually fits with what we already know, because there’s another bunch of dopamine neurons in a place called the Ventral Tegmental Area, and where the neurons go from there (the ‘projections’) is more suggestive of a role in reinforcing stimulus-choices.

(A) Electrodes were implanted into a part of the brain called the substantia nigra. (B) Participants made choices between two cues, which were selected with the left or right button. They then saw whether their choice had been a good or bad one (positive or negative feedback). In reality, it was important which cue they picked, but not which side (left or right) it was on. However, when the investigators stimulated a part of the brain called the substantia nigra, they found that participants paid more attention to which side had been producing more rewards recently.
(A) Electrodes were implanted into a part of the brain called the substantia nigra. (B) Participants made choices between two cues, which were selected with the left or right button. They then saw whether their choice had been a good or bad one (positive or negative feedback). In reality, it was important which cue they picked, but not which side (left or right) it was on. However, when the investigators stimulated a part of the brain called the substantia nigra, they found that participants paid more attention to which side had been producing more rewards recently.

Our paper summarises this result, explores some ideas about what it might mean about our understanding of this part of the brain, and contains some alternative explanations of the observed effects in terms of a computational model of learning. That last bit is a little dense, but most of it should be pretty easy to swallow. You can find the paper here, or if you don’t have institutional access, the pdf is here.