How to use stdp_dopamine Synapse correctly


#1

Hallo,
I’m trying to use the stdp_dopamine_synapse but I can’t find out how to use it.
First I saw in the ‘Holodeck hollie Arm Reinforcement-Learning for target reaching’ template that they create an array of 2 Volume_Transmitter, connect the first one to the SynapseModel and then connect them to the dopamine input

...
vt = nest.Create('volume_transmitter', 2)
# Connect Layers
nest.CopyModel('stdp_dopamine_synapse', 'syn1',
               {'Wmax': 500.0,
                'Wmin': 0.0,
                'tau_plus': 50.0,
                'A_minus': 150.0,
                'A_plus': 50.0,
                'b': 0.01,
                'tau_c': 10.0,
                'tau_n': 5.0,
                'vt': vt[0]})

nest.Connect([map(int, dm_layer.all_cells)[1]], vt)
syn_con = nest.Connect(map(int, input_layer.all_cells),
                       map(int, output_layer.all_cells),
                       syn_spec={'model': 'syn1', 'weight': 0.01, 'delay': 1.0})

What does the line

nest.Connect([map(int, dm_layer.all_cells)[1]], vt)

do?

The transfer function for giving the reward then connects to

@nrp.MapSpikeSource("dopamine_source_increase", nrp.brain.dopamine[1], nrp.poisson, weight=100.0)
@nrp.MapSpikeSource("dopamine_source_decrease", nrp.brain.dopamine[0], nrp.poisson, weight=100.0)

Is the first array element always the decreasing one?

In the transfer function which sets the movement for the arm there are the lines:

 @nrp.MapSpikeSink("motor_contract", nrp.brain.actors[0], nrp.spike_recorder)
@nrp.MapSpikeSink("motor_extend", nrp.brain.actors[1], nrp.spike_recorder)
@nrp.MapSpikeSource("teaching_motor_contract", nrp.brain.actors[0], nrp.poisson, weight=100.0)
@nrp.MapSpikeSource("teaching_motor_extend", nrp.brain.actors[1], nrp.poisson, weight=100.0)
... 
if motor_extend.spiked and not motor_contract.spiked:
            joint_delta = -movement_step
            teaching_motor_extend.rate = teaching_rate
            teaching_motor_contract.rate = 0.0
        elif motor_contract.spiked and not motor_extend.spiked:
            joint_delta = movement_step
            teaching_motor_contract.rate = teaching_rate
            teaching_motor_extend.rate = 0.0

The teaching_rate is in that example always 0.0 but should the rate be set here?

Thanks in advance,
MAbuse


#2

Hi!,
It has been a rather long time since I had my hands on this script, so I’ll try my best :wink:

nest.CopyModel('stdp_dopamine_synapse', 'syn1', {..., 'vt': vt[0]})

This line creates internally a new synapse type (this is something NEST specific) and couples this specific volume-transmitter to it. Details on the NEST implementation of stpd_dopamine can be found here (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2996144/)

nest.Connect([map(int, dm_layer.all_cells)[1]], vt)

The inner map-function takes the dm_layer.all_cells array and applies the int() function on each of the elements. Actually we’re just passing here one of the neuron-ids (as array) to nest.Connect(…). I have to look at the full script to remember why was that, it doesnt make any sense to me right now.
As far as I recall pyNN stores these cell IDs as some kind of custom type so you have to convert it to proper integer first in order to satisfy the NEST-Python binding (which is strictly typed). The Connect routine itself is just creating synaptic connections based on the syn_spec parameters.

@nrp.MapSpikeSource("dopamine_source_increase", nrp.brain.dopamine[1], nrp.poisson, weight=100.0)
@nrp.MapSpikeSource("dopamine_source_decrease", nrp.brain.dopamine[0], nrp.poisson, weight=100.0)

dm_layer.all_cells[1] should correspond to the increasing dopamine source;

if motor_extend.spiked and not motor_contract.spiked:
            joint_delta = -movement_step
            teaching_motor_extend.rate = teaching_rate
            teaching_motor_contract.rate = 0.0
        elif motor_contract.spiked and not motor_extend.spiked:
            joint_delta = movement_step
            teaching_motor_contract.rate = teaching_rate
            teaching_motor_extend.rate = 0.0

The rate is only for one of both {contract, extend} set to zero while the other gets the teaching_rate (unless both {motor_extend, motor_contract} spiked at a time)

Note that this script (as a transfer-function) gets executed by default every 20 ms

Edit:
A quick overview how that dopine/vt synapses do work (from what I recall): basically they have two (over-the-time-decaying) capacities, one for ‘dopamine’ and one ‘stdp-potential’. In order for the synapse to change the weight both capacitiy-values get multiplied, so they have both to be non-zero. In NEST this is implemented in the special stdp_dopamine-synapse; The volume-transmitter however, can be connected to multiple synapses at once, thus, you have to create manually new synapse-type copies and specify the specific transmitter to get input from (which fills up the dopamine capacity).


#3

Hello,
I just found this post. I have the same problem. Like @schulze said in

nest.Connect([map(int, dm_layer.all_cells)[1]], vt) 

only one of the neurons is passed to the connection. but the tf utilizes both of them. Is dm_layer.all_cells[0] connected somewhere else?


#4

My state on this that u only use one dopamine neuron. this regulates the concentration in the synapses.
when you set up the synapses u can set a base dopamine concentration (‘b’) and a parameter how fast the dopamine drops (‘tau_n’).
if the dopamine concentration drops under that concentration it works as a punishment otherwise as reward.
but don’t nail me down on that :sweat_smile: