[CLOSED] Recording STDP weights (SpiNNaker)


#1

Good afternoon,

I am trying to record the learning process to a file, I want to be able to store the weights of STDP synapses. Ideally I want to access them at different times, but accessing once after the simulation is ended is fine as well. I have tried following solution given here. However, I keep on getting this error:

Connections are only set after run has been called, even if you are trying to see the data before changes have been made. Try examining the [‘weight’] after the call to run.

I assume it is because weights are dynamic and “first I have to learn them, then record” or the TF is loaded before the projection is accessible? I get the following error when I try to use a TF which accesses globally defined projection. How do I solve this? Am I supposed to use virtual coach, SMACH? Thank you for your help.

Best regards
Gabi


#2

Dear Gabi,

The spinnaker board runs continuously in NRP, so it shouldn’t be a problem to access projections while it runs. Though, the board needs like 15 seconds to initialize and if TFs run meanwhile, they will raise that error. You might want to delay the call on projection until the board is initialized (just test time in the TF for a try).

Best regards
Axel


#3

Thank you for a response. I have tried delaying the call (even up to 5minutes) and it did not help, I was getting the same error. I was using a simple if, hopefully, that is what you meant as well. Do you have a clue what else I could try?


#4

Dear Gabi,

The only reason we see for this error to be raised, is that the simulation or the spinnaker board would not be running. Is your robot moving at all? Maybe you could send us the console logs (the one where you gave cle-start).

Axel


#5

Would it be okay if I copy paste the console logs here?

Here is an excerpt:

[hbp_nrp_cle.brainsim.pynn_spiNNaker.devices.__SpikeInjector][INFO] 2021-02-11 11:08:48,656: Spike injector ready to inject
[spinn_front_end_common.utilities.database.database_connection][INFO] 2021-02-11 11:08:48,656: Waiting for message to indicate that the simulation has stopped or paused
[hbp_nrp_cle.brainsim.pynn_spiNNaker.devices.__SpikeInjector][INFO] 2021-02-11 11:08:48,657: Spike injector ready to inject
[hbp_nrp_cleserver.server.ROSCLEServer][ERROR] 2021-02-11 11:09:20,642: Error in Transfer Function (Runtime): Connections are only set after run has been called, even if you are trying to see the data before changes have been made. Try examining the [‘weight’] after the call to run.
Traceback (most recent call last):
File “/home/spinnaker/Documents/NRP/CLE/hbp_nrp_cle/hbp_nrp_cle/tf_framework/_TransferFunction.py”, line 284, in run
return self._func(*self._params)
File “”, line 8, in csv_place_cell_monitor
File “/home/spinnaker/Documents/NRP/ExperimentControl/hbp_nrp_excontrol/hbp_nrp_excontrol/logs/_ClientLogger.py”, line 61, in info
self.log(ClientLoggerMessage.INFORMATION, *objs, **kwargs)
File “/home/spinnaker/Documents/NRP/ExperimentControl/hbp_nrp_excontrol/hbp_nrp_excontrol/logs/_ClientLogger.py”, line 52, in log
clm = ClientLoggerMessage(loglevel, sep.join(map(str, objs)), duration)
File “/home/spinnaker/.opt/platform_venv/local/lib/python2.7/site-packages/spynnaker/pyNN/models/neuron/connection_holder.py”, line 229, in str
data = self._get_data_items()
File “/home/spinnaker/.opt/platform_venv/local/lib/python2.7/site-packages/spynnaker/pyNN/models/neuron/connection_holder.py”, line 128, in _get_data_items
self.__data_items_to_return))
Exception: Connections are only set after run has been called, even if you are trying to see the data before changes have been made. Try examining the [‘weight’] after the call to run.
2021-02-11 11:10:48,992 [Thread-5 ] [hbp_nrp_back] [INFO] Start cleanup


#6

Dear Gabi,

Let’s have a look at the complete logs. Please open a fresh terminal, copy the whole of them from cle-start down to the last error, paste them in a text file, zip it and put it online, for example at https://www.swisstransfer.com.

Best
Axel


#7

Apologies for not the full logs. You can access them now here.


#8

Please update me if this is the right log or if any other ROS logs are necessary.


#9

Ok. There seems to be a synchronization issue between the TF execution and Spinnaker. We would need to replicate the issue and dig into it. It’s probably a bug. I reported it as
https://hbpneurorobotics.atlassian.net/browse/NUIT-249

Please zip your experiment folder and attach it to the NUIT-249 ticket, a developer will look into it.

Best regards
Axel


#10

Thank you for the update, I have uploaded the simplified experiment to the Jira ticket.


#11

Just for tracking purposes: this issue is blocked until this fix in sPyNNaker is provided:


#12

As of May 8th 2021, the Spynnaker issue is still open.


#13

As of Sept 17th, the Spynnaker issue is still open.


#14

Spynnaker issue still open (17th Feb 22)


#15

Spynnaker issue still open. Closing this thread.