Parallel PyNN simulations in the NRP


#1

Dear NRP developers/users:

I have run some scalability tests in the PyNN brain models in our local installation of the NRP. In particular, I have scaled up the network size in some example codes, such as the Braitenberg example, or other custom codes. So far when we run networks including a few thousands of neurons the simulation is around five times slower than real-time (5 seconds of clock time required to run 1 second of simulation). However, when we run the same network models with PyNN/NEST (out of the NRP), the simulation can be highly speeded up with multithreading or multiple MPI processes.

Similarly, I have tried to parallelize the PyNN simulation (in the NRP) by using “sim.setup(‘threads’: n_threads)” but it seems that the python process uses only 1 thread anyway.

I was wondering if any of the PyNN/NEST parallelizing techniques are supported in the NRP and, if so, how the simulations can enable it.

This topic is critical for our task in the NRP since we need to run long simulations (several thousands of simulated seconds) with medium-scale network models (tens of thousands of neurons).

Best regards,

Juan Carlos.


#2

Dear Juan Carlos,

We are aware of the single-thread limitation and the issue has been thoroughly investigated. There are plans to address it, and the work should start soon. Kenny and Axel will come back to you end of July, if not earlier, with more details.

Best regards,
Luc


#3

Thanks @lguyot. I hope your answer as soon as possible. We will work with single-threads until the multithreading is available.

Best regards,
Juan Carlos.


#4

Hi Juan Carlos,

I’m currently starting work on the multi-thread support, but it would be really beneficial if I had a working copy of the Experiment you are using so I can benchmark against something realistic to make sure it is working properly.

The NRP backend currently overrides the pNN.setup(threads: x) parameter to be equal to 1 due to a non-trivial threading issue in Nest (see https://github.com/nest/nest-simulator/issues/725). I am working on a solution to overcome this limitation with minimal impact and when this is complete the threads parameter will again be usable (with a reasonable limit like #cores - 1 since the NRP backend requires a single core and you don’t really get any benefit from oversubscribing and using #threads > #cores).

I will follow up with a private message with my e-mail address for you to send them to or you can simply attach it in a private message back. I will need your experiment folder from $HBP/Experiments and your brain model/any Gazebo models from $HBP/Models. If there are any special instructions for running your experiment, please also pass them along.

Best,
Kenny


#5

Hi Juan Carlos,

Our development sprint is complete and the fix for the NRP is merged into the main codebase. You won’t have to make any changes on your end, the fix on our side will scale to the number of cores in your CPU so it will be portable to other machines with different setups as well.

Maybe @vonarnim can help you update your system to take advantage of multi-threading (you’ll need to recompile Nest and update all of our repos), let me know if you have any issues moving forward.

Best,
Kenny


#6

in user-scripts, an “./update_nrp update all” should update a build everything. Let us know if it fails for you.


#7

Hi @kennysharma and @vonarnim. Firstly, sorry for the delay. These days I’ve been very busy and gmail hasn’t notified to me of yours messages. I couldn’t test the script yet. I want to test it as soon as possible. I will tell you if everything works fine.

Thanks again and sorry!


#8

Hi @kennysharma and @vonarnim. Firstly, sorry for the delay again. I tried update the NRP last week but it broke and I couldn’t fix it until this week. I’ve tested multithreading and it seems that the times improves. I want to do some experiments with this functionality since I consider that it’s very important. We keep in touch!

Thanks you very much for the support.