Synchronizing messages between transfer functions


#1

I have 3 transfer functions which are sending messages between each other, 1 publishes and 2 subscribes, but sometimes a subscriber receives the same message two iterations in a row.

Is there a method for making sure I always received the message from the current timestep? - waiting until a message arrives with matching timestamp?

I assume the problem is along the lines of TF’s being run asynchronously and my publisher sometimes not being executed before the two subscribers. Is this correct and can it be controlled? TF triggers?

Cheers and thanks in advance


#2

Hello Jakob,

TF triggers do exist. You may find this tutorial useful.

Best regards,
Luc


#3

Hi Luc and thanks for the link,

I had a look at the triggers, but I am not sure if it is the answer to my issue.

My issue is primarily that I some times receive the same message twice, but after some thought it doesn’t make sense anyway that this is caused my bad synchronization.
Point is that my TF is listening to a joint_states message for current robot position. The joint_states message is being published 20 times for each NRP iteration, but for some reason I “receive” the same message in NRP two iterations in a row.

I assume that this has to be related to some buffer or update issue somewhere outside of ROS? Could it be the NRP subscriber having a queue larger than 1? - but then I wouldn’t understand why it works most of the time.

It gives some weird bumps in my graph, which causes reactions elsewhere, so I would much appreciate finding a solution for this, though I don’t know where to look.

Any hints?

Thanks in advance

[EDIT]
Now that i think about it I may be able to throttle the publishing of joint_states to approx 0.02 sec (NRP speed) and then trigger my first TF from that topic. I could then trigger my second TF by the topic that the first one is publishing on.

Does that make sense?


#4

I tried to throttle the joint_state publishing and set its topic as a trigger but then I get an error and NRP crashes.

I did exactly as is shown in the tutorial, as far as I can tell:

@nrp.MapRobotSubscriber("joints_current_sub", Topic('/robot/current_trajectory/joints', sensor_msgs.msg.JointState))

@nrp.Robot2Neuron(triggers=["joints_current_sub"])
def planner(t, ....

The error message looks like this:

2018-09-11 17:52:00,622 [Thread-338  ] [hbp_nrp_cle ] [ERROR]  argument of type 'bool' is not iterable
Traceback (most recent call last):
  File "/home/jakob/Documents/NRP/CLE/hbp_nrp_cle/hbp_nrp_cle/cle/DeterministicClosedLoopEngine.py", line 248, in __loop
    self.run_step(self.timestep)
  File "/home/jakob/Documents/NRP/CLE/hbp_nrp_cle/hbp_nrp_cle/cle/DeterministicClosedLoopEngine.py", line 185, in run_step
    self.bca.run_step(timestep * 1000.0)
  File "/home/jakob/Documents/NRP/CLE/hbp_nrp_cle/hbp_nrp_cle/brainsim/pynn/PyNNControlAdapter.py", line 211, in run_step
    self._sim.run(dt)
  File "/home/jakob/.opt/platform_venv/local/lib/python2.7/site-packages/pyNN/common/control.py", line 110, in run
    return run_until(simulator.state.t + simtime, callbacks)
  File "/home/jakob/.opt/platform_venv/local/lib/python2.7/site-packages/pyNN/common/control.py", line 93, in run_until
    return simulator.state.t
  File "/home/jakob/.opt/platform_venv/local/lib/python2.7/site-packages/pyNN/nest/simulator.py", line 70, in t
    return max(nest.GetKernelStatus('time') - self.dt, 0.0)  # note that we always simulate one time step past the requested time
  File "/home/jakob/.local/lib/python2.7/site-packages/nest/lib/hl_api_helper.py", line 230, in stack_checker_func
    return f(*args, **kwargs)
  File "/home/jakob/.local/lib/python2.7/site-packages/nest/lib/hl_api_simulation.py", line 118, in GetKernelStatus
    d = dict((k, v) for k, v in status_root.items() if k not in status_subnet)
  File "/home/jakob/.local/lib/python2.7/site-packages/nest/lib/hl_api_simulation.py", line 118, in <genexpr>
    d = dict((k, v) for k, v in status_root.items() if k not in status_subnet)
TypeError: argument of type 'bool' is not iterable
2018-09-11 17:52:00,640 [Thread-338  ] [hbp_nrp_cles] [ERROR]  Error in CLE (General Error): argument of type 'bool' is not iterable
Traceback (most recent call last):
  File "/home/jakob/Documents/NRP/CLE/hbp_nrp_cle/hbp_nrp_cle/cle/DeterministicClosedLoopEngine.py", line 248, in __loop
    self.run_step(self.timestep)
  File "/home/jakob/Documents/NRP/CLE/hbp_nrp_cle/hbp_nrp_cle/cle/DeterministicClosedLoopEngine.py", line 185, in run_step
    self.bca.run_step(timestep * 1000.0)
  File "/home/jakob/Documents/NRP/CLE/hbp_nrp_cle/hbp_nrp_cle/brainsim/pynn/PyNNControlAdapter.py", line 211, in run_step
    self._sim.run(dt)
  File "/home/jakob/.opt/platform_venv/local/lib/python2.7/site-packages/pyNN/common/control.py", line 110, in run
    return run_until(simulator.state.t + simtime, callbacks)
  File "/home/jakob/.opt/platform_venv/local/lib/python2.7/site-packages/pyNN/common/control.py", line 93, in run_until
    return simulator.state.t
  File "/home/jakob/.opt/platform_venv/local/lib/python2.7/site-packages/pyNN/nest/simulator.py", line 70, in t
    return max(nest.GetKernelStatus('time') - self.dt, 0.0)  # note that we always simulate one time step past the requested time
  File "/home/jakob/.local/lib/python2.7/site-packages/nest/lib/hl_api_helper.py", line 230, in stack_checker_func
    return f(*args, **kwargs)
  File "/home/jakob/.local/lib/python2.7/site-packages/nest/lib/hl_api_simulation.py", line 118, in GetKernelStatus
    d = dict((k, v) for k, v in status_root.items() if k not in status_subnet)
  File "/home/jakob/.local/lib/python2.7/site-packages/nest/lib/hl_api_simulation.py", line 118, in <genexpr>
    d = dict((k, v) for k, v in status_root.items() if k not in status_subnet)
TypeError: argument of type 'bool' is not iterable
2018-09-11 17:52:00,641 [Thread-338  ] [transitions.] [INFO]  Initiating transition from state started to state halted...
2018-09-11 17:52:00,641 [Thread-338  ] [transitions.] [INFO]  Exited state started
2018-09-11 17:52:00,641 [Thread-338  ] [transitions.] [INFO]  Entered state halted
2018-09-11 17:52:01,148 [Thread-2    ] [rospy.intern] [INFO]  topic[/ros_cle_simulation/cle_error] removing connection to /virtual_coach
2018-09-11 17:52:01,148 [Thread-2    ] [rospy.intern] [INFO]  topic[/ros_cle_simulation/status] removing connection to /virtual_coach
2018-09-11 17:52:05,642 [Thread-338  ] [concurrent.f] [ERROR]  exception calling callback for <Future at 0x7fb817f7fad0 state=finished raised TypeError>
Traceback (most recent call last):
  File "/home/jakob/.local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 331, in _invoke_callbacks
    callback(self)
  File "/home/jakob/Documents/NRP/ExDBackend/hbp_nrp_cleserver/hbp_nrp_cleserver/server/SimulationServerLifecycle.py", line 120, in __handle_crash
    severity=CLEError.SEVERITY_CRITICAL)
  File "/home/jakob/Documents/NRP/ExDBackend/hbp_nrp_cleserver/hbp_nrp_cleserver/server/ROSCLEServer.py", line 160, in publish_error
    self.lifecycle.failed()
  File "/home/jakob/.opt/platform_venv/local/lib/python2.7/site-packages/transitions/extensions/locking.py", line 22, in trigger
    super(LockedEvent, self).trigger(*args, **kwargs)
  File "/home/jakob/.opt/platform_venv/local/lib/python2.7/site-packages/transitions/core.py", line 222, in trigger
    return self.machine.process(f)
  File "/home/jakob/.opt/platform_venv/local/lib/python2.7/site-packages/transitions/extensions/locking.py", line 15, in __call__
    return self.func(*args, **kwargs)
  File "/home/jakob/.opt/platform_venv/local/lib/python2.7/site-packages/transitions/core.py", line 526, in process
    return trigger()
  File "/home/jakob/.opt/platform_venv/local/lib/python2.7/site-packages/transitions/core.py", line 247, in _trigger
    if t.execute(event):
  File "/home/jakob/.opt/platform_venv/local/lib/python2.7/site-packages/transitions/core.py", line 151, in execute
    machine.callback(func, event_data)
  File "/home/jakob/.opt/platform_venv/local/lib/python2.7/site-packages/transitions/extensions/locking.py", line 15, in __call__
    return self.func(*args, **kwargs)
  File "/home/jakob/.opt/platform_venv/local/lib/python2.7/site-packages/transitions/core.py", line 516, in callback
    func(event_data)
  File "/home/jakob/Documents/NRP/ExDBackend/hbp_nrp_cleserver/hbp_nrp_cleserver/server/SimulationServerLifecycle.py", line 141, in fail
    self.__cle.stop(forced=True)
  File "/home/jakob/Documents/NRP/CLE/hbp_nrp_cle/hbp_nrp_cle/cle/DeterministicClosedLoopEngine.py", line 280, in stop
    raise Exception("The simulation loop could not be completed")
Exception: The simulation loop could not be completed
2018-09-11 17:52:05,644 [Thread-338  ] [hbp_nrp_cle ] [INFO]  Simulation loop ended

#5

Hello Jakob,

I just reported your crash in NUIT-42.

Best regards,
Luc


#6

Dear Jakob,

I just started to investigate your issue. For now, I have just checked that the examples of the tutorial do not cause any crash. Could you provide me with a minimal set-up so that I can try to reproduce the problem?

Best regards,
Luc


#7

Hi @lguyot,

Sorry for the long delay.
Since I posted about this issue I have finished my thesis and am no longer involved in HBP.
I will try to look into the issue when time allows, to wrap up properly.

will let you know.
Cheers