Record experiment on NRP


#1

Hi all,

is there some functionality to record an experiment on the NRP? It would be useful to make a smooth video, since my experiment is running slower than real time.

Thanks,

Alexander


#2

Hi Alexander,

Kenny implemented what we call the “simulation replay”. It is back-end ready but a front-end GUI is still missing.
I think Kenny will be glad to show you how to use this feature tomorrow.

Best regards,
Luc


#3

I would also be really interested in this feature, since my experiment running ratio is something like 1/89234723546248943!
Alban


#4

Hi Everyone,

The recorder is still pretty much in a development state with a couple of important things to note:

  • the backend is pretty much complete, but there is no frontend support at all for the recorder functions
  • there is also no VirtualCoach interface (yet)
  • there is no storage server integration, so the recordings are deleted by default and not saved

The VirtualCoach and storage server parts should change in the next sprint or two as other things become available, but the frontend portion will take some time and I’m not sure who/when it is planned.

It’s still mostly usable and you guys can:

  • record by using the NRP REST webpage to issue recorder commands
  • save recordings manually my copying the files
  • playback by editing one of the backend Python files for simulation launch

If you’re still interested, here are the major features:

  • recording: start/stop so you can make segmented recordings of only the parts you care about (or the whole thing if you want), reset discards all recordings so far if you mess up
  • playback: the frontend play/pause/reset buttons work as expected and the recordings will loop forever, the image/spike viewers should work as expected

Here are the actual steps, hopefully it’s not too awful but it should get way easier to use in the future.


Recording

  1. Launch your experiment of choice.

  2. Go to the REST API webpage:

http://localhost:8080/api/spec.html#!/spec/

  1. Scroll down and expand the section:

POST /simulation/{sim_id}/recorder/{command} Issue user commands to the simulator recorder.

  1. Here you can issue commands to the simulation recorder:

the sim_id (check the URL for your experiment - it’s 0 in this example): http://localhost:9000/#/esv-web/experiment-view/localhost/ExDTemplateHusky/false/0

provide a command:

start - starts recording
stop - stops and saves recording
cancel - stops and cancels recording (basically discard the last recording that has been started)
reset - discard all recordings

So basically run your simulation as normal and just start/stop/cancel as needed until you have the recordings you want.


Saving Recordings

Since there’s no storage server API yet, you’ll need to save your recordings manually when you are done. DO NOT STOP YOUR SIMULATION BEFORE THIS

  1. Open a terminal and run:

rosservice call /gazebo/recording/get_recording

  1. Copy the file path provided to some other location, e.g. (mv <path> <somewhere safe>)

  2. Now you can shutdown your simulation as normal and the tmp recording path will be cleaned up.


Playing Recordings

  1. You’ll have to cle-kill your platform and edit:

$HBP/ExDBackend/hbp_nrp_backend/hbp_nrp_backend/rest_server/__SimulationService.py

  1. Find the line:

playback_path = body.get('playbackPath', None)

and replace None with the path to the recording you saved above.

You’ll need to cle-kill and cle-start every time you make these changes, revert it to go back to normal mode.

  1. Launch the simulation as usual, it will be your simulation playback - don’t try to use the TF/brain editors or the brain visualizer right now. The spike plotter and image viewers should work as expected.

Summary

It’s not too awful, basically just run your simulation as usual and use the REST webpage to issue start/stop commands to the recorder. Then save the recordings manually before shutting down the simulation. Finally edit the backend file to launch with your recording and you should have near-relatime playback of the simulation.

I can answer questions and if it’s too difficult to figure out or if there are bugs, we can maybe have a screen share and discuss.


#5

Hi Kenny,

thanks for the explanation, I tried it out and it does work!

However it seems my local installation became somewhat instable,

after reboot I can launch an experiment once, but the second time I get a ‘gazebo died unexpectedly’:

[ INFO] [1505850890.844293222]: Gazebo ROS Recording Plugin initialized.
[ INFO] [1505850890.849042465]: Gazebo ROS Playback Plugin initialized.
X Error of failed request:  BadDrawable (invalid Pixmap or Window parameter)
  Major opcode of failed request:  154 (DRI2)
  Minor opcode of failed request:  3 (DRI2CreateDrawable)
  Resource id in failed request:  0x5600002
  Serial number of failed request:  31
  Current serial number in output stream:  33
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error> >'
  what():  boost: mutex lock failed in pthread_mutex_lock: Invalid argument
2017-09-19 21:54:52,354 [Thread-57   ] [hbp_nrp_cles] [ERROR]  Gazebo died unexpectedly
2017-09-19 21:54:53,354 [Thread-57   ] [hbp_nrp_cles] [ERROR]  Gazebo died unexpectedly

Cle-kill cle-start does not help
nor does killall gazebo/gzserver/gzclient

only a complete reboot works…

any idea? is there a process I could kill?

Alexander


#6

Hi Alexander,

Glad the recording/playback steps worked.

Unfortunately your new issue is related to your graphics driver more than anything in the NRP. Has it been updated recently? I had this happen occasionally on Ubuntu 14.04, but it seemed to go away when I moved to Ubuntu 16 (no guarantee that would do anything for you).

See: https://bitbucket.org/osrf/gazebo/issues/2229/gazebo-gui-crashes-sometimes (and there are a bunch of other similar ones)

You can try to recompile Gazebo to see if it resolves the issue first when it links to updated drivers or something, but there is also a new option to compile Gazebo to use software only (GPU-less) rendring which may be more stable in your setup. I’m not super familiar with the process, but @lguyot may be able to help if the first rebuild does not resolve your issue.

Best,
Kenny