New installation doesn't launch experiment


#1

I’ve got a new stand alone installation.
I get the experiments list, but when I launch one I get a pop up box that has:

Error
Internal server error: service [/ros_cle_simulation/create_new_simulation] responded with an error: service cannot process request: transport error completing service call: unable to receive data from sender, check sender’s logs for details (host: localhost)

I did have problems building the system, so the NRP section was completed second to last and the ros section last. Any ideas?


#2

Hi Chris,

Your error message indicates that something went wrong with ROS, and possibly Gazebo. It would be helpful if you could report here

  • the first installation problem you had (with a log)
  • the complete log of the simulation before receiving the error you mentioned.

Best regards,
Luc


#3

I don’t have a log of the first installation problem.
Here is the log of the current run:
[rospy.client][INFO] 2017-05-08 11:03:30,871: init_node, name[/ros_cle_simulation_3584_1494237810847], pid[3584]
[xmlrpc][INFO] 2017-05-08 11:03:30,871: XML-RPC server binding to 127.0.0.1:0
[xmlrpc][INFO] 2017-05-08 11:03:30,871: Started XML-RPC server [http://127.0.0.1:44950/]
[rospy.init][INFO] 2017-05-08 11:03:30,871: ROS Slave URI: [http://127.0.0.1:44950/]
[rospy.impl.masterslave][INFO] 2017-05-08 11:03:30,871: _ready: http://127.0.0.1:44950/
[xmlrpc][INFO] 2017-05-08 11:03:30,872: xml rpc node: starting XML-RPC server
[rospy.registration][INFO] 2017-05-08 11:03:30,872: Registering with master node http://localhost:11311
[rospy.init][INFO] 2017-05-08 11:03:30,972: registered with master
[rospy.rosout][INFO] 2017-05-08 11:03:30,972: initializing /rosout core topic
[rospy.rosout][INFO] 2017-05-08 11:03:30,974: connected to core topic /rosout
[rospy.simtime][INFO] 2017-05-08 11:03:30,975: /use_sim_time is not set, will not subscribe to simulated time [/clock] topic
[hbp_nrp_cleserver][WARNING] 2017-05-08 11:03:30,982: Could not write to specified logfile or no logfile specified, logging to stdout now!
[rospy.internal][INFO] 2017-05-08 11:03:31,269: topic[/rosout] adding connection to [/rosout], count 0
[hbp_nrp_cleserver][INFO] 2017-05-08 11:05:00,597: Create new simulation request
[hbp_nrp_cleserver][INFO] 2017-05-08 11:05:00,597: No simulation running, starting a new simulation.
[hbp_nrp_cleserver][INFO] 2017-05-08 11:05:00,598: Preparing new simulation with environment file: /home/chris/Documents/NRP/Models/empty_world/empty_world.sdf and ExD config file: /home/chris/Documents/NRP/Experiments/template_husky/ExDTemplateHusky.exc.
[hbp_nrp_cleserver][INFO] 2017-05-08 11:05:00,598: Starting the experiment closed loop engine.
[hbp_nrp_cleserver][INFO] 2017-05-08 11:05:00,598: Read XML Files
[hbp_nrp_cleserver][INFO] 2017-05-08 11:05:00,614: Bibi: template_husky.bibi
[hbp_nrp_cleserver][INFO] 2017-05-08 11:05:00,614: BibiAbs:/home/chris/Documents/NRP/Experiments/template_husky/template_husky.bibi
[hbp_nrp_cleserver][INFO] 2017-05-08 11:05:00,620: Creating local CLELauncher object
[hbp_nrp_cleserver.server.CLELauncher][INFO] 2017-05-08 11:05:02,421: Path is /home/chris/Documents/NRP/Experiments/template_husky
[hbp_nrp_cleserver.server.CLELauncher][INFO] 2017-05-08 11:05:02,421: Creating ROSCLEServer
[hbp_nrp_cleserver.server.CLELauncher][INFO] 2017-05-08 11:05:02,424: Setting up backend Notificator
[hbp_nrp_cleserver.server.CLELauncher][INFO] 2017-05-08 11:05:02,425: Robot: husky_model/model.sdf
[hbp_nrp_cle.user_notifications][INFO] 2017-05-08 11:05:02,425: Starting gzserver
[hbp_nrp_cle.user_notifications][INFO] 2017-05-08 11:05:06,010: Restarting Gazebo web client communication adapter
[hbp_nrp_cle.user_notifications][INFO] 2017-05-08 11:05:09,919: Cleaning model ground_plane.
[hbp_nrp_cle.user_notifications][INFO] 2017-05-08 11:05:10,344: Cleaning lights
[hbp_nrp_cle.user_notifications][INFO] 2017-05-08 11:05:10,356: Loading light “sun”.
[hbp_nrp_cle.user_notifications][INFO] 2017-05-08 11:05:10,546: Loading model “plane”.
[hbp_nrp_cleserver.server.CLELauncher][INFO] 2017-05-08 11:05:10,775: RobotAbs: /home/chris/Documents/NRP/Models/husky_model/model.sdf
[hbp_nrp_cleserver.server.ROSCLEServer][WARNING] 2017-05-08 11:05:11,386: Previous task was not closed properly, closing it now.
[hbp_nrp_cle.user_notifications][INFO] 2017-05-08 11:05:11,387: Stopping Gazebo web client communication adapter
[hbp_nrp_cleserver.server.CLELauncher][ERROR] 2017-05-08 11:05:11,517: Gazebo died unexpectedly
[hbp_nrp_cleserver.server.CLELauncher][ERROR] 2017-05-08 11:05:12,518: Gazebo died unexpectedly
[hbp_nrp_cle.user_notifications][INFO] 2017-05-08 11:05:12,668: Stopping gzserver
[hbp_nrp_cleserver.server.ROSCLEServer][INFO] 2017-05-08 11:05:12,678: Unregister error/transfer_function topic
[rospy.internal][INFO] 2017-05-08 11:05:12,679: topic impl’s ref count is zero, deleting topic /ros_cle_simulation/cle_error…
[hbp_nrp_cleserver.server.ROSCLEServer][INFO] 2017-05-08 11:05:12,681: Unregister status topic
[rospy.internal][INFO] 2017-05-08 11:05:12,681: topic impl’s ref count is zero, deleting topic /ros_cle_simulation/status…
[hbp_nrp_cleserver.server.CLELauncher][INFO] 2017-05-08 11:05:12,683: Cleaning up ROS nodes and services
[hbp_nrp_cleserver][ERROR] 2017-05-08 11:05:13,538: Initialization failed
Traceback (most recent call last):
File “/home/chris/Documents/NRP/ExDBackend/hbp_nrp_cleserver/hbp_nrp_cleserver/server/ROSCLESimulationFactory.py”, line 177, in create_new_simulation
cle_launcher.cle_function_init(environment_file, timeout, self.except_hook)
File “/home/chris/Documents/NRP/ExDBackend/hbp_nrp_cleserver/hbp_nrp_cleserver/server/CLELauncher.py”, line 249, in cle_function_init
.load_gazebo_model_file(‘robot’, robot_file_abs, rpose)
File “/home/chris/Documents/NRP/CLE/hbp_nrp_cle/hbp_nrp_cle/robotsim/GazeboHelper.py”, line 224, in load_gazebo_model_file
self.load_sdf_entity(model_name, model_sdf, initial_pose)
File “/home/chris/Documents/NRP/CLE/hbp_nrp_cle/hbp_nrp_cle/robotsim/GazeboHelper.py”, line 245, in load_sdf_entity
self.spawn_entity_proxy(model_name, model_sdf, “”, initial_pose, “world”)
File “/opt/ros/indigo/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py”, line 435, in call
return self.call(*args, **kwds)
File “/opt/ros/indigo/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py”, line 525, in call
raise ServiceException(“transport error completing service call: %s”%(str(e)))
ServiceException: transport error completing service call: unable to receive data from sender, check sender’s logs for details


#4

Dear Chris,

I re-compiled Gazebo on my side, to check whether the latest version (7.2.0.hbp.1.2.0) could be responsible for the error you got. After that, I have been able to run ExDTemplateHusky experiment without problems.

So it’s probably something else. Comparing your logs with mine, the first noticeable difference is there:

hbp_nrp_cleserver.server.ROSCLEServer][WARNING] 2017-05-08 11:05:11,386: Previous task was not closed properly, closing it now.
[hbp_nrp_cle.user_notifications][INFO] 2017-05-08 11:05:11,387: Stopping Gazebo web client communication adapter
[hbp_nrp_cleserver.server.CLELauncher][ERROR] 2017-05-08 11:05:11,517: Gazebo died unexpectedly

This is a rare scenario handled by line 779 of hbp_nrp_cleserver/server/ROSCLEServer.py and I am trying to figure out what could bring us there.

In order rule out possible errors caused by former simulations, you may try in the mean time:
cle-kill
cle-nginx
cle-start


#5

I was starting after a reboot, so I wouldn’t expect the cle-kill… approach to work. None the less, I tried, and had the same error as before.
I have the sense that there is something wrong with the lock. (I really have little idea what the lock is.) The typical way I start is nginx in one shell cle-start in another and cle-frontend in the third. In the cle-start shell I get a line:
cat: /home/chris/.gazebo/gzserver.lock: No such file or directory
Does this provide any useful information for you?


#6

Hi Chris,

You can safely ignore the lock error, it’s a trivial bug in one of the launch scripts that terminates any lingering Gazebo processes.

It looks like your Gazebo is failing to launch, can you try to run:

gzserver -s $HBP/GazeboRosPackages/devel/lib/libgazebo_ros_api_plugin.so -s $HBP/GazeboRosPackages/devel/lib/libgazebo_ros_paths_plugin.so

and see if it launches OK?

If that launches fine, you can also try to increase launch verbosity with:

$HOME/.opt/bbp/nrp-services/gzserver

modify this line (33 in my config):

$exec -s $HBP/GazeboRosPackages/devel/lib/libgazebo_ros_api_plugin.so -s $HBP/GazeboRosPackages/devel/lib/libgazebo_ros_paths_plugin.so &

to add a --verbose after the exec command, e.g.

$exec --verbose <the rest of the stuff>

Then do a cle-kill and cle-start and copy/paste the log for us there.

Kenny


#7

On second note, it’s probably better to run the first command as:

gzserver --verbose -s $HBP/GazeboRosPackages/devel/lib/libgazebo_ros_api_plugin.so -s $HBP/GazeboRosPackages/devel/lib/libgazebo_ros_paths_plugin.so

That should print our something like:

Gazebo multi-robot simulator, version 7.2.0.hbp.1.1.6 Copyright (C) 2012-2016 Open Source Robotics Foundation. Released under the Apache 2 License. http://gazebosim.org

Make sure it’s a hbp version, and you can safely ctrl+c that if it launches fine.


#8

Sorry for the spam steam of consciousness, can you verify that these files exist:

$HBP/GazeboRosPackages/devel/lib/libgazebo_ros_api_plugin.so

$HBP/GazeboRosPackages/devel/lib/libgazebo_ros_paths_plugin.so

#10

Sorry it looks like my posts are hidden due to community flagging. Perhaps they’re too big. Let me try this one.


#11

The nrp_backend.log is huge. Here is a section with a stacktrace in:
[hbp_nrp_commons.simulation_lifecycle][ERROR] 2017-05-09 14:18:54,135: Error trying to execute command initialized
[hbp_nrp_commons.simulation_lifecycle][ERROR] 2017-05-09 14:18:54,135: service [/ros_cle_simulation/create_new_simulation] responded with an error: error processing request: timeout exceeded while waiting for service /gazebo/spawn_sdf_entity
Traceback (most recent call last):
File “/home/chris/Documents/NRP/ExDBackend/hbp_nrp_commons/hbp_nrp_commons/simulation_lifecycle.py”, line 177, in accept_command
self.__machine.events[command].trigger()
File “/home/chris/Documents/NRP/ExDBackend/platform_venv/lib/python2.7/site-packages/transitions/extensions/locking.py”, line 22, in trigger
super(LockedEvent, self).trigger(*args, **kwargs)
File “/home/chris/Documents/NRP/ExDBackend/platform_venv/lib/python2.7/site-packages/transitions/core.py”, line 222, in trigger
return self.machine.process(f)
File “/home/chris/Documents/NRP/ExDBackend/platform_venv/lib/python2.7/site-packages/transitions/extensions/locking.py”, line 15, in call
return self.func(*args, **kwargs)
File “/home/chris/Documents/NRP/ExDBackend/platform_venv/lib/python2.7/site-packages/transitions/core.py”, line 526, in process
return trigger()
File “/home/chris/Documents/NRP/ExDBackend/platform_venv/lib/python2.7/site-packages/transitions/core.py”, line 247, in _trigger
if t.execute(event):
File “/home/chris/Documents/NRP/ExDBackend/platform_venv/lib/python2.7/site-packages/transitions/core.py”, line 145, in execute
machine.callback(func, event_data)
File “/home/chris/Documents/NRP/ExDBackend/platform_venv/lib/python2.7/site-packages/transitions/extensions/locking.py”, line 15, in call
return self.func(*args, **kwargs)
File “/home/chris/Documents/NRP/ExDBackend/platform_venv/lib/python2.7/site-packages/transitions/core.py”, line 516, in callback
func(event_data)
File “/home/chris/Documents/NRP/ExDBackend/hbp_nrp_backend/hbp_nrp_backend/simulation_control/__BackendSimulationLifecycle.py”, line 145, in initialize
simulation.sim_id, str(simulation.kill_datetime)
File “/home/chris/Documents/NRP/ExDBackend/hbp_nrp_backend/hbp_nrp_backend/cle_interface/ROSCLESimulationFactoryClient.py”, line 58, in create_new_simulation
brain_processes, sim_id, timeout)
File “/opt/ros/indigo/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py”, line 435, in call
return self.call(*args, **kwds)
File “/opt/ros/indigo/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py”, line 515, in call
responses = transport.receive_once()
File “/opt/ros/indigo/lib/python2.7/dist-packages/rospy/impl/tcpros_base.py”, line 696, in receive_once
p.read_messages(b, msg_queue, sock)
File “/opt/ros/indigo/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py”, line 353, in read_messages
self._read_ok_byte(b, sock)
File “/opt/ros/indigo/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py”, line 336, in _read_ok_byte
raise ServiceException(“service [%s] responded with an error: %s”%(self.resolved_name, str))
ServiceException: service [/ros_cle_simulation/create_new_simulation] responded with an error: error processing request: timeout exceeded while waiting for service /gazebo/spawn_sdf_entity
[transitions.core][INFO] 2017-05-09 14:18:54,140: Initiating transition from state created to state failed…


#12

The error box is now:
Internal server error: service [/ros_cle_simulation/create_new_simulation] responded with an error: error processing request: timeout exceeded while waiting for service /gazebo/spawn_sdf_entity (host: localhost)


#13

Ok that helps, it looks like either:

  1. Gazebo is not actually being launched
  2. Gazebo is crashing immediately after launch

To help rule out #1, can you past the contents of:

$HBP/CLE/hbp_nrp_cle/hbp_nrp_cle/config.ini

That’s the config file the platform uses to determine how to launch Gazebo.

Kenny


#14

[gazebo]
start-cmd=$HOME/.opt/bbp/nrp-services/gzserver start
stop-cmd=$HOME/.opt/bbp/nrp-services/gzserver stop
restart-cmd=$HOME/.opt/bbp/nrp-services/gzserver restart

[gzbridge]
start-cmd=$HOME/.opt/bbp/nrp-services/gzbridge start
stop-cmd=$HOME/.opt/bbp/nrp-services/gzbridge stop
restart-cmd=$HOME/.opt/bbp/nrp-services/gzbridge restart

[ros]
ros-cle-node-name=ros_cle_simulation

[network]
interfaces=eth0


#15

Ok that looks fine, for completeness sake can you try to run:

$HOME/.opt/bbp/nrp-services/gzserver restart

That should hopefully end up in the same “No ROS master” message if it launches successfully. You can run cle-kill in a terminal to shut that down.

If that launches fine, we may have to defer you to one of the developers who has made the most recent changes to our Gazebo build to help debug.


#16

I ran it with the other system up in the browser and got errors. I rebooted, and ran it, and then got the same “No ROS master” message. I’ve spent more time trying to fix this problem than the actual install took. Shall I just delete my NRP directory, and start the installation process again?


#17

I don’t think you need to blow everything away, it looks properly setup, just some weird stuff with Gazebo crashing on your system. If you have the time to run a few commands in the background, try this block:

# rebuild gazebo cd $HBP/gazebo rm -rf build mkdir build cd build cmake -DCMAKE_INSTALL_PREFIX=$HOME/.local -DENABLE_TESTS_COMPILATION:BOOL=False .. make -j8 # adapt 8 to the number of cores of your computer make install

then, close your current terminal, reopen it and:

cd $HBP/GazeboRosPackages rm -rf build devel catkin_make

then, again close and reopen a new terminal and try to launch the platform. I suspect something bad happened in either Gazebo or the Gazebo plugin build.

Kenny


#18

Ok, I rebuilt gazebo and GazeboRosPackages, but I’m back to the original error on launch.


#19

The post was blocked because of all of the links you tried to post to a single host. This is one of the filters which is applied to new users. You should be able to post such a message in the future if you redact/edit the links to replace http://127.0.0.1 or other hosts with a shorthand eg:( [localhost] ) or something else which doesn’t show up as a link.


#20

I just rebuilt everything from scratch on my platform and unfortunately it’s all working for me. We’re currently at an SP10 event, but maybe tomorrow or later @lguyot or @vonarnim may have some ideas. I’ll see if anything else comes to mind.


#21

I’ve just deleted the NRP directory, and am reinstalling from scratch. I should finish tomorrow morning. I’ll report back here if it works or not.