Dear Jakob,
You will find information about modifying, spawning and deleting objects in the NRP Guide Book. You may want to have a look at the files
$HBP/Experiments/screen_switching_holodeck/screen_switching_husky_aware_compatible.exd and $HBP/Experiments/demo_mouse_lab/mouse_smach_state.exd to get started.
Best regards,
Luc
Thank you @lguyot ,
I am not sure if it was clear that I am looking for a solution for c++ through a model plugin?
Either way, your answer pointed me in the direction of the python call “SetMaterialColorServiceState” which appears to use the “/gazebo/set_visual_properties” service, with a message of type: “gazebo_msgs/SetVisualProperties”
I am now trying to find a way to use this service in my plugin, but currently struggling a bit to include the custom message type in my CMakeLists.txt. Would you have any pointers on how to do this?
Also, do I understand this properly, that the SetVisualProperties message only contains one property at a time? Meaning that I will have to transmit 3 separate messages to set both color, pose and scale?
Thanks again for your help
To elaborate a little, my use-case is a custom muscle setup.
I need to place a visual representation of each muscle in the scene and update its position, scale and color (RGB) for each world update event.
I can do this in ROS/Gazebo but I cannot find out how to do it in NRP
I see that the gazebo_ros_muscle_interface can do this using OpenSim, but I have not been able to find out how/where the muscle visuals are being updated.
If anyone can shed some light on this as well it would be highly appreciated
Thanks
Dear Jakob,
The implementation of the service /gazebo/set_visual_properties and the definition of the message gazebo_msgs/SetVisualProperties are part of our $HBP/GazeboRospackages, see src/gazebo_ros_pkgs/gazebo_ros/src/gazebo_ros_api_plugin.cpp and src/gazebo_ros_pkgs/gazebo_msgs/.
Hopefully, the CMakeLists.txt files you will find here will help (you can share your problems).
To set color, you would use /gazebo/set_visual_properties, and to set the scale and the pose, you would use /gazebo/set_model_state.
The visualization of the muscles is done in the NRP version of gazebo, see gazebo/rendering/OpenSimMuscleVisual.cc.
Best regards,
Luc
Hi @lguyot,
Thanks for the file pointers!
I’ve looked through them, but it appears that the current implementation of SetVisualProperties only allow to set materials, not color? As I need a gradient between blue and red, I would need to be able to set RGB values. Am I missing something?
As for the OpenSim muscle visualisation, I am still trying to figure out how they are doing it. Appears they create everything inside a Visual plugin, where simply setting “this->SetDiffuse(color)” results in updating the color?
I assume that “this” in that case refers to the visuals of the link, but I am not sure how they established that pointer and I can’t seem to find too much documentation on visual plugins.
Is “this” established from the SDF file, where the visual plugin gets loaded, or by simply creating the visuals on the fly, when the plugin loads, or something else?
Also, is there a particular reason why the “~/visual” topic is not available in NRP or is it simply not implemented yet?
Cheers and thanks again
Hello Jakob,
Yes, the SetVisualProperties supports only material change at the moment. You can fill a feature request here if need be.
The Gazebo color of the muscles depend on their activation value (gazebo/rendering/OpenSimMuscleVisual.cc). This value is published by gazebo on the topic ~/muscles using a message of type msg::OpenSimMuscles (gazebo/physics/OpenSimPhysics.cc:lines 726 and 784). Activation values are computed and updated by OpenSim.
The “this” on line 212 of OpenSimMuscleVisual,cc refers to a FiberVisual object wich is used to populate the list
dPtr->muscle_visuals of an OpenSimMuscleVisual (see OpenSimMuscleVisual::CreateNewMuscleVis, lines 368 – 382).
The visuals are not created based on the content of the loaded SDF file. They are created on the fly when a message is published on the topic ~/muscles (see lines 362 -363 of OpenSimMuscleVisual::SetEnabled and OpenSimMuscleVisual::Update at line 265).
About “~/visual”, the NRP should have all the built-in Gazebo topics.
gz topic -l | grep "visual"
/gazebo/default/visual
Is it what you are looking for?
Best regards,
Luc
Hi @lguyot
Thank you very much for the great answers! I have looked through the files you referred to, but I will probably have to admit that it is beyond me to fully understand the OpenSim muscle implementation for now.
However, regarding the “gazebo/default/visual” topic, I see now that it is actually there, but for some reason isn’t responding as when run in Gazebo directly. I guess that means that I was looking for the issue in the wrong place. Thanks for pointing it out!
Indeed, if I could only get the visual topic to respond to my messages in NRP, then my problems would be solved.
I have made some tests and found that the message gets published alright. I can listen to the visual topic which shows the intended message, but for some reason NRP does not want to update the visuals.
The message is exactly the same in Gazebo and NRP and looks as follow (publisher is initiated with topic name: “/gazebo/default/visual” but appears to convert it to “~/visual” automatically):
{
"op":"publish","topic":"~/visual", "msg": {
"name": "my_muscles::m_agonist_link", "material": {
"ambient": {"r": 0.0, "g": 0.0, "b": 1.0, "a": 0.5},
"diffuse": {"r": 0.0, "g": 0.0, "b": 1.0, "a": 0.5}
},
"parent_name": "my_muscles",
"parent_id": 9,
"scale": {
"x": 0.043648510166301314, "y": 0.043648510166301314, "z": 0.42702979667397373
}
}
}
Despite this message being successfully published on the visual topic, the visual in question neither scales nor changes color as it is supposed to, and is doing when I run the same code in Gazebo.
Can you think of any potential reason for this?
My function for generating the visual message looks like this, in case it helps:
gazebo::msgs::Visual JWelnerMuscle::GetVisMsg( physics::LinkPtr link_,\
gazebo::common::Color newColor,\
ignition::math::Vector3d scale_) {
sdf::ElementPtr linkSDF = link_->GetSDF();
sdf::ElementPtr visualSDF = linkSDF->GetElement("visual");
std::string visual_name_ = visualSDF->Get<std::string>("name");
gazebo::msgs::Visual visualMsg = link_->GetVisualMessage(visual_name_);
gazebo::msgs::Vector3d* scale_factor = new gazebo::msgs::Vector3d{gazebo::msgs::Convert(scale_)};
visualMsg.set_name(link_->GetScopedName());
visualMsg.set_parent_id(link_->GetParentId());
visualMsg.set_parent_name(link_->GetParent()->GetScopedName());
visualMsg.set_allocated_scale(scale_factor);
// Set material
if ((!visualMsg.has_material()) || visualMsg.mutable_material() == NULL) {
gazebo::msgs::Material *materialMsg = new gazebo::msgs::Material;
visualMsg.set_allocated_material(materialMsg);
}
// Set color
gazebo::msgs::Color *colorMsg = new gazebo::msgs::Color(gazebo::msgs::Convert(newColor));
gazebo::msgs::Color *diffuseMsg = new gazebo::msgs::Color(*colorMsg);
gazebo::msgs::Material *materialMsg = visualMsg.mutable_material();
if (materialMsg->has_ambient())
{
materialMsg->clear_ambient();
}
materialMsg->set_allocated_ambient(colorMsg);
if (materialMsg->has_diffuse())
{
materialMsg->clear_diffuse();
}
materialMsg->set_allocated_diffuse(diffuseMsg);
return visualMsg;
}
Node is initiated with:
node_ = gazebo::transport::NodePtr(new gazebo::transport::Node());
node_->Init(this->world->GetName());
pub_visual_ = node_->Advertise<gazebo::msgs::Visual>("/gazebo/default/visual");
And the visualMsg is published with
pub_visual_->Publish( visualMsg );
Does any of this employ functions or name conventions that might change between native Gazebo an NRP? Currently I am at a loss…
Thanks a lot
Dear Jakob,
I suspect that the visual message update is not properly propagated to the NRP front-end where the threeJS rendering is done. Did you make any NRP-free tests with gazebo and its web client, i.e. gzweb?
In order to debug in the NRP, you can set a breakpoint in the visualUpdate function at lines 465 – 496 of $HBP/gzweb/gz3d/gziface.js.
Best regards,
Luc
Thanks again,
You are right, the messages do not seem to propagate to the NRP front-end. However, I don’t know where they get stuck.
I looked in the …/$HBP/gzweb/gz3d/src/gziface.js file you mentioned (You meant the one in src right?) and there is yet another silent filter, this time ignoring any messages that are not a COLLISION_VISUAL.
if (message.name.indexOf('COLLISION_VISUAL') < 0)
{
return;
}
To explore a little further I tried adding “debugger;” above the check in the .js file, as well as console.log(string), console.debug(string) and console.error(new Error(string)), and running my project that sends visual messages but nothing showed up in either of the consoles running NRP - ie. one console where I executed
$ cle-nginx; cle-start
and another where I ran:
$ cle-frontend --debug
I don’t know if I am doing something wrong or whether the command is actually never run. Must admit that I’ve never touched nodeJS before. Either way I am not sure how to move on from here. I tried a naive solution of out-commenting the check, to allow other types of visual messages to pass, in case something could work magically, but nothing changed.
As for making an NRP-free test with gzweb I assume you mean to try and run the gzserver and gzweb client directly. I tried with:
Term 1
$ gzserver
Term 2:
$ cd $HBP/gzweb
$ ./start_gzweb.sh
It started up with the following error and I couldn’t click on anything in the web interface
Starting up http-server, serving http/client
Available on:
http://127.0.0.1:8081
http://192.168.1.138:8081
Hit CTRL-C to stop the server
module.js:681
return process.dlopen(module, path._makeLong(filename));
^
Error: Module did not self-register.
at Object.Module._extensions..node (module.js:681:18)
at Module.load (module.js:565:32)
at tryModuleLoad (module.js:505:12)
at Function.Module._load (module.js:497:3)
at Module.require (module.js:596:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/home/jakob/Documents/NRP/gzweb/gzbridge/ws_server.js:15:13)
at Module._compile (module.js:652:30)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)
Due to it being part of NRP I wasn’t brave enough to try to change anything further, for the chance that I would break something else.
Bottomline is probably that what I try to do is simply not supported yet, so it is likely to require an NRP update before I can be able to change color on the fly.
At this point I might give up and try to take a different path, before spending all of my time on something that should have been a detail.
Dear Jakob,
The logs your are looking for are located in your browser console (Tools --> JavaScript console for Chrome, Tools --> Web Developer --> Web Console for Firefox). These logs will tell you, at least, if the message has been received by the front-end. If so, then we can check how exactly the message is processed.
Best regards,
Luc
Alright, I found the console and the file that is processing the visual message in the frontend, which turned out to be:
$HBP/ExDFrontend/bower_components/gz3d-hbp/gz3d/build/gz3d.js
Generally it simply doesn’t support anything else then material script changes and I can’t tell what would be required if I were to implement support for color and scale myself.
Also it seems that every time something falls outside of the required template it is failing silently, so I’m struggling with finding out IF anything actually failed.
I will probably have to give up on fixing this myself so I’ve logged a feature request through the link you sent
Thanks for all your great pointers though. They were much appreciated, and at least I know a bit more about NRP now 
A few final notes.
I’ve managed to add support for position and scale in the visual message, and it turns out that this way updates much faster than using link->SetWorldPose(pose) - for some reason? SetWorldPose seems to run only occasionally despite being executed with higher frequency.
I have some objects where I use SetWorldPose and some where I set their position with a visual msg, and the latter moves much smoother, despite setting both simultaneously in the same method.
As for color, I’ve managed to set the color in the material according to my message. However, despite the final object appears to have the right values at the end of SetMaterial in gz3d.js (before this.updateDynamicEnvMap(); and this.refresh3DViews();), nothing changes in the viewport
I can’t tell if something in my material is off without me knowing or whether the color is being silently overwritten again at some point further down the pipeline,
Dear Jakob,
Thanks for this detailed feedback. Did you have to change/fix the NRP Javascript code to update “successfully” your material?
At this point, I would need your code and some instructions as how to run it in order to check what’s wrong in the material update pipeline.
Best regards,
Luc
Hi @lguyot,
I have uploaded the files here: http://jakob.welner.dk/NRP_visMsgTestPackage.zip
In case the precompiled library doesn’t work, you can find the source here: https://bitbucket.org/jakobwelner/my_leg01_plugin/src/master/
The zip includes the plugin library, the NRP project as found in ~/.opt/nrpStorage and the edited gzweb file that implements scaling and positioning through visual messages - and tries to add color support as well. It’s all quite frankenstein’y so don’t expect any clean code if you look deeper 
The gzweb file should replace the file:
$NRP/ExDFrontend/bower_components/gz3d-hbp/gz3d/build/gz3d.js
To run:
copy nrpStorage/my_leg01_0 into your local ~/.opt/nrpStorage, put the plugin library somewhere on the gazebo plugin path, open NRP and launch the project called “My_leg01”. Click play to start sending the visual messages.
See the JavaScript console for logs of the received visual message and visual object as well as the final object to be updated.
As you can see in the final object, the material color gets set to RGB of [0,0,255] which should be blue, but this is never applied to the model, while position and scale gets updated with no problem.
I hope this data is sufficient for you to run it. Otherwise let me know and thanks again.
Hello, Jakob!
The color range that is used to visualize muscles and their activation is currently “hard-coded” in the NRP.
What you need to look for if you want to customize geometry types and/or color range are the files
OpenSimMuscleVisual.hh and OpenSimMuscleVisual.cc in $HBP/gazebo/gazebo/rendering (referring to the Gazebo fork the NRP uses).
These are the visual representations used by the OpenSim plugin to visualize a muscle system in gzclient, Gazebo’s desktop frontend.
For the web-based frontend, the client-side (WebGL/Three.js) visualization is located in $HBP/gzweb/gz3d/src/gz3d-muscle-vis.js
Finally, to send muscle visual properties via gzbridge to the web frontend, take a look at $HBP/gzweb/gzbridge/GazeboInterface.(hh|cc), specifically for the member function
GazeboInterface::OnMuscleVisualizationMsg and the member variables it manipulates.
Hi Jakob,
I have got the following issue when using you code:
Created an instance of a muscle class
Entered InitiateMusclesgzserver: /usr/include/boost/smart_ptr/shared_ptr.hpp:648: typename boost::detail::sp_member_access<T>::type boost::shared_ptr<T>::operator->() const [with T = gazebo::physics::Model; typename boost::detail::sp_member_access<T>::type = gazebo::physics::Model*]: Assertion `px != 0' failed.
I wish I could compile the My_Leg plugin myself but I don’t have the rights to access your bitbucket repository.
Best regards,
Luc
Oh, woups. I’ve changed the permissions now. You should be able to clone the repos.
As for the error, I am not sure why. This is the kind of error I have seen a few times when I’ve misspelled or forgot to include a model/link in the SDF before referring to it in the plugin. It’s not really too informative, but I admit that my code could be checking a few more things before assuming stuff… this goes back to the frankenstein’y stage of my code.
I hope it works when you compile yourself. Let me know!
Dear Jakob,
I have been able to reproduce the problem. This is due to the fact that setRGB() (see setMaterial in gz3d.js) expects arguments to be taken in [0, 1] and not in [0, 255].
Best regards,
Luc
Hi @lguyot,
Does it mean that you got it working with changing color?
I actually tried with 0-1 range initially, because that is the required range in Gazebo. I only changed to 0-255 range after I found something that seemed to suggest it to be the expected range in NRP.
I could get neither to work, but I will try again with 0-1 range and see if it changes anything now.
Cheers and thanks
I get it “working” by hard-coding the blue value in the setRGB() instructions of the setMaterial function. Now I realise that the diffuse values of your mesh material are not the expected ones and it is only the material of the parent which is updated properly. So there is another problem and I need to investigate further.