[SOLVED] Camera trigger runs twice for same image

#1

Hello,
I save images from a camera into csv file and I found that there are images with same seq number even though trigger is set to the camera.

The (minimal) code seems like that

@nrp.Robot2Neuron(triggers="camera_left") #trigger is set to a camera
def sensor_collector(t, visual_recorder, camera_left, position_service):
    ball_msg = [position_service.value('ball', 'world')]
    camera_left_msg = [camera_left.value]
    position_x = ball_msg[0].pose.position.x #position of the ball

    visual_recorder.record_entry(camera_left_msg[0], position_x) #save into csv file

In csv file I can see 2 records with the same image and different position_x… for example,

header: 
  seq: 14
  stamp: 
    secs: 2
    nsecs: 602010000
... other info
data: [55, 56, IMAGE..138],1.515..
header:   #next record
  seq: 14  #same seq
  stamp:  #same stamp
    secs: 2
    nsecs: 602010000
... other info
data: [55, 56,.SAME IMAGE..138], 1.517..

Therefore, it seems that the camera trigger runs twice for same image. Is there any way how to run the collector only once for an image and avoid multiple records with same seq number?

Thank you in advance :slight_smile:

Zdenek

#2

P.S. It seems that it happens mostly at the beginning of a recording. It might be only at the beginning but not sure about that…

#3

Dear Zdenek,

This sounds like a bug, that I reported to the development team as
https://hbpneurorobotics.atlassian.net/browse/NUIT-245
I guess you checked the documentation already:
https://neurorobotics.net/Documentation/nrp/tutorials/transfer_function/triggers.html

Best regards
Axel

#4

Dear @strakzd1,
unfortunately I cannot reproduce your issue, could you provide the full Transfer Function you are using? In the best case send me a Transfer Function that I could add to the template Husky experiment so that we share the overall experiment setup.
Thanks!
Benedikt

#5

Hi @BenediktFeldotto,
thank you for your answer. I created a minimal version of the project sufficient for obtaining the bug (with iCub). I added it as an attachment here https://hbpneurorobotics.atlassian.net/browse/NUIT-245
It should be very easy and direct to reproduce it. In case of any problems, please contact me.
Thanks.

Best,
Zdenek

#6

Dear Zdenek,

We are still investigating this anomalous behavior of the camera sensor, and have uncovered some potential issue with Gazebo itself. We will let you know as soon as we have found a solution. Thank you for your patience.

Best,
Fabrice

#7

Dear Fabrice,

thank you for an update and help.
Just a new observation (in case you test it only on your powerful computers)… It seems that the bug is more apparent when NRP is running on weaker computer (local docker NRP installation on my laptop). In that case, it did not happen only at the beginning.

Best,
Zdenek

#8

Dear Zdenek,

Your observation is in line with what we think the problem is. As I previously mentioned, this seems to be an issue with Gazebo itself. We’ll get to the bottom of that. Thanks for your patience.

Best,

Fabrice

#11

Dear Zdenek,
thank you very much for sending the example experiment, I can now replicate your issue. As Fabrice said, we will need some time to inspect the interface of Gazebo-ROS-NRP.

If you need a quick solution there is two workarounds:

  1. You can leave out the trigger of the transfer function. This way the Transfer Function will be executed every 20ms in sync with the NRP and Gazebo cycle time, that is one image per 20ms is saved.
  2. If this is not accurate enough for you, e.g. because the framerate of the camera does not exactly fit the NRP cycle time, you can also in this Transfer Function (no trigger, executed every 20ms) request a new image. While ROS topics send data continuously, you can also use a rosservice to request a fresh image. In this case you need to do a rosservice call /icub/image_raw_service “{}”.

Best Regards,
Benedikt

#12

Dear @BenediktFeldotto,
thank you very much for your help!
It seems that I need to use the second solution. As I’m a ROS beginner, could you please describe this solution in more details?

Thank you in advance.

Best,

Zdenek

#13

Dear @strakzd1,
I think I found the bug that is resulting in the multiple image recordings: Basically, whenever a Transfer Function is loaded or changed/enabled the Transfer Function runs a few times with the periodically executing Closed Loop Engine. I am implementing a fix for this bug.

The camera topic in the example you sent to me is publishing at a higher frequency than the CLE cycle time of 20ms. Therefore, in your case it should be sufficient to just leave out the camera trigger for now. This way the Transfer Function will be executed every 20ms and add this time retrieving the latest camera image. Saving this image to csv, you should be able to log camer images in periods of 20ms to your csv file for further processing.

The bug fix will be delivered with the next NRP release.

Best,
Benedikt

#14

Dear @BenediktFeldotto

nice to hear that you found the bug!
I tried to leave out the trigger, but new problem appeared. I obtained multiple instances of the same image which would not be alone a problem because I can detect which image has already appeared and remove/ignore it. But even though I removed duplicit images, the temporal difference (simulation time timestamps) between frames varied significantly. For example, image1 has timestamp 0.5s, image 2 has 0.65s, image3 has 0.72s, image4 has 0.87s. It means that temporal difference between image2 and image1 is 0.15s, but between image3 and image2 it is only 0.07s. As my task is closely related to next frame video prediction, it is quite important to have stable frame rate of the image data. Do you have any suggestion how to deal with that?

Thank you in advance.

Best,
Zdenek

#15

It seems that I had wrong setting of an update rate (only 12) in iCub’s camera in model.sdf file. When I change it to a higher value, it seems that the problem with the unstable frame rate is solved… exactly as you said :slight_smile:
@BenediktFeldotto thank you again. You really helped me a lot!

#16

Dear @strakzd1,
thanks for the update and glad you found a solution.
Yes, please make sure the camera update frequency in the model.sdf is higher than the CLE cycle time of 20ms.

Best,
Benedikt

#17

Dear @BenediktFeldotto,
thank you for your reply. I found that I cannot achieve higher camera fps than approximately 22. For example, if I set camera update rate in model.sdf to 100 (or 1000) I get 50 camera records (no trigger) in 1 second but only 22 are unique (again multiple records with same seq number). In the model.sdf, I set up update rate for both camera and camera controller. If I set update rate to lower value than 22 (e.g. 12 as I had before), camera fps decreases. Please, do you have any idea why it is not possible to achieve higher fps than 22? Is there something else what should be changed than the camera update rate in model.sdf? Thanks a lot.

#18

Dear @strakzd1,
could you please check the camera ros topic rate with ‘rostopic hz /topicname’ and let me know what is the highest you can achieve with setting the parameter in the model.sdf?
I suggest to only set the camera update rate in the model.sdf to the desired goal update rate and set the update rate of the ros controller in the .sdf to 0. Hereby, every new image from the gazebo camera will be handled and transmitted by the controller.

#19

Dear @BenediktFeldotto ,
Ok, I set the update rate of the ros controller in the .sdf to 0. When camera update rate was set to 100 I got usually something like this

average rate: 21.234
	min: 0.000s max: 0.064s std dev: 0.00806s window: 86

with update rate set to 1000 it slightly increased to

average rate: 24.365
	min: 0.003s max: 0.071s std dev: 0.01082s window: 118

Thank you for your help!

#20

Dear @strakzd1,
Unfortunately, Gazebo only performs a soft synchronization of sensors to the physics engine. This means the physics engine does not wait for all sensors to be executed and hereby in some configurations the frame rate defined in the sdf can not be reached.
To circumvent this issue, please add the following lines in your world.sdf (the one that is located in your storage directly)

<physics type='ode'> <max_step_size>0.001</max_step_size> <real_time_factor>1.0<real_time_factor> <real_time_update_rate>100.0</real_time_update_rate> </physics>

the physics tag shall be included right after the world definition:

<sdf version="1.6"> <world name="default"> <physics ....

Here, the one parameter that is of importance is the real_time_update rate. Its default value is 1000.0, here we decrease it to 100.0. Please test the camera update rate with this new configuration, by setting the desired fps in the model.sdf as before and check the update rate with rostopic hz. If you need even higher frequencies you can always decrease the real_time_update_rate further.
Gazebo introduced a hard synchronization of sensors in a newer release, which we will integrate in one of the next NRP releases.

#21

Dear @BenediktFeldotto ,
your solution works! I can achieve 50 fps with real_time_update_rate equal to 70. Thank you very much. I would not find the solution without your help!