Error in launching the NRP


#1

Hi,

I am trying to use different Nest and ROS version on the same laptop where I have the NRP installed.
I have a conditional statement in the .bashrc in order to set different path in case I am using the NRP or not.
Once installed, the NRP used to work, but after using again the original Nest and Ros version I could not run the platform anymore and the cle-start gives the following error:

detected binary path: /usr/bin/uwsgi-core
*** WARNING: you are running uWSGI without its master process manager ***
your processes number limit is 62830
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /home/hbp/.local/etc/nginx/nrp-services.sock fd 3
Python version: 2.7.6 (default, Oct 26 2016, 20:33:43)  [GCC 4.8.4]
Set PythonHome to /home/hbp/.opt/platform_venv
Python main interpreter initialized at 0x1a43640
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145088 bytes (141 KB) for 8 cores
*** Operational MODE: threaded ***
/home/hbp/Documents/NRP/ExDBackend/hbp-flask-restful-swagger-master/flask_restful_swagger/swagger.py:7: ExtDeprecationWarning: Importing flask.ext.restful is deprecated, use flask_restful instead.
  from flask.ext.restful import Resource, fields
/home/hbp/Documents/NRP/ExDBackend/hbp-flask-restful-swagger-master/flask_restful_swagger/swagger.py:7: ExtDeprecationWarning: Importing flask.ext.restful.fields is deprecated, use flask_restful.fields instead.
  from flask.ext.restful import Resource, fields
!!! uWSGI process 31239 got Segmentation Fault !!!

2018-01-25 12:54:40 GMT+0100 Failed to execute request http://localhost:8080/simulation. ERROR: Error: Status code: 502
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.4.6 (Ubuntu)</center>
</body>
</html>

Thank you! :smile:

Marie Claire


#2

Hi Marie-Claire,

A simple thing to try out before going any further would be

cle-nginx

#3

Hi,

thank you but I did that, unfortunately it does not solve the problem.


#4

Hi Marie-Claire,

I don’t know yet where the problem lies (the SEGFAULT of uWSGI may or may not be the reason of the subsequent 502 error). So, let’s try the obvious troubleshooting instructions first:

  cle-kill # even better if you can leave your terminal and open a new one
  bash $HBP/user-scripts/update_nrp update # be patient!
  bash $HBP/user-scripts/update_nrp build all #
  bash $HBP/user-scripts/configure_nrp # yes to all prompts
  cle-start

After this, we’ll have theoretically the same NRP installation and we will look into log files if need be.

Best regards,
Luc


#5

Hi Luc,

I tried these step before, and I tried again but it gives these error:

[hbp:22940] *** Process received signal ***
[hbp:22940] Signal: Segmentation fault (11)
[hbp:22940] Signal code: Address not mapped (1)
[hbp:22940] Failing at address: 0x44000098
[hbp:22940] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330) [0x7f2e64476330]
[hbp:22940] [ 1] /usr/lib/libmpi.so.1(PMPI_Comm_rank+0x4b) [0x7f2e4189963b]
[hbp:22940] [ 2] /home/hbp/.local/lib/python2.7/site-packages/nest/../../../../lib/libmusic.so.1(_ZN5MUSIC5Setup4initERiRPPc+0x3a) [0x7f2e4159f63a]
[hbp:22940] [ 3] /home/hbp/.local/lib/python2.7/site-packages/nest/../../../../lib/libmusic.so.1(_ZN5MUSIC5SetupC2ERiRPPciPi+0x14c) [0x7f2e415a0a4c]
[hbp:22940] [ 4] /home/hbp/.local/lib/python2.7/site-packages/nest/../../../libnestkernel.so(_ZN4nest12MUSICManager10init_musicEPiPPPc+0x38) [0x7f2e42881fc8]
[hbp:22940] [ 5] /home/hbp/.local/lib/python2.7/site-packages/nest/../../../libnestkernel.so(_ZN4nest10MPIManager8init_mpiEPiPPPc+0x112) [0x7f2e428906a2]
[hbp:22940] [ 6] /home/hbp/.local/lib/python2.7/site-packages/nest/../../../libnestkernel.so(_ZN4nest9init_nestEPiPPPc+0x2d) [0x7f2e428844bd]
[hbp:22940] [ 7] /home/hbp/.local/lib/python2.7/site-packages/nest/../../../libnest.so(_Z11neststartupPiPPPcR14SLIInterpreterSs+0x1d) [0x7f2e434f0a7d]
[hbp:22940] [ 8] /home/hbp/.local/lib/python2.7/site-packages/nest/pynestkernel.so(+0x2bd29) [0x7f2e43733d29]
[hbp:22940] [ 9] python(PyEval_EvalFrameEx+0x41d) [0x523f6d]
[hbp:22940] [10] python(PyEval_EvalFrameEx+0xc9a) [0x5247ea]
[hbp:22940] [11] python(PyEval_EvalCodeEx+0x2b1) [0x555551]
[hbp:22940] [12] python(PyEval_EvalCode+0x32) [0x5b41e2]
[hbp:22940] [13] python(PyImport_ExecCodeModuleEx+0xaa) [0x5b429a]
[hbp:22940] [14] python() [0x5942af]
[hbp:22940] [15] python() [0x465804]
[hbp:22940] [16] python() [0x55642f]
[hbp:22940] [17] python() [0x556943]
[hbp:22940] [18] python() [0x556c4b]
[hbp:22940] [19] python() [0x569cd8]
[hbp:22940] [20] python(PyEval_CallObjectWithKeywords+0x6b) [0x4c8c8b]
[hbp:22940] [21] python(PyEval_EvalFrameEx+0x2958) [0x5264a8]
[hbp:22940] [22] python(PyEval_EvalCodeEx+0x2b1) [0x555551]
[hbp:22940] [23] python(PyEval_EvalCode+0x32) [0x5b41e2]
[hbp:22940] [24] python(PyImport_ExecCodeModuleEx+0xaa) [0x5b429a]
[hbp:22940] [25] python() [0x5942af]
[hbp:22940] [26] python() [0x465804]
[hbp:22940] [27] python() [0x55642f]
[hbp:22940] [28] python() [0x556838]
[hbp:22940] [29] python() [0x5570ae]
[hbp:22940] *** End of error message ***
/bin/bash: line 11: 22940 Segmentation fault      (core dumped) python $HBP/ExDBackend/hbp_nrp_cleserver/hbp_nrp_cleserver/server/ROSCLESimulationFactory.py
[4]+  Exit 139                PYTHONPATH=$PYTHONPATH:$HOME/.opt/platform_venv/lib/python2.7/site-packages && python $HBP/ExDBackend/hbp_nrp_cleserver/hbp_nrp_cleserver/server/ROSCLESimulationFactory.py
[4] 22964
[uWSGI] getting INI configuration from /home/hbp/.local/etc/nginx/uwsgi-nrp.ini
*** Starting uWSGI 1.9.17.1-debian (64bit) on [Thu Jan 25 15:19:52 2018] ***
compiled with version: 4.8.2 on 23 March 2014 17:15:32
os: Linux-4.4.0-87-generic #110~14.04.1-Ubuntu SMP Tue Jul 18 14:51:32 UTC 2017
nodename: hbp
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 8
current working directory: /home/hbp
detected binary path: /usr/bin/uwsgi-core
*** WARNING: you are running uWSGI without its master process manager ***
your processes number limit is 62830
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /home/hbp/.local/etc/nginx/nrp-services.sock fd 3
Python version: 2.7.6 (default, Oct 26 2016, 20:33:43)  [GCC 4.8.4]
Set PythonHome to /home/hbp/.opt/platform_venv
Python main interpreter initialized at 0x1ac7750
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145088 bytes (141 KB) for 8 cores
*** Operational MODE: threaded ***
/home/hbp/Documents/NRP/ExDBackend/hbp-flask-restful-swagger-master/flask_restful_swagger/swagger.py:7: ExtDeprecationWarning: Importing flask.ext.restful is deprecated, use flask_restful instead.
  from flask.ext.restful import Resource, fields
/home/hbp/Documents/NRP/ExDBackend/hbp-flask-restful-swagger-master/flask_restful_swagger/swagger.py:7: ExtDeprecationWarning: Importing flask.ext.restful.fields is deprecated, use flask_restful.fields instead.
  from flask.ext.restful import Resource, fields
!!! uWSGI process 22966 got Segmentation Fault !!!
*** backtrace of 22966 ***
uwsgi(uwsgi_backtrace+0x2e) [0x45121e]
uwsgi(uwsgi_segfault+0x21) [0x4515f1]
/lib/x86_64-linux-gnu/libc.so.6(+0x36cb0) [0x7f3b3ed69cb0]
/usr/lib/libmpi.so.1(PMPI_Comm_rank+0x4b) [0x7f3b2070163b]
/home/hbp/.local/lib/python2.7/site-packages/nest/../../../../lib/libmusic.so.1(_ZN5MUSIC5Setup4initERiRPPc+0x3a) [0x7f3b2040763a]
/home/hbp/.local/lib/python2.7/site-packages/nest/../../../../lib/libmusic.so.1(_ZN5MUSIC5SetupC2ERiRPPciPi+0x14c) [0x7f3b20408a4c]
/home/hbp/.local/lib/python2.7/site-packages/nest/../../../libnestkernel.so(_ZN4nest12MUSICManager10init_musicEPiPPPc+0x38) [0x7f3b21719fc8]
/home/hbp/.local/lib/python2.7/site-packages/nest/../../../libnestkernel.so(_ZN4nest10MPIManager8init_mpiEPiPPPc+0x112) [0x7f3b217286a2]
/home/hbp/.local/lib/python2.7/site-packages/nest/../../../libnestkernel.so(_ZN4nest9init_nestEPiPPPc+0x2d) [0x7f3b2171c4bd]
/home/hbp/.local/lib/python2.7/site-packages/nest/../../../libnest.so(_Z11neststartupPiPPPcR14SLIInterpreterSs+0x1d) [0x7f3b22388a7d]
/home/hbp/.local/lib/python2.7/site-packages/nest/pynestkernel.so(+0x2bd29) [0x7f3b225cbd29]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4bd4) [0x7f3b3db55b94]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4b59) [0x7f3b3db55b19]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) [0x7f3b3db57c3d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCode+0x32) [0x7f3b3db57f22]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyImport_ExecCodeModuleEx+0x8c) [0x7f3b3db5877c]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1a5ef6) [0x7f3b3dbbbef6]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1d79a9) [0x7f3b3dbed9a9]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1a6609) [0x7f3b3dbbc609]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyImport_ImportModuleLevel+0x647) [0x7f3b3dbb5c47]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x17385f) [0x7f3b3db8985f]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f3b3db2c0a3]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7f3b3dbbf5f7]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2713) [0x7f3b3db536d3]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) [0x7f3b3db57c3d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCode+0x32) [0x7f3b3db57f22]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyImport_ExecCodeModuleEx+0x8c) [0x7f3b3db5877c]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1a5ef6) [0x7f3b3dbbbef6]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1d79a9) [0x7f3b3dbed9a9]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1a6609) [0x7f3b3dbbc609]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyImport_ImportModuleLevel+0x282) [0x7f3b3dbb5882]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x17385f) [0x7f3b3db8985f]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f3b3db2c0a3]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7f3b3dbbf5f7]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2713) [0x7f3b3db536d3]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) [0x7f3b3db57c3d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCode+0x32) [0x7f3b3db57f22]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyImport_ExecCodeModuleEx+0x8c) [0x7f3b3db5877c]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1a5ef6) [0x7f3b3dbbbef6]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1d79a9) [0x7f3b3dbed9a9]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1a6609) [0x7f3b3dbbc609]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyImport_ImportModuleLevel+0x282) [0x7f3b3dbb5882]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x17385f) [0x7f3b3db8985f]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f3b3db2c0a3]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7f3b3dbbf5f7]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2713) [0x7f3b3db536d3]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) [0x7f3b3db57c3d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCode+0x32) [0x7f3b3db57f22]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyImport_ExecCodeModuleEx+0x8c) [0x7f3b3db5877c]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1a5ef6) [0x7f3b3dbbbef6]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1d79a9) [0x7f3b3dbed9a9]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1a6609) [0x7f3b3dbbc609]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyImport_ImportModuleLevel+0x282) [0x7f3b3dbb5882]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x17385f) [0x7f3b3db8985f]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f3b3db2c0a3]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7f3b3dbbf5f7]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2713) [0x7f3b3db536d3]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) [0x7f3b3db57c3d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCode+0x32) [0x7f3b3db57f22]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyImport_ExecCodeModuleEx+0x8c) [0x7f3b3db5877c]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1a5ef6) [0x7f3b3dbbbef6]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1a6609) [0x7f3b3dbbc609]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xa647f) [0x7f3b3dabc47f]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyImport_ImportModuleLevel+0x446) [0x7f3b3dbb5a46]
*** end of backtrace ***
[4]+  Exit 1                  PYTHONPATH=$PYTHONPATH:$HOME/.opt/platform_venv/lib/python2.7/site-packages && uwsgi --ini $HOME/.local/etc/nginx/uwsgi-nrp.ini
[4] 22979
2018-01-25 15:19:55 GMT+0100 Polling Backend Servers for Experiments, Health & Running Simulations every 5000 ms.
2018-01-25 15:19:55 GMT+0100 Listening on port: 8000
2018-01-25 15:19:55 GMT+0100 Failed to execute request http://localhost:8080/health/errors. ERROR: Error: Status code: 502
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.4.6 (Ubuntu)</center>
</body>
</html>

2018-01-25 15:19:55 GMT+0100 Failed to execute request http://localhost:8080/simulation. ERROR: Error: Status code: 502
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.4.6 (Ubuntu)</center>
</body>
</html>

The main problem could be the conflict between the nest-simulator and/or MUSIC I have on my laptop.
Thank you for the help,
Marie


#6

To check this very likely hypothesis, you could do the following:

rm -rf $HBP/MUSIC
rm -rf $HBP/nest-simulator
cd $HBP
git clone ssh://$USER@bbpcode.epfl.ch/neurorobotics/MUSIC
git clone ssh://$USER@bbpcode.epfl.ch/neurorobotics/nest-simulator
bash $HBP/update_nrp build all
# Close and re-open terminal
cle-start