Nginx error (111: Connection refused)


#1

Hi,
I’m new to the platform. I’ve installed NRP under ubuntu 16.04 . The build was successful after resolving some initial python path and GazeboRosPackages issues. But when I run cle-start, it’s stuck forever loading the experiment list. So I checked the nginx error-log and got…

[error] 5978#5978: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: 127.0.0.1, request: "GET /proxy/storage/experiments HTTP/1.1", upstream: "http://127.0.0.1:8443/storage/experiments", host: "localhost:9000", referrer: "http://localhost:9000/"

And I’m able to access http://localhost:9000/#/esv-web but as expected the experiments are not loading.

I also did ./update_nrp update all and ./configure_nrp sucessfully, and also did nginx: sudo killall and service restart to no avail.

And also did netstat -tulnp | grep 8080 and it seems to be listening on the right port…

tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      5977/nginx 

Need guidance on how to resolve this.

Thanks!
Shruthi


#2

Dear Shruthi,

The experiment list is gotten from a component called “proxy”. It runs as a node application and on port 8443. Check that this port is available on your system.
Then do the following test:

  • kill all NRP services with “cle-kill”
  • be sure to start nginx with “cle-nginx”
  • start the proxy alone with “cle-proxy”
    You should get something like:
2019-01-02 13:38:47 GMT+0100 Polling Backend Servers for Experiments, Health & Running Simulations every 5000 ms.
2019-01-02 13:38:47 GMT+0100 Listening on port: 8443
2019-01-02 13:38:47 GMT+0100 Parsing experiment file /home/vonarnim/Documents/NRP/Experiments/benchmark_p3dx/BenchmarkPioneer.exc
2019-01-02 13:38:47 GMT+0100 Parsing experiment file /home/vonarnim/Documents/NRP/Experiments/braitenberg_husky_distributed_music/ExDDistributedBrainHuskyMUSIC.exc
2019-01-02 13:38:47 GMT+0100 Parsing experiment file /home/vonarnim/Documents/NRP/Experiments/braitenberg_husky_distributed/ExDDistributedBrainHusky.exc
2019-01-02 13:38:47 GMT+0100 Parsing experiment file /home/vonarnim/Documents/NRP/Experiments/braitenberg_husky_holodeck_csv_recorders/ExDXMLExample_holodeck_csv_recorders.exc
2019-01-02 13:38:47 GMT+0100 Parsing experiment file /home/vonarnim/Documents/NRP/Experiments/braitenberg_husky_holodeck_csv_recorders/ExDXMLExample_holodeck.exc
2019-01-02 13:38:47 GMT+0100 Parsing experiment file /home/vonarnim/Documents/NRP/Experiments/braitenberg_husky_holodeck/ExDXMLExample_holodeck.exc
...

If not, eiher your port 8443 is not available for some reason, or node.js fails to start (check with “ps”).

Best,
Axel


#3

Hi Axel,
Thanks for the reply. I ran netstat and got the following, the port seems to be working fine…

$ netstat -tulnp | grep 8443
tcp6       0      0 :::8443                 :::*                    LISTEN      4149/node       

But running cle-proxy gave me the following error…

2019-01-04 10:54:17 EST Polling Backend Servers for Experiments, Health & Running Simulations every 5000 ms.
2019-01-04 10:54:17 EST Listening on port: 8443

2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/benchmark_p3dx/BenchmarkPioneer.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/braitenberg_husky_distributed_music/ExDDistributedBrainHuskyMUSIC.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/braitenberg_husky_holodeck/ExDXMLExample_holodeck.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/braitenberg_husky_sbc/ExDBraitenbergHuskySBC.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/braitenberg_husky_distributed/ExDDistributedBrainHusky.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/braitenberg_husky_spinnaker/HuskySpinnaker.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/braitenberg_husky/ExDXMLExample.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/braitenberg_lauron_sbc/ExDBraitenbergLauronSBC.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/braitenberg_lauron/ExDBraitenbergLauron.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/braitenberg_mouse_holodeck_lab/ExDBraitenbergMouseHoloLab.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/braitenberg_mouse_lab/ExDBraitenbergMouseLab.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/braitenberg_mouse/ExDBraitenbergMouse.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/cdp1_mouse_holodeck/ExDMouseCDP1_holodeck.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/cdp1_mouse_tactile/ExDMouseCDP1_tactile.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/cdp1_mouse/ExDMouseCDP1.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/demo_dvs_icub/ExDDvsIcub.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/demo_husky_sbc/ExDDemoHuskySBC.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/demo_mouse_lab/ExDMouseLabExample.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/demo_manipulation/ExDDemoManipulation.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/dvs_robot_head/ExDDvsRobotHead.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/docked_mouse/ExDDockedMouse.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/interactive_manipulation/ExDInteractiveManipulation.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/manipulation_force_holodeck/ExDManipulationForce_holodeck.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/manipulation_force/ExDManipulationForce.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/manipulation_holodeck/ExDManipulation_holodeck.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/manipulation_RL/ExDManipulationRL.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/manipulation/ExDManipulation.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/multiple_robots_tutorial/MultipleRobots.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/myoarm_nst/Myo_NST.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/myoarm_small/Myo_small.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/nao/ExDNao.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/neuronal_red_detection_holodeck/NeuronalRedDetection_Husky_holodeck.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/neuronal_red_detection/NeuronalRedDetection_Husky.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/retina_icub/ExDRetinaICubTrackingRG.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/retina_mockup/ExDRetinaICubMockup.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/roboy/roboy.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/screen_switching_holodeck/ScreenSwitchingHuskyExperiment_holodeck.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/spinnaker_async/ExDSpiNNakerAsynchronous.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/screen_switching/ScreenSwitchingHuskyExperiment.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/spinnaker_example/ExDSpiNNakerExample.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/spinnaker_mvp_I/spinnaker_mvp_i.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/spinnaker_mvp_II/spinnaker_mvp_ii.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/template_husky_timeout/ExDTemplateHuskyTimeout.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/template_husky/ExDTemplateHusky.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/template_icub_holodeck/ExDTemplateICubHolodeck.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/template_icub/ExDTemplateICub.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/template_lauron_docked/ExDTemplateDockedLauron.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/template_lauron/ExDTemplateLauron.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/template_manipulation/ExDTemplateManipulation.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/template_mmm/ExDTemplateMMM.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/template_mouse/ExDTemplateMouseV2.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/template_new/TemplateNew.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/template_pioneer/ExDTemplatePioneer.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/tigrillo-cl-ann-learning/ExDTigrilloANNFORCElight.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/tigrillo-cl-replay/ExDTigrillo_Replay.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/tigrillo-cl-snn-learning/ExDTigrillo.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/tigrillo-tf-cpg/experiment_config.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/tutorial_baseball_exercise/ExDTutorialBaseballExercise.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/tutorial_baseball_solution/ExDTutorialBaseballSolution.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/tutorial_debugging/tutorial_debugging.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/tutorial_tensorflow_husky/TensorFlowTutorial.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/user-avatar_test-environment/user-avatar_test-env.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/visual_tracking_icub_holodeck/ExDVisualTrackingICub_holodeck.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/visual_tracking_icub/ExDVisualTrackingICub.exc
2019-01-04 10:54:17 EST Parsing experiment file /home/shruthi/Documents/NRP/Experiments/zip_robot/ExDXMLExampleRobotZip.exc
2019-01-04 10:54:17 EST Failed to execute request http://localhost:8080/health/errors. ERROR: Error: Status code: 502
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.10.3 (Ubuntu)</center>
</body>
</html>

2019-01-04 10:54:17 EST Failed to execute request http://localhost:8080/simulation. ERROR: Error: Status code: 502
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.10.3 (Ubuntu)</center>
</body>
</html>

2019-01-04 10:54:22 EST Failed to execute request http://localhost:8080/health/errors. ERROR: Error: Status code: 502
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.10.3 (Ubuntu)</center>
</body>
</html>

2019-01-04 10:54:22 EST Failed to execute request http://localhost:8080/simulation. ERROR: Error: Status code: 502
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.10.3 (Ubuntu)</center>
</body>
</html>

2019-01-04 10:54:27 EST Failed to execute request http://localhost:8080/health/errors. ERROR: Error: Status code: 502
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.10.3 (Ubuntu)</center>
</body>
</html>

2019-01-04 10:54:27 EST Failed to execute request http://localhost:8080/simulation. ERROR: Error: Status code: 502
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.10.3 (Ubuntu)</center>
</body>
</html>

What should I do now?

Best,
Shruthi


#4

Might be that your nginx is not running. Nginx is providing this 8080 port.
Run cle-nginx again and check if it is running (ps). Check for errors in your $HOME/nginx.
Check nginx config in $HOME/.local/etc/nginx/conf.d/nrp-services.conf. It should define a 8080 port.
If this file is non existing, then something went wrong in the execution of $HOME/user-scripts/configure_nrp.

Axel


#5

nginx seems to be running on the right port…

$ netstat -tulnp | grep 8080
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      5729/nginx  

And no errors are now logged in $HOME/nginx…

And $HOME/.local/etc/nginx/conf.d/nrp-services.conf file exists and this is what I found in it…

server {
        listen *:8080 default_server;
        server_name 127.0.0.1 localhost;

        # UPDATE to match your local Experiments folder
        set $simulation_experiments_folder /home/shruthi/Documents/NRP/Experiments; 
        ...
        ...

But running cle-backend gave me the error “Master node unable to register”…

[uWSGI] getting INI configuration from /home/shruthi/.local/etc/nginx/uwsgi-nrp.ini
*** Starting uWSGI 2.0.12-debian (64bit) on [Fri Jan  4 12:49:51 2019] ***
compiled with version: 5.4.0 20160609 on 28 September 2018 15:49:44
os: Linux-4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016
nodename: shruthi-Lenovo-YOGA-900-13ISK
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 4
current working directory: /home/shruthi/Documents/NRP
detected binary path: /usr/bin/uwsgi-core
chdir() to /home/shruthi/Documents/NRP/ExDBackend/hbp_nrp_backend/
*** WARNING: you are running uWSGI without its master process manager ***
your processes number limit is 30679
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /tmp/nrp-services.sock fd 3
Python version: 2.7.12 (default, Dec  4 2017, 14:50:18)  [GCC 5.4.0 20160609]
Set PythonHome to /home/shruthi/.opt/platform_venv
Python main interpreter initialized at 0x1f795f0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 144896 bytes (141 KB) for 8 cores
*** Operational MODE: threaded ***
[INFO] [2019.1.4 12:49:51 /home/shruthi/Documents/NRP/nest-simulator/nestkernel/rng_manager.cpp:226 @ Network::create_rngs_] : Creating default RNGs
[INFO] [2019.1.4 12:49:51 /home/shruthi/Documents/NRP/nest-simulator/nestkernel/rng_manager.cpp:221 @ Network::create_rngs_] : Deleting existing random number generators
[INFO] [2019.1.4 12:49:51 /home/shruthi/Documents/NRP/nest-simulator/nestkernel/rng_manager.cpp:226 @ Network::create_rngs_] : Creating default RNGs
[INFO] [2019.1.4 12:49:51 /home/shruthi/Documents/NRP/nest-simulator/nestkernel/rng_manager.cpp:272 @ Network::create_grng_] : Creating new default global RNG

              -- N E S T --

  Copyright (C) 2004 The NEST Initiative
  Version 2.12.0 Nov  8 2018 15:21:32

This program is provided AS IS and comes with
NO WARRANTY. See the file LICENSE for details.

Problems or suggestions?
  Visit http://www.nest-simulator.org

Type 'nest.help()' to find out more about NEST.
CSAConnector: libneurosim support not available in NEST.
Falling back on PyNN's default CSAConnector.
Please re-compile NEST using --with-libneurosim=PATH
[ERROR] [1546624192.210013]: Unable to immediately register with master node [http://localhost:11311]: master may not be running yet. Will keep trying.
^CTraceback (most recent call last):
  File "/home/shruthi/Documents/NRP/ExDBackend/hbp_nrp_backend/hbp_nrp_backend/runserver.py", line 151, in <module>
    start_ros()
  File "/home/shruthi/Documents/NRP/ExDBackend/hbp_nrp_backend/hbp_nrp_backend/runserver.py", line 62, in start_ros
    rospy.init_node("nrp_backend")
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/client.py", line 323, in init_node
    raise rospy.exceptions.ROSInitException("init_node interrupted before it could complete")
rospy.exceptions.ROSInitException: init_node interrupted before it could complete
unable to load app 0 (mountpoint='') (callable not found or import error)
*** no app loaded. going in full dynamic mode ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 5873, cores: 8)
^C!!! uWSGI process 5873 got Segmentation Fault !!!
Segmentation fault (core dumped)

Best,
Shruthi