NRP Frontend and Backend on different docker containers causing problems on macOS


#1

Hello,

i recently installed the NRP with docker on macOS. The installation went well, but when I open the platform at localhost:9000/#/esv-private, I get a blank page. Opening the dev console the only error that I get is:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://172.19.0.2:9000/proxy/identity/me. (Reason: CORS request did not succeed).[Learn More]

I then look through the forums for some help, but when I try to run cle-kill, cle-start or cle-nginx the commands don’t work. So there must be something wrong with the nginx configuration. When I connect to the backend (using the command ./nrp_installer connect_backend), this is the output I get when I check the status from nginx (this is from the troubleshooting section of the source install of the NRP):

bbpnrsoa@b553ca750f4d:~/nrp/src$ nginx -t -c $HOME/.local/etc/nginx/nginx.conf
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2018/11/14 14:53:48 [warn] 9434#9434: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /home/bbpnrsoa/.local/etc/nginx/nginx.conf:1
2018/11/14 14:53:48 [info] 9434#9434: Using 32768KiB of shared memory for push module in /home/bbpnrsoa/.local/etc/nginx/nginx.conf:34
nginx: the configuration file /home/bbpnrsoa/.local/etc/nginx/nginx.conf syntax is ok
2018/11/14 14:53:48 [emerg] 9434#9434: open() "/home/bbpnrsoa/nginx/nginx.pid" failed (13: Permission denied)
nginx: configuration file /home/bbpnrsoa/.local/etc/nginx/nginx.conf test failed

So there is clearly a problem with the nginx configuration, but I don’t know how to fix it. I know there are a couple guys that I am working with that have the same problem. Thanks in advance for the help.

Edit: This post was hidden, because someone thought this was spam. I changed the title because I found another post with similar title but with a different problem.


(v2.1.2) Blank http://localhost:9000/#/esv-private after fresh local install
#2

Hello Juan,

One thing that might help, before digging a bit deeper, is to kill the nginx process and then run cle-nginx.
Best regards,
Luc


#3

After killing all nginx processes, cle-nginx works fine. Or at least appears to be. After that I ran cle-start and I got these output.

[1] 3701
... logging to /home/bbpnrsoa/.ros/log/1e034592-e861-11e8-bf47-0242ac130003/roslaunch-b553ca750f4d-3701.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://172.19.0.3:36283/
ros_comm version 1.12.14


SUMMARY
========

PARAMETERS
 * /rosdistro: kinetic
 * /rosversion: 1.12.14

NODES

roscore cannot run as another roscore/master is already running.
Please kill other roscore/master processes before relaunching.
The ROS_MASTER_URI is http://172.19.0.3:11311/
The traceback for the exception was written to the log file
[1]+  Exit 1                  roscore
[1] 3713
[ INFO] [1542237055.571243053]: Waiting For connections on 0.0.0.0:8081
[2] 3749
[INFO] [2018.11.14 23:10:57 /home/bbpnrsoa/nrp/src/nest-simulator/nestkernel/rng_manager.cpp:226 @ Network::create_rngs_] : Creating default RNGs
[INFO] [2018.11.14 23:10:57 /home/bbpnrsoa/nrp/src/nest-simulator/nestkernel/rng_manager.cpp:221 @ Network::create_rngs_] : Deleting existing random number generators
[INFO] [2018.11.14 23:10:57 /home/bbpnrsoa/nrp/src/nest-simulator/nestkernel/rng_manager.cpp:226 @ Network::create_rngs_] : Creating default RNGs
[INFO] [2018.11.14 23:10:57 /home/bbpnrsoa/nrp/src/nest-simulator/nestkernel/rng_manager.cpp:272 @ Network::create_grng_] : Creating new default global RNG

              -- N E S T --

  Copyright (C) 2004 The NEST Initiative
  Version 2.12.0 Oct 12 2018 13:27:56

This program is provided AS IS and comes with
NO WARRANTY. See the file LICENSE for details.

Problems or suggestions?
  Visit http://www.nest-simulator.org

Type 'nest.help()' to find out more about NEST.
[3] 3757
[uWSGI] getting INI configuration from /home/bbpnrsoa/.local/etc/nginx/uwsgi-nrp.ini
*** Starting uWSGI 2.0.12-debian (64bit) on [Wed Nov 14 23:10:57 2018] ***
compiled with version: 5.4.0 20160609 on 28 September 2018 15:49:44
os: Linux-4.9.93-linuxkit-aufs #1 SMP Wed Jun 6 16:55:56 UTC 2018
nodename: b553ca750f4d
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 4
current working directory: /home/bbpnrsoa/nrp/src
detected binary path: /usr/bin/uwsgi-core
chdir() to /home/bbpnrsoa/nrp/src/ExDBackend/hbp_nrp_backend/
*** WARNING: you are running uWSGI without its master process manager ***
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /home/bbpnrsoa/.local/etc/nginx/nrp-services.sock fd 3
Python version: 2.7.12 (default, Dec  4 2017, 14:50:18)  [GCC 5.4.0 20160609]
Set PythonHome to /home/bbpnrsoa/.opt/platform_venv
Python main interpreter initialized at 0x23f88f0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 144896 bytes (141 KB) for 8 cores
*** Operational MODE: threaded ***
CSAConnector: libneurosim support not available in NEST.
Falling back on PyNN's default CSAConnector.
Please re-compile NEST using --with-libneurosim=PATH
2018-11-14 23:10:58,085 [MainThread  ] [hbp_nrp_cles] [WARNING]  Could not write to specified logfile or no logfile specified, logging to stdout now!
[INFO] [2018.11.14 23:10:58 /home/bbpnrsoa/nrp/src/nest-simulator/nestkernel/rng_manager.cpp:226 @ Network::create_rngs_] : Creating default RNGs
[INFO] [2018.11.14 23:10:58 /home/bbpnrsoa/nrp/src/nest-simulator/nestkernel/rng_manager.cpp:221 @ Network::create_rngs_] : Deleting existing random number generators
[INFO] [2018.11.14 23:10:58 /home/bbpnrsoa/nrp/src/nest-simulator/nestkernel/rng_manager.cpp:226 @ Network::create_rngs_] : Creating default RNGs
[INFO] [2018.11.14 23:10:58 /home/bbpnrsoa/nrp/src/nest-simulator/nestkernel/rng_manager.cpp:272 @ Network::create_grng_] : Creating new default global RNG

              -- N E S T --

  Copyright (C) 2004 The NEST Initiative
  Version 2.12.0 Oct 12 2018 13:27:56

This program is provided AS IS and comes with
NO WARRANTY. See the file LICENSE for details.

Problems or suggestions?
  Visit http://www.nest-simulator.org

Type 'nest.help()' to find out more about NEST.
2018-11-14 23:10:58,301 [Thread-3    ] [rospy.intern] [INFO]  topic[/rosout] adding connection to [/rosout], count 0
[4] 3781
bash: line 1: cd: /home/bbpnrsoa/nrp/src/nrpBackendProxy: No such file or directory
registered capabilities (classes):
 - rosbridge_library.capabilities.call_service.CallService
 - rosbridge_library.capabilities.advertise.Advertise
 - rosbridge_library.capabilities.publish.Publish
 - rosbridge_library.capabilities.subscribe.Subscribe
 - <class 'rosbridge_library.capabilities.defragmentation.Defragment'>
 - rosbridge_library.capabilities.advertise_service.AdvertiseService
 - rosbridge_library.capabilities.service_response.ServiceResponse
 - rosbridge_library.capabilities.unadvertise_service.UnadvertiseService
CSAConnector: libneurosim support not available in NEST.
Falling back on PyNN's default CSAConnector.
Please re-compile NEST using --with-libneurosim=PATH
Traceback (most recent call last):
  File "/home/bbpnrsoa/nrp/src/GazeboRosPackages/src/rosbridge_suite-develop/rosbridge_server/scripts/rosbridge_websocket", line 155, in <module>
    application.listen(port, address)
  File "/home/bbpnrsoa/nrp/src/GazeboRosPackages/src/rosbridge_suite-develop/rosbridge_server/src/tornado/web.py", line 1699, in listen
    server.listen(port, address)
  File "/home/bbpnrsoa/nrp/src/GazeboRosPackages/src/rosbridge_suite-develop/rosbridge_server/src/tornado/tcpserver.py", line 129, in listen
    sockets = bind_sockets(port, address=address)
  File "/home/bbpnrsoa/nrp/src/GazeboRosPackages/src/rosbridge_suite-develop/rosbridge_server/src/tornado/netutil.py", line 149, in bind_sockets
    sock.bind(sockaddr)
  File "/usr/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 98] Address already in use
2018-11-14 23:10:59,342 [MainThread  ] [hbp_nrp_back] [WARNING]  Application started with uWSGI or any other framework. logging to console by default !
WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x23f88f0 pid: 3759 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 3759, cores: 8)
[4]+  Exit 1                  cd $HBP/nrpBackendProxy && node app.js

The address is still in use.


#4

Hello Juan,

You may need to run also

cle-kill

and then

cle-start

But there is something that worries me more:

bash: line 1: cd: /home/bbpnrsoa/nrp/src/nrpBackendProxy: No such file or directory

This suggests that one NRP component is missing from your Docker image. Did you encounter any Docker error during the installation process?

Best regards,
Luc


#5

Hello,
I am encountering exactly the same issue with the docker installation on macos.
During the docker installation process, there weren’t any errors.

Regarding the missing directory:

bash: line 1: cd: /home/bbpnrsoa/nrp/src/nrpBackendProxy: No such file or directory

This folder actually exists in the frontend container, but not in the backend container.

Best regards,
Olivier


#6

I run the commands again after killing all nginx processes and I got the same result. The directory doesn’t exist on the backend container, only on the frontend container. My guess is that they should be on the same container, but it does not explain why it works on windows machines and not on macOS.


#7

Hi,

I’m looking into this problem at the moment but meanwhile can you instead do the following commands to check if they fix the problem.
Do these on both containers (to connect to the frontend: ./nrp_installer.sh connect_frontend , to connect to the backend: ./nrp_installer.sh connect_backend):

sudo /etc/init.d/supervisor stop
cle-kill
sudo /etc/init.d/supervisor start

And then open in an incognito window http://localhost:9001/#/esv-private ?


#8

The cle-start and cle-nginx commands are only useful for full local installs and not for when using the docker containers.


#9

The commands run without a problem, but the page appeared empty on both firefox and chrome private windows, and did not send warnings on the dev console. I also tried on port 9000 in case it was a typo and got the same cross-origin warning with some extra warnings that I guess are caused by the incognito window. Here is a screenshot of the warnings:


#10

Hi Juan,

I’ve been investigating this the past few days (we don’t actually officially support Macs), anyway the problem seemed to be due to docker on Macs not supporting the same networking features (for more info see: https://docs.docker.com/docker-for-mac/networking/).

I have a fix in the works, it hasn’t been merged yet but if you want you can be my guinea pig to see if it fixes your problems.

  1. Firstly you will need to have a docker version >= 16.03.

  2. Run the following command:

    echo “127.0.0.1 host.docker.internal” | sudo tee -a /etc/hosts

  3. Download the following file, and rename it to be called nrp_installer.sh (this forum wouldnt allow me to upload a bash script with the proper name). nrp_installer.txt (14.7 KB)

  4. Run the following commands:

    chmod 755 nrp_installer.sh
    ./nrp_installer.sh reset

And then open your browser and go to http://localhost:9000/#/esv-private (If you are still seeing CORS errors, please disable any adblockers etc you have on your browser, or check that your browser is not rejecting Cross-Origin requests, you might need to install an add-on e.g. : https://chrome.google.com/webstore/detail/allow-control-allow-origi/nlfbmbojpeacfghkpbjhddihlkkiljbi , this however shouldn’t be necessary, if it is, please let us know the browser and version you are running)

Let me know if this helps with your problem.


#11

Hello,

I tried the instructions above and it didn’t work, but the IP address did change, here is a screenshot from firefox with no adblocker:

after that I installed Google Chrome, and this is the console output without using the extension that you recommended:

My docker engine version is 18.09. So I guess it is good that the IP changed but it still cannot reach the backend.
I know that macs are not normally supported so thanks for the extra help.


#12

Your problem with firefox seems to be that the browser is blocking our requests. Can you tell me what version of Firefox and Chrome you are using?

The problem with chrome seems to be more a problem with our containers. But i can’t seem to reproduce it on my own machine. Could you try the following command on your terminal ? (if the result is some html, then it everything looks ok):

curl -XGET host.docker.internal:9000/

Assuming that wont return an error. These two commands will leave you will two files, proxy.out and proxy.err:

docker cp frontend:/var/log/supervisor/proxy/proxy.out .
docker cp frontend:/var/log/supervisor/proxy/proxy.err .

Could you then attach these two files here?


#13

Firefox: 63.0.3
Chrome: 70.0.3538.110

The curl command doesn’t give any output, I have to cancel the command manually or it runs indefinitely. Here are the files that you requested. The error file looks like its looking for files on the backend, the other one is parsing all the experiments.
proxy.err.txt (964 Bytes) proxy.out.txt (16.4 KB)


#14

Thanks Juan. The proxy logs look fine, and since the curl fails, it seems its a networking issue. Could you check if this command also just hangs?

curl -XGET 127.0.0.1:9000/

Could you then paste the contents of your /etc/hosts file here please?

And also paste the output of this command too:

docker ps

In terms of your firefox issue, we can look into that once we have the networking correct, as that is actually a separate issue.


#15

Hey, it worked, my mac wasn’t mapping the host.docker.internal address properly, it worked after a restart. After that I ran both commands and got this html:

➜ curl -XGET host.docker.internal:9000
<!doctype html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="description" content=""> <meta name="viewport" content="width=device-width,initial-scale=1,user-scalable=no"> <title>Neurorobotics platform</title> <link rel="stylesheet" href="node_modules/n3-charts/build/LineChart.min.c7ddbc16.css"> <link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons"> <link rel="stylesheet" href="styles/vendor.7dcde6d2.css"> <link rel="stylesheet" href="styles/main.525bb1db.css"> </head> <body class="ng-cloak"> <!-- Google analytics --> <script> // Get analytics only for production/staging/development servers
        var hostname = document.location.hostname;
        (function(i, s, o, g, r, a, m) {
            i['GoogleAnalyticsObject'] = r;
            i[r] = i[r] || function() {
                (i[r].q = i[r].q || []).push(arguments)
            }, i[r].l = 1 * new Date();
            a = s.createElement(o),
                m = s.getElementsByTagName(o)[0];
            a.async = 1;
            a.src = g;
            m.parentNode.insertBefore(a, m)
        })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga'); </script> <browser-support-alert></browser-support-alert> <cluster-reservation-alert></cluster-reservation-alert> <demo-carousel-alert></demo-carousel-alert> <!-- Add your site or application content here --> <div ui-view></div> <changelog-widget></changelog-widget> <script src="scripts/vendor.a271ae9d.js"></script> <script src="node_modules/d3/d3.min.5bc24506.js"></script> <script src="node_modules/n3-charts/build/LineChart.min.8f6c136d.js"></script> <script src="node_modules/rxjs/bundles/Rx.min.72f38afc.js"></script> <script src="node_modules/chart.js/dist/Chart.min.c8fbc7d4.js"></script> <script src="scripts/app.f97d4d2c.js"></script>

I did not do anything extra with the settings or /etc/hosts files. So It would be good if in the instructions you mention that a restart my be necessary. Now the webpage is also working, I can see the experiment overview. I will try the tutorial experiment later in case I find further problems. It work on both firefox and chrome.

Just for completion, here is the output of docker ps

CONTAINER ID        IMAGE                               COMMAND             CREATED             STATUS              PORTS                    NAMES
fd098d88bdc4        hbpneurorobotics/nrp_frontend:dev   "/bin/bash"         47 hours ago        Up 8 minutes        0.0.0.0:9000->9000/tcp   frontend
4b72ccfb49cd        hbpneurorobotics/nrp:dev            "/bin/bash"         47 hours ago        Up 8 minutes        0.0.0.0:8080->8080/tcp   nrp

And my /etc/hosts file:

127.0.0.1	localhost
255.255.255.255	broadcasthost
::1             localhost
127.0.0.1 host.docker.internal

Thanks a lot for the help. Hope this helps some else :smile:


#16

Thats great Juan :grinning:. I’m glad its working now but could you tell me what do you mean by a reset?


#17

I meant a reboot/restart. Turn my mac on and off. I will edit my previous post.


#18

Thanks Juan. Thats the new script with Mac support deployed on our website.