[SOLVED] Error after fresh install from source

[pkrack]

Hello,
I did a fresh install of the NRP by following the instructions on bitbucket.
The NRP was installed on a fresh install of ubuntu 18.04 in a VM (qemu with kvm).

The problem is that the netifaces package is not found:
Traceback (most recent call last):
File “/home/pkrack/Documents/NRP/ExDBackend/hbp_nrp_cleserver/hbp_nrp_cleserver/server/ROSCLESimulationFactory.py”, line 41, in
from hbp_nrp_cleserver.server import ROS_CLE_NODE_NAME, SERVICE_CREATE_NEW_SIMULATION,
File “/home/pkrack/Documents/NRP/ExDBackend/hbp_nrp_cleserver/hbp_nrp_cleserver/server/init.py”, line 31, in
from hbp_nrp_cle import config
File “/home/pkrack/Documents/NRP/CLE/hbp_nrp_cle/hbp_nrp_cle/config.py”, line 38, in
import netifaces
ImportError: No module named netifaces
[3]+ Exit 1 $HOME/.opt/platform_venv/bin/python $HBP/ExDBackend/hbp_nrp_cleserver/hbp_nrp_cleserver/server/ROSC

also further down the line there is another problem when importing NRP modules:
Traceback (most recent call last):
File “/home/pkrack/Documents/NRP/ExDBackend/hbp_nrp_backend/hbp_nrp_backend/runserver.py”, line 32, in
from hbp_nrp_backend.rest_server import app
File “./hbp_nrp_backend/rest_server/init.py”, line 83, in
from hbp_nrp_backend.rest_server.__SimulationResources import SimulationResources
File “./hbp_nrp_backend/rest_server/__SimulationResources.py”, line 34, in
from hbp_nrp_backend.rest_server.__SimulationControl import _get_simulation_or_abort
File “./hbp_nrp_backend/rest_server/__SimulationControl.py”, line 34, in
from hbp_nrp_commons.bibi_functions import docstring_parameter
File “/home/pkrack/Documents/NRP/ExDBackend/hbp_nrp_commons/hbp_nrp_commons/bibi_functions.py”, line 27, in
from hbp_nrp_commons.generated import bibi_api_gen
ImportError: cannot import name bibi_api_gen

What I tried:
activating the virtual env of the nrp platform and installing netifaces. Works but then the next package is not found so i stopped there and uninstalled the pip installed version again, since it is recommended to take the official versions from the ubuntu repos.
Also i checked that python-netifaces is installed.

I also have the entire build log. There were a few errors during the fixing problems with pynn were files were not found two sed calls. But the files were deleted jsut after the sed so I guess it is normal.

[vonarnim]

Hi,

Yes, in general, the NRP’s venv does not like concurring system packages. In the case of netifaces, I would rather uninstall the system python-netifaces and then rerun user-scripts/update_nrp build to have it installed in the NRP’s platform_venv. Also, make sure your PYTHONPATH is void before running build or cle-start.
For the bibi_api_gen missing, this is a package that is created by compiling schema definitions XSD files in the Experiment repo. This is normally done automatically by user-scripts/configure_nrp. Try running this script separately.

Hope this helps,
Axel

[pkrack]

Thank you for your answer.

I found the cause(s) of the problem.
First, the reason why the virtual environment does not find the system packages installed with apt, is that the python executable in the venv does not look in /usr/lib/python2.7/dist-packages. This is because this folder is debian (so also ubuntu) specific. A python executable installed by apt is aware of the dist-package folder, a normal python executable is not, it looks only in site-packages. See https://stackoverflow.com/questions/9387928/whats-the-difference-between-dist-packages-and-site-packages#9388115
This means that all the steps in the installation guide in which python packages are installed with apt are useless since those packages are installed again with pip at a later stage.
To use the system packages installed with apt, one must add /usr/lib/python2.7/dist-packages to the PYTHONPATH. This breaks something else at a later stage (I don’t remember what, sry) in ./update_nrp build

Next the reason why update_nrp build does not solve that problem by installing the missing packages in the venv, is because of the opencv package’s setup.py is causing a ModuleNotFoundError: No module named 'skbuild'. This results in none of the packages in the CLE requirements to be installed. Reason for the error: since version opencv 4.3.* (july 2020) the opencv package uses some build system that older pip versions do not know about, or something like that, see https://pypi.org/project/opencv-python/ under FAQ the second question for an explanation and a solution (update pip).

As mentioned in another post (and also in the README in bitbucket), updating pip with pip install --upgrade pip breaks something else. This was solved by downgrading pip and even automatically installing pip 9.0.3 if the venv has a newer one.
This is not the right approach.
The system pip (installed with apt) should indeed only be upgraded with apt and never with pip. The pip executable in the virtual environment can however be a newer one. If the error mentioned in that post appears, just use python -m pip instead of directly using the pip executable (a wrapper) which causes the problem.
See https://github.com/pypa/pip/issues/5599 for a better explanation

I can’t find where in the build process an older pip version is enforced. It seems to be in the Makefiles of each nrp component. They include ‘user_makefile’ which does something according to the pip version. Since Makefiles are no fun to read, any help would be appreciated at this point.

[pkrack]

Found a fix.

Remove python-opencv from the requirements.txt file in the CLE repo, e.g.:

sed -i ‘/opencv-python/d’ $HBP/CLE/hbp_nrp_cle/requirements.txt

Link (or copy should work too) /usr/lib/python2.7/dist-packages/cv2.x86_64-linux-gnu.so to ~/.opt/platform_venv/lib/python2.7/site-packages, e.g.:

sudo ln /usr/lib/python2.7/dist-packages/cv2.x86_64-linux-gnu.so ~/.opt/platform_venv/lib/python2.7/site-packages/

Regarding what I said in the post before:
The dist-packages folder should not be added to the PYTHONPATH this breaks more stuff and its not how it’s intended to be used.
The clean solution would be to update pip in the venv and use python -m pip instead of the pip executable.

Also note that none of the packages in /usr/lib/dist-packages is used by the venv. Even when adding that folder to the PYTHONPATH, most of the packages are reinstalled anyways because of the version restrictions in the requirements.

[vonarnim]

Dear Pierre,

The point is exactly to not use system packages, reason why we don’t include system directories in pythonpath. There are too many possible version conflicts with nrp dependencies. We download all necessary packages in .opt/platform_venv.

Removing opencv from requirements is not a problem as long as you don’t run experiments with image processing transfer functions, like the size of husky braitenberg ones.

About the downgrade of pip, I think this is enforced in makefiles indeed, but I’m in vacation and can’t verify. Maybe a developer could jump in here

Hope this clears it a bit up.
Axel

[pkrack]

The bitbucket instructions mention that the deb packages are the reference, hence the confusion. I thought they were supposed to be used by the CLE. Thank you for clearing up the confusion.

The husky braitenberg experiments also work after manually putting the system installed opencv bindings (cv2.x86_64-linux-gnu.so) in the venv as a quick and dirty fix.

Maybe either the install instructions should be changed or a newer pip should be allowed, along with using python -m pip instead of using the ~/.opt/platform_venv/bin/pip wrapper in the Makefiles in order for the installation instructions to still work.

Everything works for my purposes now, thank you for taking time out of your holidays!

Best regards,
Pierre

[vahidzolf]

Dear Pierre,

Please change these two files and add specific version for the opencv-python:
$HBP/CLE/hbp_nrp_cle/requirements.txt
$HBP/CLE/hbp_nrp_cle/hbp_nrp_cle.egg-info/requires.txt
Change the line of opencv-python in this way:
opencv-python==4.2.0.32
then ./update_nrp build all