Nrp tensorflow tutorial confusion about virtualenv

#1

The point is to create an isolated virtualenv. But in the instructions in the tensorflow installation, which the tensorflow tutorial from nrp defers to, the virtualenv is created with flag --system-site-packages. Doesn’t this defeat the purpose of having an isolated virtualenv? What’s correct?

#2

Hi Akshay

The virtualenv is created with flag --system-site-packages, doesn’t this defeat the purpose of having an isolated virtualenv?

It doesn’t if you don’t have too specific packages installed system-wide. With the --system-site-packages, you simply won’t have to reinstall packages that are already present on your system, but the one you do install in the virtualenv are still properly isolated.

Dealing with experiment-specific python packages and PYTHONPATH is actually quite a headache. Personally, I use virtualenvwrapper with the --system-site-packages flag. I have custom postactivate/postdeactivate scripts that add the virtualenv’s libraries to my PYTHONPATH when I activate it. Let me know if that interests you - I was planning on writing a post on our blog about it.

#3

Hi Jacques,

Thanks for your response. So as I understand it, the packages outside the virtualenv cannot access or use the packages within it for its dependences, and packages within the virtualenv firstly triy to use packages within it for its dependencies and when it has access to the system, uses system packages when it cannot fulfill its dependencies from venv internal packages. Is that right?

As you may have noticed from another post of mine, I’m having some issues importing keras-rl from a virtual environment. So yes indeed I am interested in learning more about how you work with specific packages for experiments! Do you think your scripts would help? I should probably install virtualenvwrapper with apt-get instead of pip right?

#4

So as I understand it, the packages outside the virtualenv cannot access or use the packages within it for its dependences

It definitely can, if you activated the virtualenv or if you you have the virtualenv in your PYTHONPATH. Note that this situation is mostly a source of error. For example, you have a system package depending on numpy, and you have a newer version of numpy in your virtualenv, that one might be picked.

packages within the virtualenv firstly triy to use packages within it for its dependencies and when it has access to the system, uses system packages when it cannot fulfill its dependencies from venv internal packages. Is that right?

The order of processing of the python import is really important. It’s quite clear that the path in your PYTHONPATH are ordered, but what about activated virtualenv?

Well this is the weird thing: when you activate a VIRTUALENV, it does not add itself to your PYTHONPATH. Instead, it changes the path of your python executable to the virtualenv. The virtualenv’s libraries are alongside this executable, so magically resolved. Testing on my laptop, it seems that these packages are resolved before the PYTHONPATH is checked for these dependencies.

Do you think your scripts would help? I should probably install virtualenvwrapper with apt-get instead of pip right?

Okay I’ll write a blog post soon then.

#5

Hi Akshay,

I have found the time to write a blog post on how I handle my experiment-specific python packages.

Here it is: https://hbpneuroroboticsblog.wordpress.com/2018/03/16/handling-experiment-specific-python-packages-in-the-nrp/

Let me know if that works for you,
Cheers,
Jacques

Problem importing tensorflow / keras-rl from tutorial
#6

Okay thanks Jacques. We have found a workaround in the meantime. But I will take a look at it and let you know.

Best,
Akshay