I don't understand python venv

Question

This is going to be a growing pain for a lot of people that pick up the Raspberry Pi 5 with Bookworm, too.

A lot of guides and instructions say to do:

pip install <package>

Or

apt-get install python3-<package>

The 'problem' from a usability perspective now, is that when you try to do:

pip install <package>

It complains at you with "this environment is externally managed" and to use a venv, and I don't understand them. It doesn't direct you on how to use a venv either.

I understand why we're doing it this way, so that we do not have a clash between operating system repository packages and the python pip installed packages.

Can someone explain to me how to use pip with a venv with an example package? I'm trying to find guides online and of course everyone has an opinion on how to do it three different ways but they also don't explain what exactly venv is doing or how to use it after installing the package with pip.

JWx · Answer

Python suffers from multitude of interpreter and library versions - which can lead to the situation that one library cannot co-exist with another, because they - for example - are dependent on another library, but in different versions. 
 To overcome this - as I understand - virtual environments were created, to have local execution sandbox with all the required libraries 
 To create one: 
 #python3 -m venv /root/test-venv 
 optionally - install required package first: 
 #apt install python3.10-venv 
 To activate it: 
 #source /root/test-venv/bin/activate 
 and then pip is local copy and packages should be installed inside this venv (to not interfere with other versions) 
 ( test-venv ) root@e14-panda:~# which pip /root/test-venv/bin/pip /root/test-venv/bin/pip (test-venv) root@e14-panda:~# pip list Package Version ---------- ------- pip 22.0.2 setuptools 59.6.0 
 
 Install something 
 
 (test-venv) root@e14-panda:~# pip install pandas Collecting pandas Downloading pandas-2.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.3 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.3/12.3 MB 5.1 MB/s eta 0:00:00 Collecting tzdata>=2022.1 Downloading tzdata-2023.3-py2.py3-none-any.whl (341 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 341.8/341.8 KB 3.3 MB/s eta 0:00:00 Collecting python-dateutil>=2.8.2 Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 KB 3.8 MB/s eta 0:00:00 Collecting numpy>=1.22.4 Downloading numpy-1.26.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.2/18.2 MB 5.1 MB/s eta 0:00:00 Collecting pytz>=2020.1 Downloading pytz-2023.3.post1-py2.py3-none-any.whl (502 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 502.5/502.5 KB 5.2 MB/s eta 0:00:00 Collecting six>=1.5 Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) Installing collected packages: pytz, tzdata, six, numpy, python-dateutil, pandas Successfully installed numpy-1.26.1 pandas-2.1.1 python-dateutil-2.8.2 pytz-2023.3.post1 six-1.16.0 tzdata-2023.3 
 List again: 
 (test-venv) root@e14-panda:~# pip list Package Version --------------- ------------ numpy 1.26.1 pandas 2.1.1 pip 22.0.2 python-dateutil 2.8.2 pytz 2023.3.post1 setuptools 59.6.0 six 1.16.0 tzdata 2023.3 
 Deactivate venv: 
 (test-venv) root@e14-panda:~# deactivate 
 Try pip again: 
 root@e14-panda:~# pip list Command 'pip' not found, but can be installed with: apt install python3-pip 
 
 There is even no pip command installed outside of venv

shabaz · Answer

I watched a training video on it a while back. It suggested the following: 
 (1) First off, check you've got 'virtualenv' installed. That's easy to do by typing which virtualenv 
 (2) If you don't have it installed then install it : ) You can use root (e.g. sudo ) and type apt-get install python3-virtualenv to install the default one for your Linux build. 
 (3) Now, it's a good idea to create a folder for housing your virtual environments. Personally, I create a folder off my home folder, called development, and then in there you could create a folder called (say) virt_python or whatever 
 (4) In that virt_python folder, whenever you're ready for a new virtual environment, simply type virtualenv bob (or whatever name you wish to call it). What's going on in the background, is that the virt_python/bob folder gets created, and inside that are folders for Python executable and libraries, and all these are completely independent from your normal /bin/python or wherever you normally have python and any libraries installed. 
 (5) Whenever you wish to use your new environment! Type: source ~/development/virt_python/bob/bin/activate 
 (6) Now you can use Python as normal (i.e. create a folder anywhere and write your app just as you normally would), but it will be in your virtual environment. You'll know that, because your command prompt will state (bob) . Whenever you use pip install , it will be into your virtual environment. 
 (7) When you're done, just type deactivate and you'll be out of the wonderland. 
 
 With PyCharm (here's a Windows screenshot) it tends to create a separate virtual environment for every project (you can see it creates a subfolder called venv ) but it's not necessary. If you wished, you could share a virtual environment with multiple projects, although it's just disk space, so not really necessary, since storage is cheap, and at least guarantees no clashes of libraries.

michaelkellett · Answer

There's a comprehensive, but not perfect, description here. 
 https://realpython.com/python-virtual-environments-a-primer/ 
 which may help. 
 MK

BigG · Answer

Yes, I find this new requirement rather painful. If it's my device and the library download is just for that device then I just use the option "--break-system-packages". 
 Otherwise, I think Python Poetry is worth considering instead of using venv. 
 https://python-poetry.org/

I don't understand python venv

Top Replies