Install Python Extensions Automatically

Instructions how to install the CSXCAD & openEMS Python interface.

Methods

CSXCAD and Python extensions can be installed using two methods: automatic install via ./update_openEMS.sh using the --python flag, or building each Python extension separately.

This page exclusively deals with the recommended script-assisted auto-installation process.

See also

Quick Start

In the simplest cases, all one needs is adding the --python flag in ./update_openEMS.sh. For example, to install the C++ project and its Python extensions simultaneously to the prefix ~/opt/openEMS, run:

./update_openEMS.sh ~/opt/openEMS --python

In the latest openEMS development version (to be released as v0.0.37), Python extensions and their dependencies are installed into an isolated “virtual environment” in the venv subdirectory (e.g. ~/opt/openEMS/venv). This environment must be activated before using Python with CSXCAD or openEMS.

source ~/opt/openEMS/venv/bin/activate

# leave the venv with "deactivate"

Important

In a venv, its environment is isolated from the operating system’s own. System-wide Python packages are invisible, only Python packages installed specifically here can be seen. If you need other third-party packages, install them via pip3 provided within this venv.

For example, to analyze S-parameters via scikit-rf:

source ~/opt/openEMS/venv/bin/activate  # if not activated
pip3 install scikit-rf

Customize Install

To satisfy the needs of users from different backgrounds, users are:

  1. Not required to install Python extensions to an isolated venv, the venv can be disabled.

  2. Not required to use the created venv environment, a pre-existing venv or an alternative venv path can be used.

  3. Not required to manage packages via pip3, or to have Internet access to PyPI. It’s possible to manage dependencies manually using the system’s package manager, without pip3.

Arguments

The Python installation behavior of update_openEMS.sh can be customized using the following arguments:

--python-venv-mode <mode>

Python extensions installation mode:

  • auto: create a new Python venv if no venv is already activated, otherwise use the existing venv (default).

  • venv: create a new Python venv.

  • site: create a new Python venv with --system-site-packages.

  • disable: don’t create a new venv, install Python extension directly to a default path (usually in home directory (e.g. ~/.local).

--python-venv-dir

Override default Python venv creation path. By default, use the venv subdirectory of the installation path.

--python-use-network <option>

Download needed Python pip packages from Internet

  • auto: use Internet when needed (default).

  • disable: all dependencies must be manually installed, or installation fails (create venv with --system-site-packages, run pip with --no-build-isolation, disable pip self-update and setuptools_scm).

Typical Customization Cases

Q: I don’t want update_openEMS.sh to create a new venv for me, because I have my own.

Active your venv before calling ./update_openEMS.sh. By default, --python-venv-mode auto means that no new venv is created if an venv has already been activated.

# activate your own venv
source ~/venvs/snake/bin/activate
./update_openEMS.sh ~/opt/openEMS --python

Q: I want to create a new venv, but not under the /venv subdirectory of openEMS.

Use the --python-venv-dir argument to specify an alternative path. If the same path contains an existing venv, it will be overwritten.

./update_openEMS.sh ~/opt/openEMS --python-venv-dir ~/venvs/openEMS

Q: All system packages are invisible in the Python venv. I don’t want to reinstall them via pip3 for the venv, I want to use the existing system Python packages.

Use the --python-venv-mode site argument to create a Python venv with the flag --system-site-packages. In this mode, all existing system packages are exposed, at the same time, you can still install your own packages in this venv.

./update_openEMS.sh ~/opt/openEMS --python --python-venv-mode site

Warning

It’s recommended to install as many packages as possible using the system’s own package manager, and only to install a package via pip3 if it doesn’t exist within the system. Otherwise, it’s possible to install incompatible versions of the same packages.

All Python packages marked as optional in the Requirements page should be installed to ensure this.

Q: I don’t want ./update_openEMS.sh to download dependencies using pip3 because I don’t have Internet access.

Use the --python-use-network disable argument. In this mode, you’re required to manage Python dependencies manually using your operating system’s package manager. It enables --python-venv-mode site implicitly, in addition, it disables many other behaviors that trigger network downloads, such as self-updating pip3.

./update_openEMS.sh ~/opt/openEMS --python --python-use-network disable

Warning

All Python packages marked as optional in the Requirements page should be already installed, because the operating system is responsible for package management here.

Tip

In theory, one can use a DVD, a USB drive, or any file:// path as a software repository, this use case is supported by mature package managers such as apt, rpm, dnf , and was widely used in the past for DVD installations. Today, it’s still a potential solution for offline systems. See Use a Debian DVD ISO as an Upgrade Source and How to Set Up yum Repository for Locally-mounted DVD on Red Hat Enterprise Linux 7.

Q: I don’t want to use pip3 to manage Python package dependencies, I want to use my system’s own package manager.

Use the --python-use-network disable argument, see Q: I don’t want ./update_openEMS.sh to download dependencies using pip3 because I don’t have Internet access.

Q: I have Internet access, but behind a proxy.

Your have three possible solutions, pick one of them.

  1. Use --python-use-network disable: If you’re using the system on a regular basis, you probably already have a solution for downloading new packages using the OS package manager via a proxy. In this case, treat the system as a special case of an “offline system” or “system with manual package management”.

    Therefore, use the --python-use-network disable argument, see Q: I don’t want ./update_openEMS.sh to download dependencies using pip3 because I don’t have Internet access.

  2. Use an HTTP Proxy: See Q: I have Internet access, but behind an HTTP proxy.

  3. Use a SOCKS Proxy: See Q: I have Internet access, but behind a SOCKS proxy.

Q: I have Internet access, but behind an HTTP proxy.

If you want to use pip3 to manage packages (rather than manage it manually), but behind a proxy, it’s possible to set a global proxy using the standard environment variable https_proxy with the format [protocol://]<host>[:port]:

# HTTP proxy server for HTTP/HTTPS URLs
export http_proxy="http://proxy.example.com:8080"
export https_proxy="http://proxy.example.com:8080"

# build and install openEMS as usual
./update_openEMS.sh ~/opt/openEMS --python

Q: I have Internet access, but behind a SOCKS proxy.

A SOCKS proxy has an additional complication.

# SOCKS5 proxy server (with remote DNS) for HTTP/HTTPS URLs
export http_proxy="socks5h://proxy.example.com:8080"
export https_proxy="socks5h://proxy.example.com:8080"

By default, pip3 is not compatible with a proxy, due to a missing optional dependency named pysocks. The error ERROR: Could not install packages due to an OSError: Missing dependencies for SOCKS support is raised if the package is not installed. This forces one to unset http_proxy && unset https_porxy, making the proxy unusable.

Important

To use a SOCKS proxy with pip3, an optional dependency pysocks must be installed. This package is usually named pysocks, python3-socks, or a similar name. Check your package manager.

However, this brings us to the next problem: a standard venv is isolated from all system packages by default. If pysocks is installed via the system’s package manager, it’s still invisible and unusable, even if it has already been installed!

To solve this sub-problem, there are three sub-solutions, pick one of them.

  1. Treat the system as a special case of an “offline system” or “system with manual package management”. Use the --python-use-network disable argument.

    See Q: I don’t want ./update_openEMS.sh to download dependencies using pip3 because I don’t have Internet access.

  2. Expose system packages via --python-venv-mode site.

    See Q: All system packages are invisible in the Python venv. I don’t want to reinstall them via pip3 for the venv, I want to use the existing system Python packages.

  3. Prepare a “good” venv with pysocks preinstalled, and active your venv before calling ./update_openEMS.sh. This is a special case of Q: I don’t want update_openEMS.sh to create a new venv for me, because I have my own.

    # create and activate your own venv
python3 -m venv ~/venvs/snake/
source ~/venvs/snake/bin/activate

# Install pysocks manually while you still have Internet access,
# pip3 install pysocks

# or if you have an alternative proxy solution without using
# Python, such as proxychains.
# proxychains pip3 install pysocks

# SOCKS5 proxy server (with remote DNS) for HTTP/HTTPS URLs
export http_proxy="socks5h://proxy.example.com:8080"
export https_proxy="socks5h://proxy.example.com:8080"

# build and install openEMS as usual
./update_openEMS.sh ~/opt/openEMS --python

    The third solution is exceedingly difficult on a fully offline system. In a fresh venv, there’s no setuptools or pysocks, making it difficult to bootstrap the venv to a usable state. It is practical, only if the venv has already been prepared while direct Internet access is still available, or if an external tool such as proxychains can be used to enable the proxy without relying on pip3. Otherwise, the bootstrapping process would be a long battle. Make sure to make a backup copy of the venv directory to save your work after completion.

    Tip

    If using proxychains-ng, you may have to uncomment the last line of /etc/proxychains.conf, and replace it with the parameters of the SOCKS proxy, such as:

    socks5 127.0.0.1 8080

Q: I don’t want to use a Python venv at all, I want to install Python extensions to the default paths, which is the legacy behavior in previous openEMS versions.

Use --python-venv-mode disable.

./update_openEMS.sh ~/opt/openEMS --python --python-venv-mode disable

However, it’s strongly recommended to use --python-venv-mode site as an alternative to this legacy mode. You can still manage your Python packages manually like the legacy behavior, but it’s both supported by default (e.g. you can use pip3 to install packages without overriding it via --break-system-packages), and it allows multiple Python environments and openEMS installations to coexist without polluting ~/.local (e.g. ~/opt/openEMS_stable/venv and ~/opt/openEMS_dev/venv).

Warning

To avoid installing conflicting versions of packages, install as few packages as possible, meaning that all Python packages marked as optional in the Requirements page are recommended be installed.

Q: ./update_openEMS.sh is a black box, I need explicit control over the installation.