Machine Learning

TensorFlow#

TensorFlow is an open source software library for high performance numerical computation. It is used for machine learning applications such as neural networks. It is developed by Google and released under the Apache 2.0 open source license.

TensorFlow is not readily available in the IMAXT environment. However it is straightforward to install using the instructions below.

Using Python environments#

The easiest way to install TensorFlow is to create a Python environment and install it using pip. For example, to create an environment with the Python 3.9 version:

mkenv tf --python /opt/conda/envs/py39/bin/python \
  --jupyter "TensorFlow"

This will create a Python environment called tf and install TensorFlow in it. The --jupyter option will add the environment to the JupyterLab launcher.

After this step is complete, activate the newly created environment and install TensorFlow using pip:

workon tf
pip install tensorflow==2.10.1

This will install TensorFlow version 2.10.1. The version can be changed to any other version available in the TensorFlow PyPI repository but note that not all versions are available for all Python versions.

You can check that TensorFlow is installed correctly by running the following command:

python -c "import tensorflow as tf; print(tf.__version__)"

If you have a GPU available, you can check that TensorFlow is using it by running the following command:

python -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

Using Conda environments#

Alternatively one may use conda in order to create an environment. Conda is more appropiate when it is also required to install the GPU binary libraries. In order to use conda, the useconda function loads the necessary variables

useconda

For example, in order to install TensorFlow in a conda environment:

conda create -n tf tensorflow=2.10.1 -c conda-forge -c defaults

This will create a conda environment called tf and install TensorFlow in it. The version can be changed to any other version available in the TensorFlow conda repository but note that not all versions are available for all Python versions.

After this step is complete, activate the newly created environment:

conda activate tf

You can check that TensorFlow is installed correctly by running the following command:

env LD_LIBRARY_PATH=$HOME/.conda/envs/tf/lib python -c "import tensorflow as tf; print(tf.__version__)"

If you have a GPU available, you can check that TensorFlow is using it by running the following command:

env LD_LIBRARY_PATH=$HOME/.conda/envs/tf/lib python -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

Note that in order to install pip packages in a conda environment you may need to set “PIP_REQUIRE_VIRTUALENV=false” in your environment:

env PIP_REQUIRE_VIRTUALENV=false pip install pandas

Adding the environment to the JupyterLab launcher is done as follows. First install the ipykernel package:

conda install -n tf ipykernel

Then add the environment to the JupyterLab launcher:

python -m ipykernel install --user --name tf --display-name "TensorFlow"

This creates a new kernel called TensorFlow that can be selected from the JupyterLab launcher. The kernel.json file for this kernel is located in ~/.local/share/jupyter/kernels/tf/kernel.json and can be edited to change the name of the kernel. In our case we need to add the LD_LIBRARY_PATH environment variable to the kernel so that TensorFlow can find the GPU libraries. This can be done by editing the kernel.json file and adding the following line:

"env": {"LD_LIBRARY_PATH": "/home/username/.conda/envs/tf/lib"}

PyTorch#

PyTorch is an open source software library for high performance numerical computation. It is used for machine learning applications such as neural networks. It is developed by Facebook and released under the BSD 3-Clause open source license.

PyTorch is not readily available in the IMAXT environment. However it is straightforward to install using the instructions below.

Using Python environments#

Using pip to install PyTorch in a Python environment is straightforward. For example, to create an environment with the Python 3.9 version:

mkenv torch --python /opt/conda/envs/py39/bin/python --jupyter "PyTorch"

This will create a Python environment called torch and install PyTorch in it. The --jupyter option will add the environment to the JupyterLab launcher.

After this step is complete, activate the newly created environment and install PyTorch using pip:

workon torch
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

You can check that PyTorch is installed correctly by running the following command:

python -c "import torch; print(torch.__version__)"

If you have a GPU available, you can check that PyTorch is using it by running the following command:

python -c "import torch; print(torch.cuda.is_available())"

Using Conda environments#

Alternatively one may use conda in order to create an environment. Conda is more appropiate when it is also required to install the GPU binary libraries. In order to use conda, the useconda function loads the necessary variables

useconda

For example, in order to install PyTorch in a conda environment:

conda create -n torch pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

This will create a conda environment called torch and install PyTorch in it.