Skip to content

Jupyter notebook

Jupyter notebook is a powerful tool for small analysis or to prototype your analysis. After having created your virtualenv you can install it on your laptop with

python -m pip install jupyter

and run it:

jupyter notebook

Info

If you have set up virtualenv with LCG as in the previous section jupyter should be already installed.

it should open a page in a web browser where you can execute Python code (using Shift+Enter) and see the output, including graphical output.

Kernel in jupyter

Jupyter supports multiple kernels, not just for Python. For example, you can also execute C++ code via xeus or ROOT (both use cling).

You can create a kernel that is using your virtualenv. Sometimes this is done automatically (when creating a new jupyter notebook you should find the name of your virtualenv). To do it manually

python -m ipykernel install --user --name=myenv

where myenv is the name of the folder of your virtualenv.

Remote jupyter

Another good thing about jupyter is that it is split in client/server, so you can run the server (where the kernel runs) in a remote machine and interact with it with the browser (or other interfaces) of your laptop.

For example, log in on proof and start a jupyter notebook

jupyter notebook --no-browser --port=8889

Remember the port you have used. This is the port the server will use. If it is busy just change the number to a similar number (>1024). You can avoid specifying the port number, one will be assigned automatically. Take note about:

  • the specific remote machine you are logger, for example, proof-01.mi.infn.it
  • The full URL jupyter will prompt, something like http://localhost:8889/?token=XXXXXXXXXX"

On your laptop:

ssh -N -L localhost:8889:localhost:8889 username@remote-machine

Where you have to change username and remote-machine (and maybe the port). If you are using Putty on Windows use putty instead of ssh from the command line (for example from the Power Shell)

Info

What we are doing here is redirecting the localhost:8889 to the remote machine, in this case to the same port (the local port can be different)

Open the browser and copy and paste the link you have noted, including the token.

To create a new jupyter select the New/Python 3 (ipykernel).

When you have finished working you can turn down the jupyter server. Go to the shell when it is running and hit Ctrl+c and confirm.

Tip

You may need to run commands on the remote machine from the shell, but it is blocked by the jupyter notebook which is running. There are several solutions:

  • open a new terminal and reconnect
  • in the blocked terminal use Ctrl+z to stop the jupyter notebook and then send it to the background with bg. In this way, jupyter continues to run, but you can use the shell. To put back jupyter in the foreground use fg, for example, if you want to terminate it
  • Inside the jupyter interface (from your browser) you can create a terminal instead of a notebook and execute your command there

Remote jupyter with jump

Starting a jupyter session using a remote server can be tedious. It means connecting to the remote machine, setting up the environment, starting the jupyter server, opening an SSH tunnel and finally opening the browser. This can be done automatically with jump. In addition, if the connection is lost, with jump you can easily reconnect to the session, which remains active on the remote machine. See the instructions on the GitHub page.

Warning

Jump does not close the session when you close the browser. You have to do it manually. Remember to do that, otherwise many jupyter servers will remain active on the remote machine. In this case, you can use the command jump killall to kill all the jupyter servers. You can check the list of your processes with ps aux | grep $USER.

You can install Jump from GitHub

python -m pip install git+https://github.com/Olllom/jump.git

Jump supports setup from virtualenv, conda, mamba, and manual scripts.

Remote jupyter with vscode

If you are using vscode you can use the remote development extension to connect to a remote machine and start a jupyter server. See the documentation for more details.

Here few additional instructions if you have created a virtualenv with LCG software on a remote machine, following the instruction here. Original instructions are taken from here.

These instructions require to have the Python extension and the Jupyter extension installed in vscode.

Let's assume you have a virtual environment in the directory ~/venv-lcg103 built on to of LCG_103 x86_64-centos7-gcc11-opt. You can create a jupyter kernel using the virtualenv you have created. First, activate the virtualenv and install ipykernel:

source ~/venv-lcg103/bin/activate
python -m pip install --upgrade ipykernel

and create the kernel:

python -m ipykernel install --user --name="venv-lcg103" --display-name="LCG view 103 + venv"
Since the lcg environment must be set up before the virtual environment you have to change the configuration of the kernel. The configuration is located in ~/.local/share/jupyter/kernels/venv-lcg103/kernel.json. Edit the file and change it to (change the folder):

{
    "argv": [
        "/bin/bash",
        "/cnfs/homes_fs/home/turra/.local/share/jupyter/kernels/venv-lcg103/startup.sh",
        "-f",
        "{connection_file}"
    ],
    "display_name": "LCG view 103 + venv",
    "language": "python-custom",
    "metadata": {
        "debugger": true
    }
}

and create the startup.sh in the same directory with the following content:

#!/usr/bin/env bash

export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
# Allows for working with wrappers as well
source "${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh" --quiet || echo "~~~ERROR: setupATLAS failed!~~~"

lsetup 'views LCG_103 x86_64-centos7-gcc11-opt'

. ~/venv-lcg103/bin/activate

exec python -m ipykernel_launcher $@

Now open vscode on your machine and connect to the remote machine (F1, then Remote-SSH: Connect to Host... and select the remote machine). Create a new jupyter notebook (F1, then Create: Jupyter notebook). It will ask for the kernel, select the one you have created. You can now use the notebook as usual.