-
The following commands need to be entered into a terminal. Note that some of them will require superuser privileges:
-
Use systemctl to edit the existing guppyd service (this will open a text editor with a copy of the existing service file):
sudo systemctl edit guppyd.service --full
If a prompt appears asking about overwriting the exiting guppyd.service file, enter "y" to continue. -
Edit that new service file to contain the required GPU configuration for the guppy_basecall_server. You can change any other server arguments at the same time.
To do so, change this line in the service file:
ExecStart=/opt/ont/guppy/bin/guppy_basecall_server <arguments>
-
Save the file and exit the text editor (the filename may look odd, but systemctl should change it to the correct name later).
-
Stop the MinKNOW service:
sudo service minknow stop
-
Stop the guppyd service:
sudo service guppyd stop
-
Confirm the guppy_basecall_server process is not running:
$ ps -A | grep guppy_basecall_
If the result of the above command is not blank, manually kill the process:
sudo killall guppy_basecall_server
-
Start the guppyd service:
sudo service guppyd start
-
Confirm the guppy_basecall_server is running and is using the GPU:
nvidia-smi
If the Guppy basecall server is not launching correctly, check its log output using journalctl ("-n 100" shows the last 100 entries in the journal) to see what is going wrong:
sudo journalctl -u guppyd.service -n 100
-
Confirm that the newly updated settings are being used by the guppyd service:
sudo service guppyd status
The output should include a line starting "CGroup" which will contain the arguments used by the basecall server. There should also be a line starting "Active: active (running)".
-
Start the MinKNOW service:
sudo service minknow start
-
Monitor your first sequencing run using the MinKNOW GUI to make sure basecalling is working as expected.
-
Troubleshooting
If some part of the above process does not work, then it is possible the guppyd service may end up misconfigured, and may be automatically disabled by the system. There are a few diagnostic checks that can be performed.
- Look at the Guppy basecall server logs and check for error messages that indicate that the GPU has not been configured or not found. Guppy log files are stored in
/var/log/guppy
- Use journalctl to directly read the log entries produced by Guppy and systemctl, and check for any error messages:
sudo journalctl -u guppyd.service -n 100
- Check whether the service is enabled:
systemctl list-unit-files | grep guppyd.service
If the service is not listed as "enabled", then it will either be marked as "disabled" or "masked". You can reset those statuses as described below. a. If the service is marked as "disabled":
sudo systemctl enable guppyd.service
b. If the service is marked as "masked":
sudo systemctl unmask guppyd.service
Then enable the service:
sudo systemctl enable guppyd.service
- Reinstall the service:
sudo apt install --reinstall ont-guppyd-for-minion sudo systemctl revert guppyd.service sudo service guppyd restart
- Look at the Guppy basecall server logs and check for error messages that indicate that the GPU has not been configured or not found. Guppy log files are stored in
-
Reconfiguration case: changing the set of GPUs used.
By default, MinKNOW will configure the guppy_basecall_server with
--device cuda:all
, which tells Guppy to use all the GPUs on the host machine. If this is not desired, edit the guppyd service file and change the--device
parameter to select specific devices (e.g.--device cuda:0
). See the Guppy protocol for more information on the--device
argument. -
Setting GPU parameters for lower-memory graphics cards (8 GB or less).
When performing GPU basecalling, Guppy divides data up into chunks and then combines together a certain number of chunks which are basecalled at the same time by a basecall "runner". The GPU memory use of one of these runners is governed by two things:
- The number of chunks in it (or "chunks per runner").
- The complexity of the basecall model being used (higher-accuracy basecall models use more memory).
For basecalling to occur, enough GPU memory needs to be allocated for at least one runner, and each basecalling configuration file has a default number of chunks per runner. This means that by default, each basecall model has a minimum amount of memory required to perform basecalling. When using graphics cards with lower amounts of memory, larger basecall models (such as HAC and Sup) may not run, instead returning an "out of memory" error. This can be addressed by lowering the number of chunks per runner.
Changing the Guppy "chunks per runner" value
- When using the standalone basecaller (
guppy_basecaller
), set the--chunks_per_runner
command-line parameter. - When using the basecall server with MinKNOW: edit the guppyd-cpu service file and add
--chunks_per_runner <value>
to theExecStart
line, before restarting the service.
The following settings are recommended for 8 GB graphics cards. For cards with less GPU memory, or if the GPU is being used by other processes, these numbers may need to be lowered.
- For HAC, use
--chunks_per_runner 160
- For Sup basecalling models, use
--chunks_per_runner 10
After changing these settings, restart the guppyd-cpu service so that the changes take effect.
Configure Guppy to use TCP and allow remote connections
For security purposes, by default the Guppy basecall server will only allow connections from other processes running on the same computer. In some cases, you may need to connect from other PCs to perform basecalling, while at the same time allowing MinKNOW to run as normal and also make use of the Guppy basecall server.
Edit the guppyd service file and add
--use_tcp
and--allow_non_local
to theExecStart
line, before restarting the service.
"C:\Program Files\OxfordNanopore\MinKNOW\bin\config_editor" ^
--conf application --filename "C:\Program Files\OxfordNanopore\MinKNOW\conf\app_conf" ^
--set guppy.server_config.extra_arguments="--allow_non_local"
Edit the file
/opt/ont/minknow/conf/app_conf file
and add the following lines:
- name: server_port
type: int
default: 5555
- name: server_ipc_file
type: std::string
default: '"/tmp/.guppy/5555"'
- name: use_tcp
type: bool
default: defaults::guppy_use_tcp()
Then restart MinKNOW with the command:systemctl restart minknow
Recommended procedure
To find the best configuration values for GPU basecalling, it is recommended to have a small number of input files available that you can use to measure basecalling speed. These files should take at least a minute or so to basecall, to minimise the overhead that comes from loading the files into Guppy.
It is also recommended to only adjust the
--chunks_per_runner
value and not any other parameters. See the next section for further details.- Begin by setting
--chunks_per_runner
to 1, and confirm that GPU basecalling works with the configuration file you have chosen. Basecalling will likely be extremely slow at this point. Note the basecalling speed, which is listed in "samples per second" once basecalling has completed (e.g.samples/s: 3.22e+06
). - Increase the value of
--chunks_per_runner
until basecalling stops working due to a memory error, or you stop seeing an increase in basecalling speed. The final value of--chunks_per_runner
will likely be less than 1000, and for very high accuracy models it could be significantly less: use this range as a guide to choose the values of--chunks_per_runner
to test.
Note that it will likely be necessary to choose different values of
--chunks_per_runner
for different types of basecalling configuration files. For example, HAC configuration files will likely support higher values of--chunks_per_runner
than Sup configuration files.Other parameters
There are two other parameters that can affect basecalling speed and memory use:
--gpu_runners_per_device
(where "device" is a GPU)--chunk_size
Once Guppy can allocate enough GPU memory for a single runner, it will dynamically create additional runners as needed, up to the maximum number listed by
--gpu_runners_per_device
. However, Guppy will only create these additional runners if there is enough GPU memory available to do so. This means that you can leave--gpu_runners_per_device
at a high value (e.g. 20), and you are less likely to see performance changes by adjusting this downwards.It is not recommended to change the value of
--chunk_size
. This is because adjustments to this value will change your basecall results. -
Reconfiguration case: Setting GPU parameters for high-power graphics cards.
The default parameters for Guppy are designed to work reliably for all cards from the minimum specification upwards. However, the settings may not be optimal for newer or higher-power GPUs. It may be possible to get higher basecalling performance by experimenting with increasing the
--chunks_per_runner
setting, following the instructions above.