-
The following commands need to be entered into a Windows command prompt which has been run as an adminstrator. They also assume a standard MinKNOW installation, where the location of MinKNOW is C:\Program Files\OxfordNanopore\MinKNOW.
-
Modify MinKNOW's application configuration to enable GPU basecalling and set the appropriate settings.
For example, to change the used CUDA devices to just use device 1, you would run:
"C:\Program Files\OxfordNanopore\MinKNOW\bin\config_editor.exe" ^
--conf application --filename "C:\Program Files\OxfordNanopore\MinKNOW\conf\app_conf" ^
--set guppy.server_config.gpu_devices="cuda:1"
Information about Guppy settings can be found in the appropriate section of the Guppy protocol. -
Restart the MinKNOW service:
- Open the Windows menu.
- Type "services" and select the Services app that is displayed.
- Scroll down and find the MinKNOW service.
- Right-click on it and select Restart.
-
Confirm the guppy_basecall_server is using the GPU:
nvidia-smi
Note that in some instances, nvidia-smi will not report running processes if launched through CMD. Running nvidia-smi in PowerShell will show the correct processes if this occurs. -
Monitor your first sequencing run using the MinKNOW GUI to make sure basecalling is working as expected.
-
Troubleshooting
If step 3 above does not show
guppy_basecall_server
using the GPU, or if Guppy crashes frequently, then it is recommended to check the Guppy log files. These files are normally found inC:\data\guppy_logs
, and a new file will be created every time the basecall server is launched.- If there is no server log with a timestamp that roughly matches step 2 ("Restart the MinKNOW service") above, then a new basecall server has not been launched. In this case, restart the MinKNOW service. If there is still no new log file created, restart your computer.
- If there is a new server log file but it does not contain the parameters that were set as part of step 5 ("Modify MinKNOW's application configuration") above, then repeat steps 1 and 2.
- If, during step 1, you see the following error message:
Failed to open C:\Program Files\OxfordNanopore\MinKNOW\conf\app_conf for writing.
, then your terminal has not been run as an administrator.
-
Reconfiguration case: changing the set of GPUs used.
By default, MinKNOW will configure the guppy_basecall_server with
--device cuda:all
, which tells Guppy to use all the GPUs on the host machine. If this is not desired, the--device
parameter can be changed to select specific devices (e.g.--device cuda:0
). See the Guppy protocol for more information on the--device
argument.This setting can be changed by specifying a new
guppy.server_config.gpu_devices
:
"C:\Program Files\OxfordNanopore\MinKNOW\bin\config_editor.exe" ^
--conf application --filename "C:\Program Files\OxfordNanopore\MinKNOW\conf\app_conf" ^
--set guppy.server_config.gpu_devices="cuda:1"
-
Reconfiguration case: setting GPU parameters for lower-memory graphics cards (8 GB or less).
When performing GPU basecalling, Guppy divides data up into chunks and then combines together a certain number of chunks which are basecalled at the same time by a basecall "runner". The GPU memory use of one of these runners is governed by two things:
- The number of chunks in it (or "chunks per runner").
- The complexity of the basecall model being used (higher-accuracy basecall models use more memory).
For basecalling to occur, enough GPU memory needs to be allocated for at least one runner, and each basecalling configuration file has a default number of chunks per runner. This means that by default, each basecall model has a minimum amount of memory required to perform basecalling. When using graphics cards with lower amounts of memory, larger basecall models (such as HAC and Sup) may not run, instead returning an "out of memory" error. This can be addressed by lowering the number of chunks per runner.
Changing the Guppy "chunks per runner" value
- When using the standalone basecaller (
guppy_basecaller
), set the--chunks_per_runner
command-line parameter. - When using the basecall server with MinKNOW: edit MinKNOW's application configuration to add the
--chunks_per_runner
option to Guppy's "extra arguments" section:
"C:\Program Files\OxfordNanopore\MinKNOW\bin\config_editor" ^
--conf application --filename "C:\Program Files\OxfordNanopore\MinKNOW\conf\app_conf" ^
--set guppy.server_config.extra_arguments="--chunks_per_runner <value>"
Or--set guppy.server_config.extra_arguments=""
to undo your changes.The following settings are recommended for 8 GB graphics cards. For cards with less GPU memory, or if the GPU is being used by other processes, these numbers may need to be lowered.
- For HAC, use
--chunks_per_runner 160
- For Sup basecalling models, use
--chunks_per_runner 10
After changing these settings, restart the MinKNOW service so that the changes take effect.
Configure Guppy to use TCP and allow remote connections
For security purposes, by default the Guppy basecall server will only allow connections from other processes running on the same computer. In some cases, you may need to connect from other PCs to perform basecalling, while at the same time allowing MinKNOW to run as normal and also make use of the Guppy basecall server.
Edit MinKNOW's application configuration to add
--allow_non_local
to Guppy's "extra arguments" section:
"C:\Program Files\OxfordNanopore\MinKNOW\bin\config_editor" ^
--conf application --filename "C:\Program Files\OxfordNanopore\MinKNOW\conf\app_conf" ^
--set guppy.server_config.extra_arguments="--allow_non_local"
Then restart MinKNOW with the command:systemctl restart minknow
Recommended procedure
To find the best configuration values for GPU basecalling, it is recommended to have a small number of input files available that you can use to measure basecalling speed. These files should take at least a minute or so to basecall, to minimise the overhead that comes from loading the files into Guppy.
It is also recommended to only adjust the
--chunks_per_runner
value and not any other parameters. See the next section for further details.- Begin by setting
--chunks_per_runner
to 1, and confirm that GPU basecalling works with the configuration file you have chosen. Basecalling will likely be extremely slow at this point. Note the basecalling speed, which is listed in "samples per second" once basecalling has completed (e.g.samples/s: 3.22e+06
). - Increase the value of
--chunks_per_runner
until basecalling stops working due to a memory error, or you stop seeing an increase in basecalling speed. The final value of--chunks_per_runner
will likely be less than 1000, and for very high accuracy models it could be significantly less: use this range as a guide to choose the values of--chunks_per_runner
to test.
Note that it will likely be necessary to choose different values of
--chunks_per_runner
for different types of basecalling configuration files. For example, HAC configuration files will likely support higher values of--chunks_per_runner
than Sup configuration files.Other parameters
There are two other parameters that can affect basecalling speed and memory use:
--gpu_runners_per_device
(where "device" is a GPU)--chunk_size
Once Guppy can allocate enough GPU memory for a single runner, it will dynamically create additional runners as needed, up to the maximum number listed by
--gpu_runners_per_device
. However, Guppy will only create these additional runners if there is enough GPU memory available to do so. This means that you can leave--gpu_runners_per_device
at a high value (e.g. 20), and you are less likely to see performance changes by adjusting this downwards.It is not recommended to change the value of
--chunk_size
. This is because adjustments to this value will change your basecall results. -
Reconfiguration case: Setting GPU parameters for high-power graphics cards.
The default parameters for Guppy are designed to work reliably for all cards from the minimum specification upwards. However, the settings may not be optimal for newer or higher-power GPUs. It may be possible to get higher basecalling performance by experimenting with increasing the
--chunks_per_runner
setting, following the instructions above.