-
Memory usage for Guppy
If the memory requirements for Guppy exceed available memory on the host machine, or if other computationally-intensive work is performed while Guppy is running, then Guppy may run out of memory and crash. If Guppy fails for this reason the cause may not be directly apparent - the application may either simply crash or it may return a "segmentation fault" or "killed" error. In such cases, either the number of threads should be reduced or other computationally-intensive tasks should be stopped.
-
CUDA_ERROR_OUT_OF_MEMORY, CUBLAS_STATUS_ALLOC_FAILED, or CUBLAS_STATUS_EXECUTION_FAILED
If GPU basecalling is enabled, and available memory on the GPU device is exceeded, Guppy may exit reporting an error of type
CUDA_ERROR_OUT_OF_MEMORY
,CUBLAS_STATUS_ALLOC_FAILED
, orCUBLAS_STATUS_EXECUTION_FAILED
. If this happens, there are two possible fixes:- Limit the amount of GPU memory that Guppy uses by appending a colon followed by a percentage to the device string (
-x
or--device
) passed to Guppy. Start with 50%, e.g. using-x cuda:0:50%
instead of-x cuda:0
, and, if this succeeds, increase the percentage until Guppy fails to run. Reduce the number of runners or the number of chunks per runner to allow Guppy to run. See the section CPU/GPU basecalling usage above about how to estimate the amount of memory to use. When in doubt, setting:
--num_callers 1
--gpu_runners_per_device 1
--chunks_per_runner 1
should result in very low GPU memory use (and correspondingly low basecalling speed), and you can experiment with increasing those numbers until running out of GPU memory again.
- Limit the amount of GPU memory that Guppy uses by appending a colon followed by a percentage to the device string (
-
Advanced troubleshooting for Guppy crashes
Despite our best efforts, Guppy will occasionally crash due to bugs, and being able to diagnose and fix these is very important. This generally involves a two-step process:
-
Provide general configuration information from Guppy
Provide general configuration information
This is just information about how you ran Guppy. It should include:- Your Guppy version (find this by running
guppy_basecaller
with--version
). - The full command used to call Guppy.
- Any relevant files, such as your configuration file (if a custom one was used), or read files if available.
- Any other information you think is relevant, such as the hardware Guppy was running on (GPU model, etc).
If possible, reproduce the crash with tracing enabled and provide these trace files along with any other log files. To do this follow these steps:
Create a file called trace_config.txt containing the following line:
* trace_file
Pass the path to this file to the relevant guppy application either using a command-line parameter, i.e.
--trace_domains_config /path/to/trace_config.txt
or using an environment variable, i.e.
ONT_GUPPY_TRACE_CONFIG=/path/to/trace_config.txt
If using a Guppy application, this step can be ignored. If using the client lib as opposed to a guppy application, the save folder for output trace will need to be set using an environment variable, i.e.
ONT_GUPPY_TRACE_FOLDER=/folder/to/output/trace/files/
Rerun the application to reproduce the error.
Any generated trace and logging files will be written to the save folder.
Note: Remove the
trace_config.txt
file the command-line parameter and any environment variables. This will prevent the application continuing to fill up the hard disk with unnecessary trace files.
- Your Guppy version (find this by running
-
Attempt to isolate the read which caused the crash in Guppy
Frequently, Guppy crashes will be due to a single read, and figuring out which read caused the crash will make it much easier to reproduce it. Do that like this:
- If you are outputting reads to subfolders, check your
guppy_basecaller_<timestamp>.log
file to see which subfolder Guppy was writing to when it crashed. Any problem reads are likely in this folder. - Run the same Guppy command again (on just the affected subfolder if possible), enabling verbose logging.
- Wait for Guppy to crash again.
- Look in your log file for the last read which was sent to Guppy (it will probably not have a "finished processing" message present in the log).
- Run Guppy again with only that one read, and see if it crashes. If it did, you've found the problem read and can send it to customer services.
- If Guppy did not crash, try again with the last few reads listed in the log file, as Guppy may be processing several reads in parallel (especially while running on GPU).
- If you are outputting reads to subfolders, check your