Spaces:
Running
No CUDA GPUs are available — persists after fixing all code-level causes
Space: https://huggingface.co/spaces/HateezAbullah/dawn-news-rag-chatbot
I'm getting "No CUDA GPUs are available" inside worker_init, triggered as soon as a real @spaces.GPU-decorated function is called from a user request. This happens consistently, every single time, not intermittently.
Traceback:
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/spaces/zero/wrappers.py", line 148, in worker_init
torch.init(nvidia_uuid)
File "/usr/local/lib/python3.12/site-packages/spaces/zero/torch/patching.py", line 417, in init
torch.Tensor([0]).cuda()
File "/usr/local/lib/python3.12/site-packages/torch/cuda/init.py", line 478, in _lazy_init
torch._C._cuda_init()
RuntimeError: No CUDA GPUs are available
What I've already ruled out:
- Code follows the official pattern exactly: model loaded to 'cuda' at module/root level, generation wrapped in @spaces.GPU(duration=30), import spaces placed before other ML imports
- No undecorated CUDA calls anywhere else in the app — confirmed by explicitly forcing my embedding model to device="cpu" so the only CUDA usage is inside the decorated function
- Tried "Restart this Space" — no change
- Tried pinning specific torch/spaces versions — no change
- Status page shows all services green, not a platform-wide outage
This looks like a worker/GPU-attachment issue specific to this Space rather than anything in my app code. Would appreciate help diagnosing — happy to share full app.py if useful.
For now.
Update: root cause identified
After further isolation, this does not appear to be a Space-specific GPU allocation failure.
The trigger is a CUDA runtime query performed by Transformers 5 during the CPU-only E5 forward:
Transformers mask preparation
→ is_tracing()
→ torch.cuda.is_current_stream_capturing()
At that point, the ZeroGPU parent has no GPU assigned. The CUDA exception is caught by Transformers, but the next forked ZeroGPU worker then fails to initialize its assigned MIG device with:
RuntimeError: No CUDA GPUs are available
I confirmed the cause in both directions:
- calling
torch.cuda.is_current_stream_capturing()in the GPU-less parent reproduces the failure; - skipping only that query allows CPU embedding followed by ZeroGPU inference to work.
A minimal application-level workaround has been proposed here:
https://huggingface.co/spaces/HateezAbullah/dawn-news-rag-chatbot/discussions/1
The PR changes only app.py and skips the stream-capture query only in the GPU-less ZeroGPU parent. GPU workers retain the original Transformers behavior.
The PR is still pending, so I will report the final Space result after it is tested or merged.
A possible upstream fix would be for Transformers to return False without querying the CUDA runtime when CUDA has not been initialized. ZeroGPU may also benefit from more general protection or diagnostics when the parent process has already touched the CUDA runtime.