HELP BLACKWELL ERROR
#9
by BLOOD-IRON - opened
https://github.com/nicolab28/ComfyUI-ClipProj/issues/11
[INFO] got prompt
[INFO] [ClipProj] graph watch: 1 pinned (135), 23 reachable nodes, 0 released
[INFO] [ClipProj] encoder qwen3vl_4b_fp8_scaled.safetensors on cuda:0 moved back to RAM (4.88 GB), ComfyUI had lost track of it
[MultiGPU_Memory_Management] Triggering PromptExecutor cache reset. Reason: forced_soft_empty
[MultiGPU_Memory_Management] Triggering PromptExecutor cache reset. Reason: forced_soft_empty
[INFO] [ClipProj] encoder qwen3vl_4b_fp8_scaled.safetensors on cuda:0 unloaded (4.88 GB freed)
[INFO] Found quantization metadata version 1
[MultiGPU Core Patching] text_encoder_device_patched returning device: cuda:0 (current_text_encoder_device=cuda:0)
[INFO] Using MixedPrecisionOps for text encoder
[INFO] Model storage policy: fast_disk=False paths=['/home/ubuntu/NVME_DATA/COMFYUI_MODEL/models/text_encoders/qwen3vl_4b_fp8_scaled.safetensors']
[INFO] Requested to load Krea2TEModel_
[INFO] loaded completely; 4999.47 MB loaded, full load: True
[INFO] CLIP/text encoder model load device: cuda:0, offload device: cuda:0, current: cuda:0, dtype: torch.float16
[INFO] [ClipProj] encoder qwen3vl_4b_fp8_scaled.safetensors on cuda:0 pinned on cuda:0 (ComfyUI will not move it)
[INFO] [ClipProj] qwen3vl_4b_fp8_scaled.safetensors (krea2 [4B detected]) loaded in resident mode on cuda:0: 4.88 GB
[INFO] [ClipProj] mmh3-4b-ClipProj-v3.1.safetensors | tap 24 | 2560 -> 5120 | cos_test 0.6874
[INFO] Requested to load Krea2TEModel_
[INFO] loaded completely; 14225.73 MB usable, 4999.47 MB loaded, full load: True
[INFO] Requested to load MiniMaxH3VideoVAE
[INFO] loaded completely; 14480.11 MB usable, 2665.86 MB loaded, full load: True
[MultiGPU P2P] can_access_peer(1, 0) = False
[MultiGPU DLPack] CPU-staging tensor from cuda:1 to cuda:0 (P2P unavailable)
[MultiGPU CUDA Guard] Switching CUDA current device 1 -> 0 (comfy_kitchen._wrap_for_dlpack(staged))
[MultiGPU CUDA Guard] Restored CUDA current device 0 -> 1 (comfy_kitchen._wrap_for_dlpack(staged))
[MultiGPU DLPack] CPU-staging tensor from cuda:1 to cuda:0 (P2P unavailable)
[MultiGPU CUDA Guard] Switching CUDA current device 1 -> 0 (comfy_kitchen._wrap_for_dlpack(staged))
[MultiGPU CUDA Guard] Restored CUDA current device 0 -> 1 (comfy_kitchen._wrap_for_dlpack(staged))
[MultiGPU DLPack] CPU-staging tensor from cuda:1 to cuda:0 (P2P unavailable)
[MultiGPU CUDA Guard] Switching CUDA current device 1 -> 0 (comfy_kitchen._wrap_for_dlpack(staged))
[MultiGPU CUDA Guard] Restored CUDA current device 0 -> 1 (comfy_kitchen._wrap_for_dlpack(staged))
[MultiGPU DLPack] CPU-staging tensor from cuda:1 to cuda:0 (P2P unavailable)
[MultiGPU CUDA Guard] Switching CUDA current device 1 -> 0 (comfy_kitchen._wrap_for_dlpack(staged))
[MultiGPU CUDA Guard] Restored CUDA current device 0 -> 1 (comfy_kitchen._wrap_for_dlpack(staged))
[MultiGPU DLPack] CPU-staging tensor from cuda:1 to cuda:0 (P2P unavailable)
[MultiGPU CUDA Guard] Switching CUDA current device 1 -> 0 (comfy_kitchen._wrap_for_dlpack(staged))
[MultiGPU CUDA Guard] Restored CUDA current device 0 -> 1 (comfy_kitchen._wrap_for_dlpack(staged))
[ERROR] !!! Exception during processing !!! CUDA error: an illegal memory access was encountered
Search for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
[ERROR] Traceback (most recent call last):
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/execution.py", line 547, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/execution.py", line 352, in get_output_data
return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/custom_nodes/comfyui-lora-manager/py/metadata_collector/metadata_hook.py", line 177, in async_map_node_over_list_with_metadata
results = await original_map_node_over_list(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/execution.py", line 326, in _async_map_node_over_list
await process_inputs(input_dict, i)
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/execution.py", line 314, in process_inputs
result = f(**inputs)
^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/comfy_api/internal/__init__.py", line 149, in wrapped_func
return method(locked_class, **inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/comfy_api/latest/_io.py", line 2053, in EXECUTE_NORMALIZED
to_return = cls.execute(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/comfy_extras/nodes_minimax_h3.py", line 160, in execute
kf["latent"] = vae.encode(kf.pop("image"))
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/comfy/sd.py", line 1408, in encode
model_management.raise_non_oom(e)
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/comfy/model_management.py", line 399, in raise_non_oom
raise e
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/comfy/sd.py", line 1398, in encode
out = self.first_stage_model.encode(pixels_in, device=self.device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/comfy/ldm/minimax/vae.py", line 775, in encode
moments = self._adaptive_encode(self._normalize_pixels(x.to(device)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/comfy/ldm/minimax/vae.py", line 497, in _adaptive_encode
return self.tiled_encode(x)
^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/comfy/ldm/minimax/vae.py", line 566, in tiled_encode
row.append(self._encode_moments(tile))
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/comfy/ldm/minimax/vae.py", line 473, in _encode_moments
return self.quant_conv(self.encoder(x))
^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/miniconda3/envs/COMFYUI_CP312_TR210_CU130/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/miniconda3/envs/COMFYUI_CP312_TR210_CU130/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/comfy/ldm/minimax/vae.py", line 222, in forward
h = self.down[i_level].block[i_block](h)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/miniconda3/envs/COMFYUI_CP312_TR210_CU130/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/miniconda3/envs/COMFYUI_CP312_TR210_CU130/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/comfy/ldm/minimax/vae.py", line 170, in forward
h = self.conv1(x, pre_norm=self.norm1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/miniconda3/envs/COMFYUI_CP312_TR210_CU130/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/miniconda3/envs/COMFYUI_CP312_TR210_CU130/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/comfy/ldm/minimax/vae.py", line 103, in forward
out = super().forward(x, autopad="causal_zero")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/comfy/ops.py", line 631, in forward
return self.forward_comfy_cast_weights(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/comfy/ops.py", line 626, in forward_comfy_cast_weights
return self._conv_forward(input, weight, bias, autopad=autopad)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/comfy/ops.py", line 622, in _conv_forward
return super()._conv_forward(input, weight, bias, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/miniconda3/envs/COMFYUI_CP312_TR210_CU130/lib/python3.12/site-packages/torch/nn/modules/conv.py", line 718, in _conv_forward
return F.conv3d(
^^^^^^^^^
torch.AcceleratorError: CUDA error: an illegal memory access was encountered
Search for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
[INFO] Prompt executed in 9.13 seconds
[WARNING] [ClipProj] could not move encoder qwen3vl_4b_fp8_scaled.safetensors on cuda:0 back to RAM: CUDA error: an illegal memory access was encountered
Search for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Exception in thread Thread-2 (prompt_worker):
Traceback (most recent call last):
File "/home/ubuntu/NVME_DATA/miniconda3/envs/COMFYUI_CP312_TR210_CU130/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
self.run()
File "/home/ubuntu/NVME_DATA/miniconda3/envs/COMFYUI_CP312_TR210_CU130/lib/python3.12/threading.py", line 1012, in run
self._target(*self._args, **self._kwargs)
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/main.py", line 390, in prompt_worker
comfy.model_management.unload_all_models()
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/custom_nodes/comfyui-clipproj/clipproj_pinning.py", line 306, in unload_all_models
release_all()
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/custom_nodes/comfyui-clipproj/clipproj_pinning.py", line 183, in release_all
got = unload_patcher(ref(), label)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/custom_nodes/comfyui-clipproj/clipproj_pinning.py", line 126, in unload_patcher
mm.soft_empty_cache(force=True)
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/custom_nodes/ComfyUI-MultiGPU/device_utils.py", line 259, in soft_empty_cache_distorch2_patched
original_soft_empty_cache(force)
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/custom_nodes/comfyui-multigpu/device_utils.py", line 259, in soft_empty_cache_distorch2_patched
original_soft_empty_cache(force)
File "/home/ubuntu/NVME_DATA/COMFYUI_CP312_TR210_CU130/comfy/model_management.py", line 2110, in soft_empty_cache
torch.cuda.synchronize()
File "/home/ubuntu/NVME_DATA/miniconda3/envs/COMFYUI_CP312_TR210_CU130/lib/python3.12/site-packages/torch/cuda/__init__.py", line 1108, in synchronize
return torch._C._cuda_synchronize()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.AcceleratorError: CUDA error: an illegal memory access was encountered
Search for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.