Multi-GPU inference.
I can run on 2 GPU, but with 3 or 4 GPUs i get errors like these:RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cuda:1)
orRuntimeError: Expected all tensors to be on the same device, but found at least two devices
Why?
I run the inference like this:
checkpoint = "HuggingFaceM4/idefics-80b-instruct"
device = "cuda"
with init_empty_weights():
model = IdeficsForVisionText2Text.from_pretrained(
checkpoint,
torch_dtype=torch.bfloat16,
low_cpu_mem_usage=True,
trust_remote_code=True,
)
model_cache = '/workspace/HF_HOME/hub/models--HuggingFaceM4--idefics-80b-instruct/snapshots/a14d258b1be2a74a3604483de552c33121a98391'
model = load_checkpoint_and_dispatch(
model,
model_cache,
device_map="auto",
)
model = model.eval()
processor = AutoProcessor.from_pretrained(checkpoint)
model = load_checkpoint_and_dispatch(
model,
model_cache,
device_map="auto",
)
processor = AutoProcessor.from_pretrained(checkpoint)
def generate_single(image):
prompt = [
"User:",
image,
(
"Write caption for the image"
"<end_of_utterance>"),
(
"\nAssistant: an icon"
),
]
inputs = processor(prompt, return_tensors="pt").to("cuda")
generated_ids = model.generate(**inputs, bad_words_ids=bad_words_ids, max_length=100)
input_ids = inputs['input_ids']
generated_text = processor.decode(generated_ids[:, input_ids.shape[1]:][0], skip_special_tokens=True)
return f"an icon {generated_text}"
Hello, have you solved the problem? I try to load the model like this:
with init_empty_weights():
self.model = IdeficsForVisionText2Text.from_pretrained(
model_name_or_path,
torch_dtype=torch.bfloat16,
)
self.model = load_checkpoint_and_dispatch(
self.model,
checkpoint,
device_map="auto",
).eval()
self.processor = AutoProcessor.from_pretrained(model_name_or_path)
But all the weights are loaded on cuda:0, and this is strange.🤥
Hello, have you solved the problem? I try to load the model like this:
with init_empty_weights(): self.model = IdeficsForVisionText2Text.from_pretrained( model_name_or_path, torch_dtype=torch.bfloat16, ) self.model = load_checkpoint_and_dispatch( self.model, checkpoint, device_map="auto", ).eval() self.processor = AutoProcessor.from_pretrained(model_name_or_path)But all the weights are loaded on
cuda:0, and this is strange.🤥
I'm so stupid😅, I just set CUDA_VISIBLE_DEVICES=0, and forgot it.
@Lemoncoke I could not even think about CUDA_VISIBLE_DEVICES, almost never used it, so chances of me helping you would be low anyways :(
I'm so stupid😅, I just set CUDA_VISIBLE_DEVICES=0, and forgot it.
haha, glad you found it though @Lemoncoke !