LlamaIndex
Log in

Find answers from the community

Updated 2 years ago

GPU/CPU Choice

VVaylonn
·
Is there a part of the code we need to specify if we want to use gpu instead of cpu ? Or it depends directly of the llm we are using
V
L
10 comments
VVaylonn
ok maybe found the answer
VVaylonn
it's in the llm_predictor, then in his args there is device_map
VVaylonn
Attachment
image.png
VVaylonn
GPU/CPU Choice
VVaylonn
with this explanation:
Attachment
image.png
LLogan M
Yup you got it!
VVaylonn
tried to do device_map = "cuda"

i get the error:
VVaylonn
which doesnt fit the doc ?
LLogan M
Tbh I think I misunderstood what device map does when I wrote the doc lol

Try loading the model directly and moving it to cuda yourself? Then you can pass the model in directly
LLogan M
Rather than model_name
Add a reply
Sign up and join the conversation on Discord