-
Thanks for your help! |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 8 replies
-
Hello, good question!
So batch size is at the application level, while ubatch size is at the device level. batch_size >= ubatch_size. You can find some references here: |
Beta Was this translation helpful? Give feedback.
-
@phymbert |
Beta Was this translation helpful? Give feedback.
The default values are here:
https://github.com/ggerganov/llama.cpp/blob/557410b8f06380560155ac7fcb8316d71ddc9837/common/common.h#L57