Feature Request: Support Qwen VL?

[Bojun-Feng]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Is it possible to add support for the Qwen VL models?

The special thing about Qwen VL is that it supports drawing "boxes" to identify objects in images. It would be great to have that in the web UI. Here is a crude example from Hugging Face:

https://huggingface.co/Qwen/Qwen-VL#%F0%9F%A4%97-transformers

Motivation

Qwen VL is a open source SOTA light weight vision model, the model got 47k downloads in just last month on Hugging Face.

Possible Implementation

Llama.cpp has a hacky solution, not sure if it is sustainable though: ggml-org/llama.cpp#10361

Here is an example of a working GGUF file: https://www.reddit.com/r/LocalLLaMA/comments/1he4ars/comment/m22kree

[cjpais]

Right now without syncing upstream, this will be quite hard to add.