Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2025-01-18-llava_v1.5_7b_Q4_0_gguf_en #14501

Open
wants to merge 1 commit into
base: models_hub
Choose a base branch
from
Open

2025-01-18-llava_v1.5_7b_Q4_0_gguf_en #14501

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
172 changes: 172 additions & 0 deletions docs/_posts/DevinTDHa/2025-01-18-llava_v1.5_7b_Q4_0_gguf_en.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,172 @@
---
layout: model
title: LLaVA v1.5 7B Q4 GGUF
author: John Snow Labs
name: llava_v1.5_7b_Q4_0_gguf
date: 2025-01-18
tags: [gguf, llamacpp, llava, en, quantized, open_source]
task: Image Captioning
language: en
edition: Spark NLP 6.0.0
spark_version: 3.0
supported: true
engine: llamacpp
annotator: AutoGGUFVisionModel
article_header:
type: cover
use_language_switcher: "Python-Scala-Java"
---

## Description

LLaVA is an open-source chatbot trained by fine-tuning LLaMA/Vicuna on GPT-generated multimodal instruction-following data. It is an auto-regressive language model, based on the transformer architecture.

Originally from https://huggingface.co/Mozilla/llava-v1.5-7b-llamafile

## Predicted Entities



{:.btn-box}
<button class="button button-orange" disabled>Live Demo</button>
<button class="button button-orange" disabled>Open in Colab</button>
[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/models/llava_v1.5_7b_Q4_0_gguf_en_6.0.0_3.0_1737207768652.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
[Copy S3 URI](s3://auxdata.johnsnowlabs.com/public/models/llava_v1.5_7b_Q4_0_gguf_en_6.0.0_3.0_1737207768652.zip){:.button.button-orange.button-orange-trans.button-icon.button-copy-s3}

## How to use



<div class="tabs-box" markdown="1">
{% include programmingLanguageSelectScalaPythonNLU.html %}
```python
import sparknlp
from sparknlp.base import *
from sparknlp.annotator import *
from pyspark.ml import Pipeline
from pyspark.sql.functions import lit

documentAssembler = DocumentAssembler() \
.setInputCol("caption") \
.setOutputCol("caption_document")
imageAssembler = ImageAssembler() \
.setInputCol("image") \
.setOutputCol("image_assembler")

imagesPath = "src/test/resources/image/"
data = ImageAssembler \
.loadImagesAsBytes(spark, imagesPath) \
.withColumn("caption", lit("Caption this image.")) # Add a caption to each image.

nPredict = 40
model = AutoGGUFVisionModel.pretrained() \
.setInputCols(["caption_document", "image_assembler"]) \
.setOutputCol("completions") \
.setBatchSize(4) \
.setNGpuLayers(99) \
.setNCtx(4096) \
.setMinKeep(0) \
.setMinP(0.05) \
.setNPredict(nPredict) \
.setNProbs(0) \
.setPenalizeNl(False) \
.setRepeatLastN(256) \
.setRepeatPenalty(1.18) \
.setStopStrings(["</s>", "Llama:", "User:"]) \
.setTemperature(0.05) \
.setTfsZ(1) \
.setTypicalP(1) \
.setTopK(40) \
.setTopP(0.95)

pipeline = Pipeline().setStages([documentAssembler, imageAssembler, model])
pipeline.fit(data).transform(data) \
.selectExpr("reverse(split(image.origin, '/'))[0] as image_name", "completions.result") \
.show(truncate = False)


```
```scala
import com.johnsnowlabs.nlp.ImageAssembler
import com.johnsnowlabs.nlp.annotator._
import com.johnsnowlabs.nlp.base._
import org.apache.spark.ml.Pipeline
import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.functions.lit

val documentAssembler = new DocumentAssembler()
.setInputCol("caption")
.setOutputCol("caption_document")

val imageAssembler = new ImageAssembler()
.setInputCol("image")
.setOutputCol("image_assembler")

val imagesPath = "src/test/resources/image/"
val data: DataFrame = ImageAssembler
.loadImagesAsBytes(ResourceHelper.spark, imagesPath)
.withColumn("caption", lit("Caption this image.")) // Add a caption to each image.

val nPredict = 40
val model = AutoGGUFVisionModel.pretrained()
.setInputCols("caption_document", "image_assembler")
.setOutputCol("completions")
.setBatchSize(4)
.setNGpuLayers(99)
.setNCtx(4096)
.setMinKeep(0)
.setMinP(0.05f)
.setNPredict(nPredict)
.setNProbs(0)
.setPenalizeNl(false)
.setRepeatLastN(256)
.setRepeatPenalty(1.18f)
.setStopStrings(Array("</s>", "Llama:", "User:"))
.setTemperature(0.05f)
.setTfsZ(1)
.setTypicalP(1)
.setTopK(40)
.setTopP(0.95f)

val pipeline = new Pipeline().setStages(Array(documentAssembler, imageAssembler, model))
pipeline
.fit(data)
.transform(data)
.selectExpr("reverse(split(image.origin, '/'))[0] as image_name", "completions.result")
.show(truncate = false)

```
</div>

## Results

```bash
+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|image_name |result |
+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|palace.JPEG |[ The image depicts a large, ornate room with high ceilings and beautifully decorated walls. There are several chairs placed throughout the space, some of which have cushions] |
|egyptian_cat.jpeg|[ The image features two cats lying on a pink surface, possibly a bed or sofa. One cat is positioned towards the left side of the scene and appears to be sleeping while holding] |
|hippopotamus.JPEG|[ A large brown hippo is swimming in a body of water, possibly an aquarium. The hippo appears to be enjoying its time in the water and seems relaxed as it floats] |
|hen.JPEG |[ The image features a large chicken standing next to several baby chickens. In total, there are five birds in the scene: one adult and four young ones. They appear to be gathered together] |
|ostrich.JPEG |[ The image features a large, long-necked bird standing in the grass. It appears to be an ostrich or similar species with its head held high and looking around. In addition to] |
|junco.JPEG |[ A small bird with a black head and white chest is standing on the snow. It appears to be looking at something, possibly food or another animal in its vicinity. The scene takes place out] |
|bluetick.jpg |[ A dog with a red collar is sitting on the floor, looking at something. The dog appears to be staring into the distance or focusing its attention on an object in front of it.] |
|chihuahua.jpg |[ A small brown dog wearing a sweater is sitting on the floor. The dog appears to be looking at something, possibly its owner or another animal in the room. It seems comfortable and relaxed]|
|tractor.JPEG |[ A man is sitting in the driver's seat of a green tractor, which has yellow wheels and tires. The tractor appears to be parked on top of an empty field with] |
|ox.JPEG |[ A large bull with horns is standing in a grassy field.] |
+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
```

{:.model-param}
## Model Information

{:.table-model}
|---|---|
|Model Name:|llava_v1.5_7b_Q4_0_gguf|
|Compatibility:|Spark NLP 6.0.0+|
|License:|Open Source|
|Edition:|Official|
|Input Labels:|[caption_document, image_assembler]|
|Output Labels:|[completions]|
|Language:|en|
|Size:|4.2 GB|