How to run this model

[zhyhome]

Thanks for your contribution！
I have some questions about this, your generous reply to me will be sincerely appreciated.
Do I need to execute ./scripts/pretrain.sh first and then ./scripts/finetune_lora.sh ?
The format of the dataset I constructed is as follows, is it right？

    {
        "id": "1",
        "image": "1.jpg",
        "conversations": [
            {
                "from": "human",
                "value": "<image>\n Some Text"
            },
            {
                "from": "gpt",
                "value": "Some Text"
            }
        ]
    },

In this setting, I meet this bug :
/root/miniconda3/envs/llava/lib/python3.10/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
warnings.warn(
/root/miniconda3/envs/llava/lib/python3.10/site-packages/torch/utils/checkpoint.py:61: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
warnings.warn(
Traceback (most recent call last):
File "/root/autodl-tmp/MoVA/mova/train/train_mem.py", line 5, in
train(attn_implementation="flash_attention_2")
File "/root/autodl-tmp/MoVA/mova/train/train.py", line 973, in train
trainer.train()
File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/transformers/trainer.py", line 1539, in train
return inner_training_loop(
File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/transformers/trainer.py", line 1869, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs)
File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/transformers/trainer.py", line 2772, in training_step
loss = self.compute_loss(model, inputs)
File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/transformers/trainer.py", line 2795, in compute_loss
outputs = model(**inputs)
File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
ret_val = func(*args, **kwargs)
File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1833, in forward
loss = self.module(*inputs, **kwargs)
File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/autodl-tmp/MoVA/mova/model/language_model/mova_llama.py", line 87, in forward
) = self.prepare_inputs_labels_for_multimodal(
File "/root/autodl-tmp/MoVA/mova/model/arch.py", line 244, in prepare_inputs_labels_for_multimodal
image_features, cached_features = self.encode_images(images, high_images, flattened_patches, routing_weights, cached_features, prompts)
File "/root/autodl-tmp/MoVA/mova/model/arch.py", line 162, in encode_images
image_features = self.get_model().mm_projector(image_features, routing_weights, prompts)
File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
TypeError: Linear.forward() takes 2 positional arguments but 4 were given

I find this code in MoVA/mova/model/arch.py.

    def encode_images(self, images, high_images, flattened_patches, routing_weights=None, cached_features=None, prompts=None):
        if hasattr(self.get_vision_tower(), "vision_tower_2"):
            image_features, cached_features = self.get_model().get_vision_tower()(images, high_images, flattened_patches, routing_weights, cached_features)
        else:
            image_features = self.get_model().get_vision_tower()(images)
            cached_features = None
        **if isinstance(self.get_model().mm_projector, nn.Sequential):
            if isinstance(image_features, list):
                image_features = image_features[0]
            image_features = self.get_model().mm_projector(image_features)**
        else:
            image_features = self.get_model().mm_projector(image_features, routing_weights, prompts)

I change the "nn.Sequential "to "nn.linear", I got "llava-vicuna-7b-v1.3-finetune_lora" and "llava-vicuna-7b-v1.3-pretrain", but how to test this model ?

Thansk for your contribution!