add modelscope and lora testcase #1506
Conversation
zhulinJulia24
commented
Apr 26, 2024
•
zhulinJulia24
commented
- change torch into 2.2.1 version in daily test and benchmark test
- triton>=2.2.0 has a bad performance. Just update torch version in function test. Keep torch 2.1 in benchmark.
- change log lever in benchmark, currently the log is too big(1G+)
- add more supported models in config.yaml
- fix testcase assert in autotest/interface/pipeline/test_pipeline_turbomind_func.py and autotest/interface/restful/test_restful_interface_func_pytorch.py, because of "" tokens in turbomind backend is filterd
- add modelscope testcase in cli chat, pipeline and restful with turbomind and pytorch backend
- add lora testcase in cli chat, pipeline and restful with pytorch backend
- simplify testcase name and move common case into common case and add some more testcase
- some optimization to make the error more clear and save more log info for issue reproduce
|
daily test： https://github.com/zhulinJulia24/lmdeploy/actions/runs/8867146159 |
| @@ -32,12 +32,22 @@ on: | |||
| required: true | |||
| description: 'Dependency packages, you can also set a specific version' | |||
| type: string | |||
| default: 'packaging transformers_stream_generator transformers datasets matplotlib' | |||
| default: 'packaging transformers_stream_generator transformers datasets matplotlib jmespath' | |||
There was a problem hiding this comment.
what's jmespath used for?
There was a problem hiding this comment.
what's
jmespathused for?
by using torch >= 2.2.1 need this packages. just keep for future torch version update.
There was a problem hiding this comment.
Odd. What is it? Which model brings it in?
There was a problem hiding this comment.
Odd. What is it? Which model brings it in?
I check test history . I add it because of this error. It seems only affect profile_generation script.
python3 benchmark/profile_generation.py /nvme/qa_test_models/internlm/internlm2-chat-20b --tp 2 --backend pytorch -c 128 -ct 128 2048 128 -pt 128 128 2048
profiling ... concurrency: 128, n_prompt_token: 128, n_completion_token: 128, test_round: 3, warmup_round: 1
Traceback (most recent call last):
File "/home/zhulin1/lmdeployMainNew/benchmark/profile_generation.py", line 472, in <module>
main()
File "/home/zhulin1/lmdeployMainNew/benchmark/profile_generation.py", line 425, in main
output = _process_map(profile_target, (args.model_path, ))
File "/home/zhulin1/lmdeployMainNew/benchmark/profile_generation.py", line 372, in _process_map
raise ret
ModuleNotFoundError: No module named 'jmespath'
There was a problem hiding this comment.
I think it not related to torch 2.2.1 version. It's related to latest main code?
| @@ -119,7 +130,7 @@ jobs: | |||
| ref: ${{github.event.inputs.repo_ref || 'main'}} | |||
| - name: Copy repository - offline | |||
| if: ${{inputs.offline_mode}} | |||
| run: cp -r /nvme/qa_test_models/offline_pkg/lmdeploy/* . | |||
| run: cp -r /nvme/qa_test_models/offline_pkg/lmdeploy/. . | |||
There was a problem hiding this comment.
what's the motivation of changing * to .?
There was a problem hiding this comment.
cp -r /* does not copy .github folder. cp -r /. will copy .X file and folders too. It will fix this error
| @@ -52,6 +53,7 @@ emoji_case: | |||
| - 赞 | |||
| - 好 | |||
| - '!' | |||
| - u1f44d | |||
There was a problem hiding this comment.
this case is not stable, sometimes it will output unicode string，this is uincode string of 👍
| - name: Install lmdeploy - dependency | ||
| run: | | ||
| python3 -m pip install ${{inputs.dependency_pkgs || 'packaging transformers_stream_generator transformers datasets matplotlib openai'}} | ||
| python3 -m pip install ${{inputs.dependency_pkgs || 'packaging transformers_stream_generator transformers datasets matplotlib openai attrdict timm modelscope jmespath'}} |
There was a problem hiding this comment.
Could we put the extra dependencies into a requirements.txt and only change it once for all test pipelines? It will be more easy to manage. and only need to change once.
| @@ -1,7 +1,7 @@ | |||
| allure-pytest | |||
| coverage | |||
| pynvml | |||
| pytest==8.0.2 | |||
| pytest | |||
There was a problem hiding this comment.
why unlock the version of pytest? Is the version issue resolved?
There was a problem hiding this comment.
why unlock the version of pytest? Is the version issue resolved?
this condition is added for bug on version 8.1 pytest-dev/pytest#12069
this bug is fixed, I want to remove the restriction,
