Skip to content

DavidZyy/tensorLib

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

tensorLib

TensorLib is a tensor library for inference of large language models. It supports cpu and cuda backend.

pic

Screenshot

Build

mkdir build
cd build
cmake ..
make

Make sure you have installed cuda, openmp and other dependencies.

Debug

Uncomment set(CMAKE_BUILD_TYPE Debug) in CmakeLists.txt.

Test

C++ or python frameworks is used to test the code.

python test

I have use pybind11 to wrap the c++ code to python. So you can use python to test the code. Pytorch and numpy is used to test the correctness of the Tensor functions.

Model

Llama2-7b is supported now. see llama2.c about how to acquire the model weghts.

The script which can export fp16 weights is in script folder. It is modified from above project.

Acknowledgement

This project inspired by following projects:

llama2.c

CMU10_414

llm.c

llama.cpp

simpleTensor

Miscellaneous

This project is in very early stage. Limmited by my knowledge and time, I have not considered many details. So some inplementation details are not well thought out and not elegant. But I will try to improve it in the future.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published