Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pull] master from kserve:master #194

Merged
merged 17 commits into from
Feb 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
1bfba02
Pass missing infer parameters during conversion (#3368)
sivanantha321 Jan 26, 2024
7a9f7f7
feat: Support S3 transfer acceleration (#3305)
terrytangyuan Jan 27, 2024
4bfa60a
Implement HuggingFace model server (#3334)
yuzisun Jan 27, 2024
9189831
fix: Add missing volume mount to transformer container when using mod…
rhuss Jan 27, 2024
8a0df88
align cloudevents/sdk-go dependency (#3387)
spolti Jan 27, 2024
41f0883
Update google-cloud-storage dependecy to >=2.3.0,<3.0.0 and ray depen…
sivanantha321 Jan 31, 2024
ad88ab4
Adding script for setting up Telepresence for local development (#3104)
rhuss Feb 3, 2024
6f4b7a9
Fix v2 model ready url in kserve client (#3403)
sivanantha321 Feb 3, 2024
0b2c7d9
chore: Remove unused version update script for Python SDK (#3394)
terrytangyuan Feb 6, 2024
d977cc0
Fix no space left error in CI environment (#3411)
sivanantha321 Feb 6, 2024
5172dc8
Add exception handler for model server and Add ability to specify cus…
sivanantha321 Feb 7, 2024
b162b35
Streamline python release update script to scan multi-depth (#3351)
andyi2it Feb 9, 2024
5c452e7
feat: Add HuggingFace runtime out-of-the-box support (#3395)
terrytangyuan Feb 10, 2024
e21b913
docs: Update link to Slack instructions (#3421)
terrytangyuan Feb 12, 2024
d59d14e
Fix no space left error on huggingface docker publisher workflow (#3431)
sivanantha321 Feb 14, 2024
4237c1b
Fix parameters value type conversion by pydantic (#3430)
sivanantha321 Feb 14, 2024
d3ec987
Fix Raw Logger E2E (#3434)
israel-hdez Feb 15, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 22 additions & 4 deletions .github/actions/free-up-disk-space/action.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: 'Free-up disk space action'
description: 'Removes non-essential tools, libraries and cached files from GitHub action runner node'
description: 'Removes non-essential tools, libraries and cached files from GitHub action runner node and changes the docker data directory to /mnt/docker'

runs:
using: "composite"
Expand All @@ -8,7 +8,7 @@ runs:
shell: bash
run: |
echo "Disk usage before cleanup:"
df -h
df -hT

# remove non-essential tools and libraries, see:
# https://github.com/actions/runner-images/issues/2840#issuecomment-790492173
Expand All @@ -23,12 +23,30 @@ runs:
sudo rm -rf /usr/share/swift

echo "Disk usage after cleanup:"
df -h
df -hT

- name: Prune docker images
shell: bash
run: |
echo "Pruning docker images"
docker image prune -a -f
docker system df
df -h
df -hT

- name: Move docker data directory
shell: bash
run: |
echo "Stopping docker service ..."
sudo systemctl stop docker
DOCKER_DEFAULT_ROOT_DIR=/var/lib/docker
DOCKER_ROOT_DIR=/mnt/docker
echo "Moving ${DOCKER_DEFAULT_ROOT_DIR} -> ${DOCKER_ROOT_DIR}"
sudo mv ${DOCKER_DEFAULT_ROOT_DIR} ${DOCKER_ROOT_DIR}
echo "Creating symlink ${DOCKER_DEFAULT_ROOT_DIR} -> ${DOCKER_ROOT_DIR}"
sudo ln -s ${DOCKER_ROOT_DIR} ${DOCKER_DEFAULT_ROOT_DIR}
echo "$(sudo ls -l ${DOCKER_DEFAULT_ROOT_DIR})"
echo "Starting docker service ..."
sudo systemctl daemon-reload
sudo systemctl start docker
echo "Docker service status:"
sudo systemctl --no-pager -l -o short status docker
87 changes: 87 additions & 0 deletions .github/workflows/huggingface-docker-publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
name: Huggingface Docker Publisher

on:
push:
# Publish `master` as Docker `latest` image.
branches:
- master

# Run tests for any PRs.
pull_request:

env:
IMAGE_NAME: huggingfaceserver

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
# Run tests.
# See also https://docs.docker.com/docker-hub/builds/automated-testing/
test:
runs-on: ubuntu-latest

steps:
- name: Checkout source
uses: actions/checkout@v2

- name: Free-up disk space
uses: ./.github/actions/free-up-disk-space

- name: Run tests
run: |
if [ -f docker-compose.test.yml ]; then
docker-compose --file docker-compose.test.yml build
docker-compose --file docker-compose.test.yml run sut
else
cd python
docker buildx build . --file huggingface_server.Dockerfile
fi

# Push image to GitHub Packages.
# See also https://docs.docker.com/docker-hub/builds/
push:
# Ensure test job passes before pushing image.
needs: test

runs-on: ubuntu-latest
if: github.event_name == 'push'

steps:
- name: Checkout source
uses: actions/checkout@v2

- name: Free-up disk space
uses: ./.github/actions/free-up-disk-space

- name: Build image
run: |
cd python
docker buildx build . --file huggingface_server.Dockerfile --tag $IMAGE_NAME

- name: Log into registry
run: docker login -u ${{ secrets.DOCKER_USER }} -p ${{ secrets.DOCKER_PASSWORD }}

- name: Push image
run: |
IMAGE_ID=kserve/$IMAGE_NAME

# Change all uppercase to lowercase
IMAGE_ID=$(echo $IMAGE_ID | tr '[A-Z]' '[a-z]')

# Strip git ref prefix from version
VERSION=$(echo "${{ github.ref }}" | sed -e 's,.*/\(.*\),\1,')

# Strip "v" prefix from tag name
# [[ "${{ github.ref }}" == "refs/tags/"* ]] && VERSION=$(echo $VERSION | sed -e 's/^v//')

# Use Docker `latest` tag convention
[ "$VERSION" == "master" ] && VERSION=latest

echo IMAGE_ID=$IMAGE_ID
echo VERSION=$VERSION

docker tag $IMAGE_NAME $IMAGE_ID:$VERSION
docker push $IMAGE_ID:$VERSION

56 changes: 53 additions & 3 deletions .github/workflows/python-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,14 +29,19 @@ jobs:
uses: py-actions/flake8@v1
with:
max-line-length: "120"
exclude: "docs/samples/v1beta1/onnx/assets/*.py,python/kserve/test/test_v1*.py,python/kserve/kserve/__init__.py,python/kserve/test/test_knative*.py,python/kserve/kserve/protocol/grpc/grpc_predict_v2*.py"
exclude: "docs/samples/v1beta1/onnx/assets/*.py,python/kserve/kserve/*.py,python/kserve/test/test_v1*.py,python/kserve/kserve/__init__.py,python/kserve/test/__init__.py,python/kserve/test/test_knative*.py,python/kserve/kserve/protocol/grpc/grpc_predict_v2*.py"
build:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11"]
steps:
- uses: actions/checkout@v2
- name: Checkout source
uses: actions/checkout@v2

- name: Free-up disk space
uses: ./.github/actions/free-up-disk-space

- name: Set up Python ${{ matrix.python-version }}
id: setup-python
uses: actions/setup-python@v4
Expand All @@ -46,12 +51,19 @@ jobs:
- name: Install Poetry and version plugin
run: ./test/scripts/gh-actions/setup-poetry.sh

- name: Configure Poetry cache
run: |
sudo mkdir -p /mnt/poetry
# change permission so that poetry can install without sudo
sudo chown -R $USER /mnt/poetry
poetry config cache-dir /mnt/poetry

# load poetry cache if cache exists
- name: Load poetry cache
id: cached-poetry-dependencies
uses: actions/cache@v3
with:
path: ~/.cache/pypoetry
path: /mnt/poetry
key: poetry-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}

# ----------------------------------------Kserve Unit Tests--------------------------------------------------------
Expand Down Expand Up @@ -221,3 +233,41 @@ jobs:
cd python
source alibiexplainer/.venv/bin/activate
pytest --cov=alibiexplainer ./alibiexplainer

# ----------------------------------------Huggingface Server Unit Tests------------------------------------------------
# load cached huggingface venv if cache exists
- name: Load cached huggingface venv
if: ${{ !startsWith(steps.setup-python.outputs.python-version, '3.8') }}
id: huggingface-dependencies
uses: actions/cache@v3
with:
path: /mnt/python/huggingfaceserver-venv
key: huggingface-venv-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/kserve/poetry.lock', '**/huggingfaceserver/poetry.lock') }}
# install huggingface server dependencies if cache does not exist
- name: Configure poetry for huggingface server
if: ${{ !startsWith(steps.setup-python.outputs.python-version, '3.8') }}
run: |
poetry config virtualenvs.path /mnt/python/huggingfaceserver-venv
poetry config virtualenvs.in-project false
- name: Install huggingface dependencies
if: ${{ steps.cached-huggingface-dependencies.outputs.cache-hit != 'true' && !startsWith(steps.setup-python.outputs.python-version, '3.8') }}
run: |
sudo mkdir -p /mnt/python/huggingfaceserver-venv
# change permission so that poetry can install without sudo
sudo chown -R $USER /mnt/python/huggingfaceserver-venv
cd python/huggingfaceserver
make install_dependencies
- name: Install huggingface server
if: ${{ !startsWith(steps.setup-python.outputs.python-version, '3.8') }}
run: |
cd python/huggingfaceserver
make dev_install
- name: Test huggingfaceserver
if: ${{ !startsWith(steps.setup-python.outputs.python-version, '3.8') }}
run: |
cd python/huggingfaceserver
poetry run -- pytest --cov=huggingfaceserver

- name: Free space after tests
run: |
df -hT
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
[![OpenSSF Best Practices](https://bestpractices.coreinfrastructure.org/projects/6643/badge)](https://bestpractices.coreinfrastructure.org/projects/6643)
[![Releases](https://img.shields.io/github/release-pre/kserve/kserve.svg?sort=semver)](https://github.com/kserve/kserve/releases)
[![LICENSE](https://img.shields.io/github/license/kserve/kserve.svg)](https://github.com/kserve/kserve/blob/master/LICENSE)
[![Slack Status](https://img.shields.io/badge/slack-join_chat-white.svg?logo=slack&style=social)](https://kubeflow.slack.com/archives/CH6E58LNP)
[![Slack Status](https://img.shields.io/badge/slack-join_chat-white.svg?logo=slack&style=social)](https://github.com/kserve/community/blob/main/README.md#questions-and-issues)

KServe provides a Kubernetes [Custom Resource Definition](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) for serving machine learning (ML) models on arbitrary frameworks. It aims to solve production model serving use cases by providing performant, high abstraction interfaces for common ML frameworks like Tensorflow, XGBoost, ScikitLearn, PyTorch, and ONNX.

Expand Down
29 changes: 29 additions & 0 deletions charts/kserve-resources/templates/clusterservingruntimes.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -344,3 +344,32 @@ spec:
limits:
cpu: "1"
memory: 2Gi
---
apiVersion: serving.kserve.io/v1alpha1
kind: ClusterServingRuntime
metadata:
name: kserve-huggingfaceserver
spec:
annotations:
prometheus.kserve.io/port: '8080'
prometheus.kserve.io/path: "/metrics"
supportedModelFormats:
- name: huggingface
version: "1"
autoSelect: true
priority: 1
protocolVersions:
- v2
containers:
- name: kserve-container
image: "{{ .Values.kserve.servingruntime.huggingfaceserver.image }}:{{ .Values.kserve.servingruntime.huggingfaceserver.tag }}"
args:
- --model_id={{.Name}}
- --model_dir=/mnt/models
resources:
requests:
cpu: "1"
memory: 2Gi
limits:
cpu: "1"
memory: 2Gi
3 changes: 3 additions & 0 deletions charts/kserve-resources/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,9 @@ kserve:
xgbserver:
image: kserve/xgbserver
tag: *defaultVersion
huggingfaceserver:
image: kserve/huggingfaceserver
tag: *defaultVersion
tritonserver:
image: nvcr.io/nvidia/tritonserver
tag: 23.05-py3
Expand Down
5 changes: 5 additions & 0 deletions config/configmap/inferenceservice.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,7 @@ data:
"s3Region": "",
"s3VerifySSL": "",
"s3UseVirtualBucket": "",
"s3UseAccelerate": "",
"s3UseAnonymousCredential": "",
"s3CABundle": ""
}
Expand Down Expand Up @@ -186,6 +187,9 @@ data:

# s3UseVirtualBucket configures whether it is a virtual bucket or not.
"s3UseVirtualBucket": "",

# s3UseAccelerate configures whether to use transfer acceleration.
"s3UseAccelerate": "",

# s3UseAnonymousCredential configures whether to use anonymous credentials to download the model or not.
"s3UseAnonymousCredential": "",
Expand Down Expand Up @@ -489,6 +493,7 @@ data:
"s3Region": "",
"s3VerifySSL": "",
"s3UseVirtualBucket": "",
"s3UseAccelerate": "",
"s3UseAnonymousCredential": "",
"s3CABundle": ""
}
Expand Down
4 changes: 4 additions & 0 deletions config/crd/patches/protocol.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@
path: /spec/validation/openAPIV3Schema/properties/spec/properties/predictor/properties/onnx/properties/ports/items/required/1
value: protocol

- op: add
path: /spec/validation/openAPIV3Schema/properties/spec/properties/predictor/properties/huggingface/properties/ports/items/required/1
value: protocol

- op: add
path: /spec/validation/openAPIV3Schema/properties/spec/properties/predictor/properties/sklearn/properties/ports/items/required/1
value: protocol
Expand Down
Loading