[release] Port KubeRay CI tests to Ray CI

[kevin85421]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Whenever KubeRay tries to bump the default Ray version to a newer release, it's quite painful due to Ray's fast-paced release pattern, and each minor update may introduce breaking changes. Therefore, if we can port KubeRay end-to-end tests to Ray CI to run tests for each commit, we can catch some breaking changes at an early stage.

[P0] Step 1: Port existing KubeRay e2e tests to Ray CI.

• This should be very simple because it only requires a Kind cluster.
• https://github.com/ray-project/kuberay/blob/master/.buildkite/test-e2e.yml
• https://github.com/ray-project/kuberay/blob/master/.buildkite/test-sample-yamls.yml

[P1] Step 2: Add conformance tests in Ray CI.

• Summarize the Ray APIs and behaviors that KubeRay relies on, and add conformance tests to Ray CI to ensure contract compliance. For example, KubeRay uses an endpoint in the Ray dashboard to check GCS health and an endpoint in the dashboard agent to check the Raylet status.

[P0] Step 3: Add release tests

• Currently, the KubeRay end-to-end tests are primarily for sanity checking. Try running the most important Ray release tests on KubeRay.

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!