Poetry v2

[MonkeyCanCode]

This is for additional changes for Poetry v2 where we will be migrating existed project to PEP 621-style configuration.

Here are the summary for the changes:

1. Make pyproject.tom PEP 621-compliant
2. Use poetry-core for the build system instead of setuptools
3. Remove reference for setuptools
4. Remove non-essential dependencies (flake8, pylint, types-python-dateutil, mypy) for running tests and introduce a test group in Poetry (if those tools are needed, we can do a new group for dev instead?)

The final pyproject.toml is validated via validate-pyproject:

(venv) ➜  python git:(poetry_v2) ✗ validate-pyproject pyproject.toml
Valid file: pyproject.toml

Here is the non-docker test status:

Mon Jan 27 23:34:19 CST 2025: Test SUCCEEDED: t_cli:test_cli.py
Mon Jan 27 23:34:19 CST 2025: Starting test t_hello_world:hello_world.sh
Mon Jan 27 23:34:19 CST 2025: Test run concluded for t_hello_world:hello_world.sh
Mon Jan 27 23:34:19 CST 2025: Test SUCCEEDED: t_hello_world:hello_world.sh
Mon Jan 27 23:34:19 CST 2025: Starting pytest t_pyspark:test_spark_sql_s3_with_privileges.py
...
Mon Jan 27 23:34:21 CST 2025: Test SUCCEEDED: t_pyspark:test_spark_sql_s3_with_privileges.py
Mon Jan 27 23:34:21 CST 2025: Azure tests not enabled, skip running test t_spark_sql/src/spark_sql_azure_dfs.sh
Mon Jan 27 23:34:21 CST 2025: GCS tests not enabled, skip running test t_spark_sql/src/spark_sql_gcp.sh
Mon Jan 27 23:34:21 CST 2025: Azure tests not enabled, skip running test t_spark_sql/src/spark_sql_azure_blob.sh
Mon Jan 27 23:34:21 CST 2025: Starting test t_spark_sql:spark_sql_views.sh
Mon Jan 27 23:34:34 CST 2025: Test run concluded for t_spark_sql:spark_sql_views.sh
Mon Jan 27 23:34:34 CST 2025: Test SUCCEEDED: t_spark_sql:spark_sql_views.sh
Mon Jan 27 23:34:34 CST 2025: AWS tests not enabled, skip running test t_spark_sql/src/spark_sql_s3.sh
Mon Jan 27 23:34:34 CST 2025: AWS cross region tests not enabled, skip running test t_spark_sql/src/spark_sql_s3_cross_region.sh
Mon Jan 27 23:34:34 CST 2025: Starting test t_spark_sql:spark_sql_basic.sh
Mon Jan 27 23:34:45 CST 2025: Test run concluded for t_spark_sql:spark_sql_basic.sh
Mon Jan 27 23:34:45 CST 2025: Test SUCCEEDED: t_spark_sql:spark_sql_basic.sh
Mon Jan 27 23:34:45 CST 2025: Tests completed with 3 successes and 0 failures

Here is the docker test status:

...
regtest-1  | Tue Jan 28 04:44:41 AM UTC 2025: Test run concluded for t_spark_sql:spark_sql_views.sh
regtest-1  | Tue Jan 28 04:44:41 AM UTC 2025: Test SUCCEEDED: t_spark_sql:spark_sql_views.sh
regtest-1  | Tue Jan 28 04:44:41 AM UTC 2025: Azure tests not enabled, skip running test t_spark_sql/src/spark_sql_azure_dfs.sh
regtest-1  | Tue Jan 28 04:44:41 AM UTC 2025: Tests completed with 3 successes and 0 failures
regtest-1 exited with code 0
Aborting on container exit
...

Noted, until 896 is merged, the github action will continue to fail for this one.

[MonkeyCanCode]

@eric-maynard may I have a review this for this?

[MonkeyCanCode]

@HonahX may i get a review for this one as well?

[MonkeyCanCode]

@jbonofre may I have a review for this one? Seems to causing blockers for couple PRs in the last couple days and similar PR raised as well.

[jbonofre]

@MonkeyCanCode sure. I gonna take a look.