GDAL 3.7.2 build #120

pomadchin · 2023-10-29T02:12:44Z

I had a bit of time to hack on the GDAL update.

This PR updates builds to be against GDAL 3.7.2. It also contains some minor test adjustments.

I think we can still use CircleCI for releases.

It's very much WIP; feel free to rush in and introduce more changes / improvements. Not tested on Mac and Windows.

Closes #100

pomadchin · 2023-10-29T02:15:27Z

src/unit_tests/cache_tests.cpp

+BOOST_AUTO_TEST_CASE(destroy)
+{
+    GDALDestroyDriverManager();
+}


We were lucky or that's the GDAL behavior change; without it tests fail.

Added it in all cases we wan't to undo GDALAllRegister() calls.

# failure example # cd .github/workflows; docker compose -f ../docker-compose.yml up build-linux-amd64 github-build-linux-amd64-1 | corrupted size vs. prev_size github-build-linux-amd64-1 | Aborted github-build-linux-amd64-1 | make[1]: *** [Makefile:16: tests] Error 134 github-build-linux-amd64-1 | make[1]: Leaving directory '/workdir/src/unit_tests' github-build-linux-amd64-1 | make: *** [Makefile:39: tests] Error 2 github-build-linux-amd64-1 | make: Leaving directory '/workdir/src'

pomadchin · 2023-10-29T02:15:46Z

src/unit_tests/dataset_tests.cpp

@@ -55,7 +55,7 @@ BOOST_AUTO_TEST_CASE(get_block_size)

    ld.get_block_size(locked_dataset::WARPED, 1, &width, &height);
    BOOST_TEST(width == 512);
-    BOOST_TEST(height == 128);
+    BOOST_TEST(height == 512);


So weird it's been 128 before.

For the context: for "-co", "BLOCKXSIZE=512", "-co", "BLOCKYSIZE=512" input warp params I would expect both width and height to be 512.

I also saw this error, but I didn't have time to find the source. Thanks.

pomadchin · 2023-10-29T02:16:03Z

src/unit_tests/dataset_tests.cpp

    auto expected = std::vector<double>{
-        -8915910.5905594081, 33.88424960091178, 0,
-        5174836.3438357478, 0, -33.88424960091178}; // Manually verified
+        -8915910.5905594081, 33.88424960091165, 0,
+        5174836.343835746, 0, -33.88424960091165}; // Manually verified


Angry doubles.

Real-valued comparisons should prefer to use a test of the form abs(target - actual) < EPSILON for some appropriately set EPSILON.

yea, I wanted to minimize changes; but agree; reverted these numbers change and added a better comparison: f1a03c9

pomadchin · 2023-10-29T02:31:36Z

.github/docker-compose.yml

+
+services:
+  # cd .github/workflows; docker compose -f ../docker-compose.yml up build-linux-amd64
+  build-linux-amd64:


Such a winner for the local development; + mb in a separate PR this can be extended to work with GitHub Actions; (just a retranslation of the CircleCI yaml file into the docker-compose shape)

The pattern is similar to what we've done in the GeoTrellis repo; the same docker-compose used for the local development and for the CI purposes; super convenient!

pomadchin · 2023-10-29T02:33:52Z

Docker/Dockerfile.environment-amd64

-    rm -f libgdal-3.1.2-hd7bf8dc_4.tar.bz2
+    wget "https://anaconda.org/conda-forge/libgdal/3.7.2/download/osx-64/libgdal-3.7.2-h85f6614_6.conda" && \
+    mkdir -p gdal/3.7.2 && \
+    cph extract libgdal-3.7.2-h85f6614_6.conda --dest gdal/3.7.2 && \


conda doesn't ship tar.bz2 archives anymore but uses it's own tar.bz2-like packaging => so I had to install conda to extract it 🤦

https://github.com/conda/conda-package-handling

Maybe we could turn these version strings into buildargs

I went mb a bit too far :D d0f5828#diff-93c714fe4d67d93f9c74b0413dc5d5b7975e04602a2ce531ac5f5c4034e5f3fcR4-R9

Two visible cons: MacOS and Windows 'GDAL versions' are not really versions, but achive names. Still not a too bad result it looks like. For linux builds its a really nice parametrization.

jamesmcclain · 2023-10-29T20:24:26Z

🙏 ❤️

Thanks, @pomadchin , I'll take a look at this in the upcoming week. Please ping me if I haven't done so by Wednesday or Thursday .

jamesmcclain

LPGTM 👍

jamesmcclain · 2023-11-01T22:42:17Z

Docker/Dockerfile.environment-amd64

-    rm -f libgdal-3.1.2-hd7bf8dc_4.tar.bz2
+    wget "https://anaconda.org/conda-forge/libgdal/3.7.2/download/osx-64/libgdal-3.7.2-h85f6614_6.conda" && \
+    mkdir -p gdal/3.7.2 && \
+    cph extract libgdal-3.7.2-h85f6614_6.conda --dest gdal/3.7.2 && \


Maybe we could turn these version strings into buildargs

jamesmcclain · 2023-11-02T23:05:29Z

We've got the whole crew in here 😎

pomadchin · 2023-11-05T22:07:33Z

Merging this one on Monday; LMK if there are any concerns / further improvements to do before I click the button!

pomadchin added 5 commits October 28, 2023 20:30

GDAL 3.7.2 Dockerfile.environment and tests adjustments

ff8e6ec

Unsure a proper GDALDestory

a6fe3fc

Fix MacOS builds

c91b10c

Minor cleanups

50a6883

pthread_yield => sched_yield

62e4e20

pomadchin requested review from moradology, jamesmcclain and jpolchlo October 29, 2023 02:12

pomadchin commented Oct 29, 2023

View reviewed changes

pomadchin force-pushed the upd/gdal-3.7.x branch 2 times, most recently from b653077 to 1b68890 Compare October 29, 2023 02:24

README.md update

dc2ea3c

pomadchin force-pushed the upd/gdal-3.7.x branch from 1b68890 to dc2ea3c Compare October 29, 2023 02:24

.gitkeep workflows dir

544c625

pomadchin force-pushed the upd/gdal-3.7.x branch from 15183a5 to 544c625 Compare October 29, 2023 02:29

pomadchin commented Oct 29, 2023

View reviewed changes

Add the rest of the jobs into the docker-compose file

8bfc8ff

pomadchin requested a review from echeipesh October 29, 2023 02:47

pomadchin force-pushed the upd/gdal-3.7.x branch from fd129d8 to 3f2678e Compare October 31, 2023 02:26

Review comments, make a better vectors comparison

f1a03c9

pomadchin force-pushed the upd/gdal-3.7.x branch from 3f2678e to f1a03c9 Compare October 31, 2023 02:29

Replace abs diff & eps comparison with the BOOST_CHECK_SMALL function

18ab1c3

jamesmcclain approved these changes Nov 1, 2023

View reviewed changes

Parmetrize env Dockerfile via Build arguments

d0f5828

pomadchin merged commit fd0cc7d into geotrellis:master Nov 6, 2023

pomadchin changed the title ~~GDAL 3.7.2~~ GDAL 3.7.2 build Nov 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GDAL 3.7.2 build #120

GDAL 3.7.2 build #120

pomadchin commented Oct 29, 2023

pomadchin Oct 29, 2023 •

edited

Loading

pomadchin Oct 29, 2023 •

edited

Loading

jpolchlo Oct 30, 2023

pomadchin Oct 29, 2023

jpolchlo Oct 30, 2023

pomadchin Oct 31, 2023 •

edited

Loading

pomadchin Oct 29, 2023 •

edited

Loading

pomadchin Oct 29, 2023 •

edited

Loading

jamesmcclain Nov 1, 2023

pomadchin Nov 5, 2023 •

edited

Loading

jamesmcclain commented Oct 29, 2023

jamesmcclain left a comment

jamesmcclain Nov 1, 2023

jamesmcclain commented Nov 2, 2023

pomadchin commented Nov 5, 2023 •

edited

Loading

GDAL 3.7.2 build #120

GDAL 3.7.2 build #120

Conversation

pomadchin commented Oct 29, 2023

pomadchin Oct 29, 2023 • edited Loading

Choose a reason for hiding this comment

pomadchin Oct 29, 2023 • edited Loading

Choose a reason for hiding this comment

jpolchlo Oct 30, 2023

Choose a reason for hiding this comment

pomadchin Oct 29, 2023

Choose a reason for hiding this comment

jpolchlo Oct 30, 2023

Choose a reason for hiding this comment

pomadchin Oct 31, 2023 • edited Loading

Choose a reason for hiding this comment

pomadchin Oct 29, 2023 • edited Loading

Choose a reason for hiding this comment

pomadchin Oct 29, 2023 • edited Loading

Choose a reason for hiding this comment

jamesmcclain Nov 1, 2023

Choose a reason for hiding this comment

pomadchin Nov 5, 2023 • edited Loading

Choose a reason for hiding this comment

jamesmcclain commented Oct 29, 2023

jamesmcclain left a comment

Choose a reason for hiding this comment

jamesmcclain Nov 1, 2023

Choose a reason for hiding this comment

jamesmcclain commented Nov 2, 2023

pomadchin commented Nov 5, 2023 • edited Loading

pomadchin Oct 29, 2023 •

edited

Loading

pomadchin Oct 29, 2023 •

edited

Loading

pomadchin Oct 31, 2023 •

edited

Loading

pomadchin Oct 29, 2023 •

edited

Loading

pomadchin Oct 29, 2023 •

edited

Loading

pomadchin Nov 5, 2023 •

edited

Loading

pomadchin commented Nov 5, 2023 •

edited

Loading