Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build as an installable package and use vcpkg manifest mode to build standalone #1026

Open
wants to merge 42 commits into
base: main
Choose a base branch
from

Conversation

timoore
Copy link
Contributor

@timoore timoore commented Dec 5, 2024

This PR has several moving parts:

  • Package files for cesium-native created on installation. Any hackery needed to create targets for external targets is moved to CMake Find modules that are installed with the package files.
  • A single vcpkg overlay port for asycplusplus
  • vcpkg.json and vcpkg-configuration.json
  • A CMakePresets.json file demonstrating how to do a manifest mode bulid.

I'm using this branch to build vsgCs using cesium-native as a vcpkg overlay port. That usage needs to be documented here too. Also, while I've tried to preserve the old behavior using exvcpkg, I haven't tested this yet with Cesium for Unreal and other integrations in the classic style of using cesium-native as a CMake subdirectory.

This is basically the addition of the vcpkg.json manifest and
vcpkg-configuration.json, which are both pretty minimal. In practice,
projects will probably copy those and add overlays.
Should be a no-op at this point, but will be required when we make
Cesium Native installable.
The intent is for public target sources to be compiled into consumers
of a target, but that is a no-op for header files. Including the
public headers, which have paths pointing inside the project, causes
an error when generating an export set.
This create the CMake files that create targets in an imported
package.
Add a find module for zlib-ng, which can be installed and included by
a config package. Other find modules may be necessary too.
VCPKG_MANIFEST_MODE isn't available before project() is called, so
define a new variable CESIUM_USE_EZVCPKG to control the early
loading of ezvcpkg.
Individual integrations can add their own overlay if they need it.
Also, try to make sure that vcpkg actually uses the desired triplet.
@azrogers
Copy link
Contributor

azrogers commented Dec 9, 2024

Looks like this causes the following issue in cesium-unreal:

  CMake Error at cesium-native/cmake_install.cmake:940 (file):
    file cannot create directory:
    C:/Dev/prs/cesium-native-1026/Plugins/cesium-unreal/extern/../Source/ThirdParty/lib/Windows-AMD64-$<IF:$<CONFIG:Debug>,Debug,Release>/cmake/cesium-native.
    Maybe need administrative privileges.
  Call Stack (most recent call first):
    cmake_install.cmake:37 (include)

Not sure why this would have broken here - the cesium-unreal/extern CMakeLists.txt is unchanged here and I don't see why the CONFIG generator expression would stop evaluating correctly here.

@timoore
Copy link
Contributor Author

timoore commented Dec 9, 2024

Thanks for being brave enough to test this with Cesium Unreal! What generator was used to build cesium-native?

@timoore
Copy link
Contributor Author

timoore commented Dec 10, 2024

The problem is in the new cesium-native/CMakeLists.txt, which defines exports to install. The install(EXPORT...) command doesn't support generator expressions in its destination argument... which makes a certain amount of sense, even if it's inconsistent. The export files won't be used in the Unreal build, but I'll change the command to use a more suitable variable.

In the Cesium for Unreal build, CMAKE_INSTALL_LIBDIR contains
a generator expression, which the install(EXPORT ...) command doesn't
process correctly. So, install the configuration files such as
"cesium-nativeTargets.cmake" in
${CMAKE_INSTALL_DATADIR}/cesium-native/cmake, which is a perfectly
fine location.

These files aren't used by the Cesium Unreal build anyway.
@timoore
Copy link
Contributor Author

timoore commented Jan 6, 2025

I've started to look at how to use this in cesium-unreal and have hit a bit of a sticking point with specifying the MSVC toolset version to build those parts of cesium-unreal/extern that wouldn't be vcpkg port overlays. Maybe this isn't a real issue, because I think the GitHub build action would work the same. It's really only a problem for developers who have multiple toolsets installed. It is possible to set the toolset version through CMakePresets.json, and on the command line, but otherwise it is really obscure. Do you think this is worth worrying about?

Just to be clear, this isn't a new problem that comes with vcpkg manifest mode.

@2xsaiko
Copy link

2xsaiko commented Jan 11, 2025

I've been trying to package this (Homebrew for now, later Gentoo) so I can use it as a dependency for vsgCs which I want to use in something I'm working on, however right now it seems pretty much impossible to do so without significantly changing the build files. This patchset looks like it could improve that, however with this it fails to even configure in the Homebrew build environment:

error: In manifest mode, vcpkg install does not support individual package arguments.

https://gist.github.com/2xsaiko/9d2bc0691529d2599e2d43d8cf7e8ec5

Strangely enough, this error doesn't occur when running it outside of the Homebrew package. Any ideas?

@timoore
Copy link
Contributor Author

timoore commented Jan 11, 2025

I've been trying to package this (Homebrew for now, later Gentoo) so I can use it as a dependency for vsgCs which I want to use in something I'm working on, however right now it seems pretty much impossible to do so without significantly changing the build files. This patchset looks like it could improve that, however with this it fails to even configure in the Homebrew build environment:

error: In manifest mode, vcpkg install does not support individual package arguments.

https://gist.github.com/2xsaiko/9d2bc0691529d2599e2d43d8cf7e8ec5

Strangely enough, this error doesn't occur when running it outside of the Homebrew package. Any ideas?

The vcpkg-build branch in vsgCs uses a commit to a personal copy of Cesium Native -- sucked in as a vcpkg port overlay -- that was the basis of this PR and is very close to it. I encourage you to give that a try.

In your build that is crashing, it looks like the ezvcpkg code is getting confused by the vcpkg.json mainifest file and is then going into manifset mode. This is interesting as I would like to keep ezvcpkg working for a while, but for experimenting with this PR you probably want to specify CESIUM_USE_EZVCPKG=OFF.

@azrogers
Copy link
Contributor

Gave this a shot using vsgCs and I'm getting this build error when vcpkg tries to install cesium-native:

C:\PROGRA~1\MICROS~4\2022\PROFES~1\VC\Tools\MSVC\1442~1.344\bin\Hostx64\x64\cl.exe   /TP -DCESIUMGEOMETRY_BUILDING -DCESIUM_SHARED=ON -DCesiumGeometry_EXPORTS -DFMT_SHARED -DGLM_ENABLE_EXPERIMENTAL -DGLM_FORCE_EXPLICIT_CTOR -DGLM_FORCE_XYZW_ONLY -DSPDLOG_COMPILED_LIB -DSPDLOG_FMT_EXTERNAL -DSPDLOG_SHARED_LIB -IC:\Tools\vcpkg\buildtrees\cesium-native\src\a7d8bb629b-dfd76a50f3.clean\CesiumGeometry\include -IC:\Tools\vcpkg\buildtrees\cesium-native\src\a7d8bb629b-dfd76a50f3.clean\CesiumUtility\include -external:ID:\Dev\vsgCs\vcpkg_installed\x64-windows\include -external:W0 /nologo /DWIN32 /D_WINDOWS /utf-8 /GR /EHsc /MP  /MDd /Z7 /Ob0 /Od /RTC1  -std:c++20 -MDd /W4 /WX /wd4201 /bigobj /w45038 /w44254 /w44242 /w44191 /w45220 /wd4251 /wd4275 /utf-8 /showIncludes /FoCesiumGeometry\CMakeFiles\CesiumGeometry.dir\src\Availability.cpp.obj /FdCesiumGeometry\CMakeFiles\CesiumGeometry.dir\ /FS -c C:\Tools\vcpkg\buildtrees\cesium-native\src\a7d8bb629b-dfd76a50f3.clean\CesiumGeometry\src\Availability.cpp
C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.42.34433\include\vector(1416): error C2280: 'std::unique_ptr<CesiumGeometry::AvailabilityNode,std::default_delete<CesiumGeometry::AvailabilityNode>> &std::unique_ptr<CesiumGeometry::AvailabilityNode,std::default_delete<CesiumGeometry::AvailabilityNode>>::operator =(const std::unique_ptr<CesiumGeometry::AvailabilityNode,std::default_delete<CesiumGeometry::AvailabilityNode>> &)': attempting to reference a deleted function
C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.42.34433\include\memory(3452): note: see declaration of 'std::unique_ptr<CesiumGeometry::AvailabilityNode,std::default_delete<CesiumGeometry::AvailabilityNode>>::operator ='
C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.42.34433\include\memory(3452): note: 'std::unique_ptr<CesiumGeometry::AvailabilityNode,std::default_delete<CesiumGeometry::AvailabilityNode>> &std::unique_ptr<CesiumGeometry::AvailabilityNode,std::default_delete<CesiumGeometry::AvailabilityNode>>::operator =(const std::unique_ptr<CesiumGeometry::AvailabilityNode,std::default_delete<CesiumGeometry::AvailabilityNode>> &)': function was explicitly deleted

I've no idea what might be causing this to happen when vcpkg tries building it, since cesium-native builds fine on its own in the same environment. The only thing I can think is that this build is using ninja, whereas the normal CMake cesium-native build on Windows uses the Visual Studio 17 2022 generator, but this only raises further questions since cmake -B build -S . -G Ninja fails on cesium-native in the configuration stage.

@HellfireATGM
Copy link

For what its worth, I have the exact same issue as above. Is there a workaround or fix available? Thanks.

@timoore
Copy link
Contributor Author

timoore commented Feb 24, 2025

I'm taking a look at this, but I'm curious: do @azrogers or @HellfireATGM have more than one version of the Visual Studio toolset installed? It can be quite tricky to insure that vcpkg packages and the main project all use the same version.

@timoore
Copy link
Contributor Author

timoore commented Feb 24, 2025

I can't reproduce this with the latest vsgCs sources.

@HellfireATGM
Copy link

Sorry, my bad. Didn't realize there was a config for windows. I did the cmake with this preset:
cmake --preset Windows-Devel
then built in visual studio.

All good now. Thanks.

@kring
Copy link
Member

kring commented Feb 28, 2025

Sorry @timoore I'm not really comfortable merging this before next week's release. I haven't reviewed it in depth, but one problem I see is that the CI build times are drastically increased. I think the caching isn't working anymore. Let's aim to get it in as soon after next week's release as we can, and we can even tag a mid-cycle native release if it's helpful.

@timoore
Copy link
Contributor Author

timoore commented Feb 28, 2025

Sorry @timoore I'm not really comfortable merging this before next week's release. I haven't reviewed it in depth, but one problem I see is that the CI build times are drastically increased. I think the caching isn't working anymore. Let's aim to get it in as soon after next week's release as we can, and we can even tag a mid-cycle native release if it's helpful.

The cache management page for cesium-native says:
Approaching total cache storage limit (16.74 GB of 10 GB Used)

This PR creates new caches for the VCPKG builds that are probably no smaller than the caches for ezvcpkg. I don't think that the caches for this PR are being kept around (or are even being created), and I wouldn't be surprised if caching for all the builds is being affected.

@timoore
Copy link
Contributor Author

timoore commented Feb 28, 2025

I've tried to set up vcpkg caching with S3, but I'm getting a spew of errors like:

warning: error: aws failed with exit code: (254).

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

so I clearly don't know what I'm doing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants