How to obtain sample data from S3 and launch the ogc-example test locally #394
Comments
|
I have currently switched the test data to COG, but I am still getting the same error as before. I suspect it's not an issue with the sample data, but rather a problem with the underlying environment. I am running on the Windows operating system, and my gdal-warp-bindings.jar version is 1.1.1. The installed GDAL version is 3.0.4. The error message I'm receiving is as follows: Is there anyone who can help take a look at this issue？ |
|
Hi! I am not sure what's the About the second comment: what is the
|
|
My configuration file is as follows： I now believe that the main reason for By the way, can tile data sliced through Geotrellis be published as WMTS or WMS services in geotrellis-server? Is changing the source in the configuration file the only way to do it, or are there other methods? |
|
That's the supported GDAL matrix: https://github.com/geotrellis/gdal-warp-bindings#installation Try without GDAL to factor GDAL out for now. (replace The data stored as GeoTrellis layer can be served as well, see layer definition example As is (example projects) the way to deal with the source is via configuration files only. However, you may use this repository as a repository of components / reference server implementation to implement smth that fits requirements better. |
|
Thank you for your patient response. I tried not to use GDAL and changed the source to 14:40:52.222 [raster-io-0] ERROR org.http4s.server.service-errors - Error servicing request: GET / from 127.0.0.1 I'm not sure if this is a specific bug in Windows or not. |
I also tried publishing the Geotrellis layer from local or HBase as a service, but it still throws the same error：Unable to construct dataset dimensions. GDAL Error Code: 4 My configuration is And I can show you my local Geotrellis layers: Is there something wrong? |
|
Indeed Windows support was never our focus, that's likely an unhandled cases in GeoTrellis; there's been locationtech/geotrellis#3507 that addressed a windows parsing case, it looks like it may help you. To use GeoTrellis layers the
|
|
Thank you, I found that the latest release version 3.7.0 of GeoTrellis has not included the Windows portion of this code locationtech/geotrellis#3507 yet. And I will try it on Linux. I've tried to use So my question is whether the
|
|
@RunBoo it should point to the directory with the catalog, not to the attributes folder; in your case Interesting error you got: https://github.com/locationtech/geotrellis/blob/133a2d076bdba4dff4a760a4bf785268924dbde4/raster/src/main/scala/geotrellis/raster/RasterSource.scala#L210 Means that no GeoTrellisRasterSourceProvider is available at runtime. Double check the class path / if all of the dependencies are properly loaded, this portion of the code relies on SPI. In the worst case try manually checking if its there. How do you start / build the server? You may try using WSL if staying on Windows is important. |
|
@pomadchin Thank you very much! And then I tried to change the source to Geotrellis tiles, which is Initially, I suspected that the issue was due to GDAL not having EPSG installed correctly. However, upon checking, GDAL was able to read the projection system of the data. Now, I suspect that there might be an issue with the code I used for slicing in Geotrellis. I am using Geotrellis version 3.6.1, and my core code is as follows: Is it because the coordinate system is incorrect when I perform the slicing, or is the slicing method incorrect? |
|
@RunBoo what's the proj4 string of the source data? I assume the issue is in the proj4 unsable to map it into epsg code to construct a proper wms layer: https://github.com/geotrellis/geotrellis-server/blob/main/ogc/src/main/scala/geotrellis/server/ogc/wms/CapabilitiesView.scala#L147 Two things can be improved: 1. get removal 2. proj4 ehancement potentially to understand that proj4 string epsg code in case its unsupported This is nothing to do with GDAL, GDAL is used in case of a GDALRasterSource usage only. |
|
The spatial coordinate of my source data is CGCS2000, of which the proj4 string is Now I've changed my source data to another data with WGS84 spatial coordinate, and its proj4 string is The related code is: https://github.com/locationtech/geotrellis/blob/133a2d076bdba4dff4a760a4bf785268924dbde4/raster/src/main/scala/geotrellis/raster/GridExtent.scala#L45 Now my data and the underlying environment appear to be normal, and I'm using the method of slicing with Geotrellis before publishing the service. I'm puzzled by this new error. |
|
@pomadchin Thank you very much for patiently helping me with each new question. I understand now what the issue is. I have been publishing Geotrellis tile data, and instead of publishing it as WMS, it should be published as WMTS. Now I have successfully published WMTS, but currently I can only retrieve attribute information through GetCapabilities. When I try to load the WMTS in QGIS, I can see the layer information：
but the images cannot be loaded, and the program throws this error： 20:26:24.649 [raster-io-3] ERROR org.http4s.server.service-errors - Error servicing request: GET / from 127.0.0.1 My tiles data and configuration file is as follows: |
|
And if I change the tiles data from HBase, like this: Exception in thread "raster-io-0" java.lang.NoClassDefFoundError: org/apache/log4j/Level Is this issue related to Hadoop data retrieving? |
|
If WMTS doesn't support GT layer, use geotrellis-server, how can I publish GT layers data to service? Should I publish to WMS? But I really have sliced source data to tiles. |
|
@pomadchin I'm planning to submit a PR, but realized that I don't have the necessary permission. Could you please add this for me? And here is the code I want to push: However, it appears that the situation is not as straightforward as I initially thought. I encountered another error: geotrellis.vector.ExtentRangeError: Invalid Extent: xmin must be less than xmax (xmin=9783.93962050256, xmax=9783.939620502026) I have been debugging for a long time, but I still can't find the issue. Could you help me take a look?
Thank you! |
After I added this code and tried another two data, now I cannot get any error anymore. But in the QGis software, I also cannot load images, the screen is totally white. One of my data is: So I suspect that if there is smth wrong with my code of slicing GT layers? In our chat logs you can find my core code. |
|
@RunBoo the way contributions work on GitHub is by making a fork and then a PR based on a push into the forked repo, will be happy to review the PR. The error is likely to be of the similar roots as the Most likely a too low (not high enough resolution) zoom level on viz. I'd recommend a higher zoom level (11-12) and WCS to work on the issue. Most likely some projection / data quality bug / feature.
It does not really matter much, wmts / wcs / wms are specs and behave as views to the original data source that can be of any shape (rasters, stac api + rasters, geotrellis layers, whatever custom file format is needed)
Totally not clear, have you tried looking at the sliced chunk without gt server?
I'll try to find time, but no guarantees on the timeline. |
|
I have submitted the PR here: https://github.com/geotrellis/geotrellis-server/pull/395
I already have the slicing level set to 12. Or are you suggesting that I should set this parameter( Last question, could you share a code snippet for slicing multi-band Geotiff data using Geotrellis 3.6.0 version? I haven't been able to find such code online either. |
|
@pomadchin I'm sorry to bother you again, but I set the zoom level to 12, and nothing happened. There are no error messages now, but the map still cannot be loaded. So I really have no idea. Allow me to summarize. I have a GeoTIFF image, I want to slice it using Geotrellis, and then publish it as a WMTS / WCS or WMS service using Geotrellis-server to visualization. Is this technically feasible? I don't think it should be a difficult process, but it actually has taken a long time. |
|
yea it should be a straightforward process; try just printing out a layer, wondering if thats a data issue; GT layer is chunked as z/x/y. |
|
@RunBoo what is the original source tiff? The layer is definitely not correct; There is only one tile 12/6990506 of an 1 x 2 size; I converted it to png and tiff - the result aligns with the gt server results you got (nothing is displayed) (: |
|
@RunBoo can you share the original tiff so I can take a look at the tiling logic? It is hardly possible that it is related to the GT versions, there were no changes done in [3.6; 3.7] around this logic. |
|
Sure, the source Geotiff and layers data is: Maybe my code is based on 2.x version, I'm not very sure. But I'm clear that 3.x version is different from 2.x version. |
|
I'd recommend you just to regenerate the layer, smth is off with the one you tried to visualize. |
|
The ingest code new style via RasterSources (I generated tiles-new.zip this way): val paths = "file:///tmp/origin.tif" :: Nil
val layerName = "tiles"
implicit val sc: SparkContext = createSparkContext("IngestRasterSource", new SparkConf(true))
val targetCRS = LatLng
val method = Bilinear
val layoutScheme = ZoomedLayoutScheme(targetCRS, tileSize = 256)
val sourceRDD: RDD[RasterSource] =
sc.parallelize(paths, paths.length)
.map(uri => RasterSource(uri).reproject(targetCRS, method = method): RasterSource)
.cache()
val summary = RasterSummary.fromRDD(sourceRDD)
val LayoutLevel(zoom, layout) = summary.levelFor(layoutScheme)
val contextRDD = RasterSourceRDD.tiledLayerRDD(sourceRDD, layout, KeyExtractor.spatialKeyExtractor, rasterSummary = summary.some)
val attributeStore = FileAttributeStore("/tmp/tiles-new")
val writer = FileLayerWriter(attributeStore)
Pyramid.upLevels(contextRDD, layoutScheme, zoom, method) { (rdd, z) =>
val layerId = LayerId(layerName, z)
if (attributeStore.layerExists(layerId)) FileLayerDeleter(attributeStore).delete(layerId)
writer.write(layerId, rdd, ZCurveKeyIndexMethod)
}The ingest code (old style like in your examples), I generated tiles using this one as well, should match the result of the code above; visually its the same result, difference can be in some tiny details on edges: val inputPath = "/tmp/origin.tif"
val layerName = "tiles"
implicit val sc: SparkContext = createSparkContext("Ingest", new SparkConf(true))
val targetCRS = LatLng
val method = Bilinear
val layoutScheme = ZoomedLayoutScheme(targetCRS, tileSize = 256)
val inputRdd: RDD[(ProjectedExtent, MultibandTile)] = sc.hadoopMultibandGeoTiffRDD(inputPath) // can be single
val (_, rasterMetaData) = CollectTileLayerMetadata.fromRDD(inputRdd, FloatingLayoutScheme(512))
val tiled: RDD[(SpatialKey, MultibandTile)] = inputRdd.tileToLayout(rasterMetaData.cellType, rasterMetaData.layout, Bilinear)
val (zoom, reprojected): (Int, RDD[(SpatialKey, MultibandTile)] with Metadata[TileLayerMetadata[SpatialKey]]) =
MultibandTileLayerRDD(tiled, rasterMetaData)
.reproject(targetCRS, layoutScheme, method)
val attributeStore = FileAttributeStore("/tmp/tiles-new-old")
val writer = FileLayerWriter(attributeStore)
Pyramid.upLevels(reprojected, layoutScheme, zoom, method) { (rdd, z) =>
val layerId = LayerId(layerName, z)
if (attributeStore.layerExists(layerId)) FileLayerDeleter(attributeStore).delete(layerId)
writer.write(layerId, rdd, ZCurveKeyIndexMethod)
}The def createSparkContext(appName: String, sparkConf: SparkConf = createSparkConf): SparkContext = {
sparkConf
.setAppName(appName)
.setIfMissing("spark.master", "local[*]")
.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.set("spark.kryo.registrator", classOf[geotrellis.spark.store.kryo.KryoRegistrator].getName)
new SparkContext(sparkConf)
}You may use https://github.com/pomadchin/vlm-performance/tree/feature/gt-3.x/src/main/scala/geotrellis/contrib/performance as a source of inspiration for the ingest code. |
|
@pomadchin Thank you very much, it's very kind of you. I'll try it. |
|
@pomadchin Hi~ I tried the tile data and slicing code you provided, thanks a lot. I found that publishing as WCS works fine, but publishing as WMS or WMTS still doesn't work. There are still some data-related errors that prevent the image from loading.
Can you help me check if there is a problem with the tile-matrix settings or other conf? And just to confirm, the current version of gt-server is able to support publishing GT layer data as WMTS, right? |
|
I think the problem is somewhere on the reprojection / source data side; It is a bit time consuming figuing our what's the actual issue there; WCS display most likely works for you by a lucky mistake / smth done with the projections. |
I wonder whether the resolution of the last layer of RasterSourceRDD can be specified, manually or relevant to the original tiff. For example, it is good if the last layer (or next to last layer) is the same as the original tiff. How can I use parameters to meet the demand? |
I'm new to geotrellis-server and currently attempting to launch the sample test for ogc-example.
I noticed that the sample data is located on S3, but I don't have an AWS S3 account to access this data. Can someone please send me a copy of the test data, such as the data from this link:
gt+s3://azavea-datahub/catalog?layer=us-ned-tms-epsg3857&zoom=14&band_count=1
https://github.com/geotrellis/geotrellis-server/blob/2a1fab9eb967bad1e8c3d1257a10dfc32182380b/ogc-example/src/main/resources/application.conf#L277
My email address is [email protected].
And if I have the data locally, how should I configure the application.conf file?
The text was updated successfully, but these errors were encountered: