Skip to content

Commit 3a4a8e3

Browse files
ioalexeidanbjosephAlexeijayqi
authored
feat: generate h3 polygons and create webmap (#46)
* Create points_to_h3.py * Revert "Create points_to_h3.py" This reverts commit 9b33217. * Create points_to_h3.py * Updates to points_to_h3 - add h3pandas to dependencies - add app launch to end of file - use single aggregation (mean) instead of multiple - change order of arguments based on defaults * Add webmap code and template * Tidying and formatting webmap script - adding Typer help and defaults - adding code to generate colours and breakpoints for map - formatting * Fix interpolation breaks - fix interpolation breaks - fix hard coded template path * Fix code to generate polygon colours Hard code colours into dataframe/json rather than using MapLibre interpolation * Delete gvi_webmap.html remove temp file * Applying ruff checks * Update maplibre_template.html - add pop to show gvi score when polygon is clicked * Update create_webmap.py - Round GVI scores to 2 decimal places for neater pop-ups * Generate legend from GVI scores * Update create_webmap.py - Reduce bounding box buffer from 0.5 degrees to 0.25 - Set default zoom to 12 * Update .env.example Update sample .env file to include placeholder for Maptiler api key * Update create_webmap.py Fix line lengths * Update points_to_h3.py Formatting * visualize step described in readme * remove reference to folder not in repo * alphabetize new dependency * Update README.md add info on MapTiler API key * Fix single digit resolution error Add a leading `0` to cell resolution if it is a single digit * Use empy basemap if no Maptiler key provided The Jinja template will use an empty basemap if there is no Maptiler API key in the .env file * Formatting * fix env variable and update template * Combine output path and filename * docs: filename and path is one argument * add logiv for missing env variable * fix for blank env variable * fix: ruff format * fix: ruff fix import block is un-sorted or un-formatted * fix: change web map colormap * fix legend by changing int to float * Ignore linter for import; needed to register accessor --------- Co-authored-by: Dan Joseph <[email protected]> Co-authored-by: Dan Joseph <[email protected]> Co-authored-by: Alexei <[email protected]> Co-authored-by: Jay Qi <[email protected]>
1 parent f960f6d commit 3a4a8e3

6 files changed

+400
-1
lines changed

‎.env.example

+2-1
Original file line numberDiff line numberDiff line change
@@ -1 +1,2 @@
1-
MAPILLARY_CLIENT_TOKEN = "MY_MAPILLARY_CLIENT_TOKEN"
1+
MAPILLARY_CLIENT_TOKEN = "MY_MAPILLARY_CLIENT_TOKEN"
2+
MAPTILER_API_KEY = "MY_MAPTILER_API_KEY"

‎README.md

+31
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ If you are interested in joining the project, please check out [`CONTRIBUTING.md
4747
- For example, download [`Three_Rivers_Michigan_USA_line.zip`](https://drive.google.com/file/d/1fpI4I5KP2WyVD5PeytW_hoXZswOt0dwA/view?usp=drive_link) to `data/raw/Three_Rivers_Michigan_USA_line.zip`. Note that this Google Drive link is only accessible to approved project members.
4848
4. Make a copy of the `.env.example` file, removing the `.example` from the end of the filename.
4949
- To download images from [Mapillary](https://www.mapillary.com/) you will need to create a (free) account and replace `MY_MAPILLARY_CLIENT_TOKEN` in the `.env` file with your own token. See the "Setting up API access and obtaining a client token" section on this [Mapillary help page](https://help.mapillary.com/hc/en-us/articles/360010234680-Accessing-imagery-and-data-through-the-Mapillary-API). You only need to enable READ access scope on your token.
50+
- To use OpenStreetMap as a basemap for any webmaps generated, you will need a MapTiler API key. To get a free API key, follow the instructions on [this page](https://docs.maptiler.com/cloud/api/authentication-key/). Once you have an API key, add a new line to your `.env` file: `MAPTILER_API_KEY = "MY_MAPTILER_API_KEY"` - replace the text between the quotes with the key generated on the MapTiler account page.
5051

5152
### 1. Sample points from roads data
5253

@@ -104,6 +105,36 @@ saves an output to a new file.
104105
python -m src.assign_gvi_to_points data/raw/mapillary data/interim/Three_Rivers_Michigan_USA_points_images.gpkg data/processed/Three_Rivers_GVI.gpkg
105106
```
106107

108+
### 4. Visualize the results
109+
110+
We provide an option here but we encourage you to explore different ways to visualize the results. We would love to know what you try that works and what doesn't. How can we improve on these guidance materials?
111+
112+
#### Generate an H3 polygon layer
113+
114+
We can generate an H3 polygon layer from the point layer. As an overlay it may make spatial trends more visible by merging some of the values from close together points.
115+
116+
### Example
117+
118+
```bash
119+
# python -m src.create_webmap path/to/input_file.gpkg path/to/output_file.gpkg cell_resolution
120+
python -m src.points_to_h3 data/processed/Three_Rivers_GVI.gpkg data/processed/Three_Rivers_h3_polygons_10.gpkg 10
121+
```
122+
123+
The larger the number for the [H3 cell resolution](https://h3geo.org/docs/core-library/restable/), the smaller the individual hexagons.
124+
125+
#### Generate a web map
126+
127+
We can generate an HTML file that contains javascript to display a web map showing the H3 polygons, styled by the mean GVI score for each polygon.
128+
129+
To display an OpenStreetMap basemap under the data, you will need an API key from [MapTiler](https://www.maptiler.com/), a vector tiles provider. Once you have an API key, add it to your `.env` file (see [Setup section](#0-setup) for how to do this).
130+
131+
### Example
132+
133+
```bash
134+
# python -m src.create_webmap path/to/input_file.gpkg path/to/output/output_file.html default_zoom_for_webmap
135+
python -m src.create_webmap data/processed/Three_Rivers_h3_polygons_10.gpkg data/processed/Three_Rivers_gvi_webmap.html 10
136+
```
137+
107138
## Config files
108139
109140
> ![NOTE]

‎pyproject.toml

+1
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ dependencies = [
1717
"folium",
1818
"geopandas",
1919
"geopy",
20+
"h3pandas",
2021
"loguru",
2122
"mapclassify",
2223
"matplotlib",

‎src/create_webmap.py

+162
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
import os
2+
from pathlib import Path
3+
4+
from dotenv import load_dotenv
5+
import geopandas as gpd
6+
from jinja2 import Environment, FileSystemLoader
7+
import matplotlib
8+
import numpy as np
9+
from shapely.geometry import box
10+
import typer
11+
12+
try:
13+
from typing import Annotated
14+
except ImportError:
15+
# for Python 3.9
16+
from typing_extensions import Annotated
17+
18+
19+
app = typer.Typer()
20+
21+
22+
def h3_to_webmap():
23+
return
24+
25+
26+
@app.command()
27+
def main(
28+
input_file: Annotated[
29+
Path,
30+
typer.Argument(help="Path to file containing h3 polygons."),
31+
],
32+
filename: Annotated[
33+
str, typer.Argument(help="(Optional) Path to file for HTML output.")
34+
] = "./data/processed/gvi_webmap.html",
35+
zoom_level: Annotated[
36+
int,
37+
typer.Argument(
38+
help="""(Optional) Starting zoom level for webmap.
39+
Takes integer between 0 (small scale) and 20 (large scale).
40+
"""
41+
),
42+
] = 12,
43+
):
44+
gdf = gpd.read_file(input_file)
45+
# if crs is not 4326, convert to 4326
46+
if gdf.crs == "EPSG:4326":
47+
pass
48+
else:
49+
gdf = gdf.to_crs("EPSG:4326")
50+
51+
# Round GVI score to make map labels more readable
52+
gdf["gvi_score"] = round(gdf["gvi_score"], 2)
53+
54+
# get central coordinates of all features
55+
centre = (
56+
str(gdf.dissolve().centroid.x.values[0])
57+
+ ", "
58+
+ str(gdf.to_crs("4326").dissolve().centroid.y.values[0])
59+
)
60+
centre_str = "[" + centre + "]"
61+
62+
# Calculate datasdet bounds (with a buffer of 0.5 degrees) to limit webmap bounds
63+
gdf_bounds = gdf.total_bounds
64+
bbox = box(*gdf_bounds).buffer(0.25, cap_style="square", join_style="mitre")
65+
bounds = bbox.bounds
66+
bounds_str = (
67+
"["
68+
+ str(bounds[0])
69+
+ ","
70+
+ str(bounds[1])
71+
+ "], ["
72+
+ str(bounds[2])
73+
+ ","
74+
+ str(bounds[3])
75+
+ "]"
76+
)
77+
78+
# Load API key for map tiles from environment variable
79+
load_dotenv()
80+
81+
if "MAPTILER_API_KEY" in os.environ:
82+
maptiler_api_key = os.getenv("MAPTILER_API_KEY")
83+
else:
84+
maptiler_api_key = "None"
85+
86+
if maptiler_api_key in ["MY_MAPTILER_API_KEY", ""]:
87+
maptiler_api_key = "None"
88+
89+
# Lookup the colourmap values for each GVI score
90+
cmap = matplotlib.colormaps["Greens"]
91+
gdf["gvi_norm"] = (gdf.gvi_score - np.min(gdf.gvi_score)) / (
92+
np.max(gdf.gvi_score) - np.min(gdf.gvi_score)
93+
)
94+
gdf["html_color"] = gdf["gvi_norm"].apply(
95+
lambda x: matplotlib.colors.rgb2hex(cmap(x))
96+
)
97+
98+
# Generate divs for legend
99+
# Pick 4 evenly-spaced values from the gvi scores to use in the legend
100+
legend_gvi = list(
101+
np.arange(
102+
gdf.gvi_score.min(),
103+
gdf.gvi_score.max(),
104+
(gdf.gvi_score.max() - gdf.gvi_score.min()) / 4,
105+
dtype=float,
106+
)
107+
)
108+
109+
# Generate labels by looking up what the GVI score would be for those values
110+
legend_label_1 = round(
111+
np.linspace(gdf.gvi_score.min(), gdf.gvi_score.max(), 100)[0], 1
112+
)
113+
legend_label_2 = round(
114+
np.linspace(gdf.gvi_score.min(), gdf.gvi_score.max(), 100)[33], 1
115+
)
116+
legend_label_3 = round(
117+
np.linspace(gdf.gvi_score.min(), gdf.gvi_score.max(), 100)[66], 1
118+
)
119+
legend_label_4 = round(
120+
np.linspace(gdf.gvi_score.min(), gdf.gvi_score.max(), 100)[99], 1
121+
)
122+
123+
# Normalise the label values to lookup against the colourmap
124+
legend_gvi_norm = (legend_gvi - np.min(legend_gvi)) / (
125+
np.max(legend_gvi) - np.min(legend_gvi)
126+
)
127+
128+
# Generate the html colour code from the normalised values
129+
legend_colours = []
130+
for i in legend_gvi_norm:
131+
legend_colours.append(matplotlib.colors.rgb2hex(cmap(i)))
132+
# Assign patch colours to use in HTML template
133+
legend_patch_1, legend_patch_2, legend_patch_3, legend_patch_4 = legend_colours
134+
135+
# Load the MapLibre HMTL template
136+
environment = Environment(loader=FileSystemLoader("src/templates"))
137+
template = environment.get_template("maplibre_template.html")
138+
139+
# Generate the HTML file from the template, filling dynamic values
140+
with open(filename, mode="w", encoding="utf-8") as message:
141+
message.write(
142+
template.render(
143+
title="GVI score hex map",
144+
geojson=gdf.to_json(),
145+
centre_coords=centre_str,
146+
zoom=zoom_level,
147+
bounds=bounds_str,
148+
maptiler_api_key=maptiler_api_key,
149+
legend_label_1=legend_label_1,
150+
legend_label_2=legend_label_2,
151+
legend_label_3=legend_label_3,
152+
legend_label_4=legend_label_4,
153+
legend_patch_1=legend_patch_1,
154+
legend_patch_2=legend_patch_2,
155+
legend_patch_3=legend_patch_3,
156+
legend_patch_4=legend_patch_4,
157+
)
158+
)
159+
160+
161+
if __name__ == "__main__":
162+
app()

‎src/points_to_h3.py

+94
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
import os
2+
from pathlib import Path
3+
4+
import geopandas as gpd
5+
import h3pandas # noqa: F401
6+
import typer
7+
8+
try:
9+
from typing import Annotated
10+
except ImportError:
11+
# for Python 3.9
12+
from typing_extensions import Annotated
13+
14+
app = typer.Typer()
15+
16+
17+
@app.command()
18+
def main(
19+
input_file: Annotated[
20+
Path,
21+
typer.Argument(help="Path to file containing point layer with GVI scores."),
22+
],
23+
output_file: Annotated[
24+
Path,
25+
typer.Argument(
26+
help="File to write output data to (can specify any GDAL-supported format)"
27+
),
28+
],
29+
cell_resolution: Annotated[
30+
int,
31+
typer.Argument(
32+
help="""H3 cell resolution to aggregate to,
33+
between 0 (largest) and 15 (smallest)"""
34+
),
35+
] = 10,
36+
):
37+
"""
38+
Aggregates points to h3 hex cells.
39+
40+
Args:
41+
input_file: Path to file containing point layer with GVI scores.
42+
cell_resolution: H3 cell resolution to aggregate to,
43+
between 0 (largest) and 15 (smallest)
44+
aggregation_operations:
45+
output_file: File to write output data to
46+
(can specify any GDAL-supported format)
47+
48+
Returns:
49+
File containing h3 polygons with aggregated GVI scores
50+
51+
"""
52+
# Check input file exists
53+
if os.path.exists(input_file):
54+
pass
55+
else:
56+
raise ValueError("Input file could not be found")
57+
58+
# Check input file is a valid file for GeoPandas
59+
try:
60+
gpd.read_file(input_file)
61+
except Exception as e:
62+
raise e
63+
64+
# Check data contains point features
65+
if "Point" in gpd.read_file(input_file).geometry.type.unique():
66+
pass
67+
else:
68+
raise Exception("Expected point data in interim data file but none found")
69+
70+
# Check data contains numeric gvi_score field
71+
72+
# Load input data
73+
gdf = gpd.read_file(input_file)
74+
75+
# Exclude points with no GVI score
76+
gdf = gdf[~gdf.gvi_score.isna()]
77+
78+
# Assign points to h3 cells at the selected resolution
79+
gdf_h3 = gdf.h3.geo_to_h3(cell_resolution).reset_index()
80+
81+
# Aggregate the points to the assigned h3 cell
82+
gvi_mean = gdf_h3.groupby("h3_" + f"{cell_resolution:02}").agg(
83+
{"gvi_score": "mean"}
84+
)
85+
86+
# Convert the h3 cells to polygons
87+
gvi_hex = gvi_mean.h3.h3_to_geo_boundary()
88+
89+
# Export the h3 polygons to the specified output file
90+
gvi_hex.to_file(output_file)
91+
92+
93+
if __name__ == "__main__":
94+
app()

0 commit comments

Comments
 (0)