Skip to content

Commit de29440

Browse files
yeesianyeesian
committed
support streaming osm files through libexpat
1 parent 2293c61 commit de29440

17 files changed

+242
-409
lines changed

‎README.md

+1
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ Pkg.add("OpenStreetMap")
3939
##### Dependencies
4040
The following packages should automatically be added as "additional packages", if you do not already have them:
4141
* LightXML.jl: parsing OpenStreetMap datafiles
42+
* LibExpat.jl: streaming OpenStreetMap datafiles
4243
* Winston.jl: map plotting
4344
* Graphs.jl: map routing
4445

‎REQUIRE

+1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
julia 0.3-
22
Compat
33
LightXML
4+
LibExpat
45
Winston
56
Graphs

‎docs/data.rst

+2-11
Original file line numberDiff line numberDiff line change
@@ -4,18 +4,12 @@ Reading OSM Data
44

55
OpenStreetMap data is available in a variety of formats. However, the easiest and most common to work with is the OSM XML format. OpenStreetMap.jl makes reading data from these files easy and straightforward:
66

7-
.. py:function:: getOSMData(filename::String [, nodes=false, highways=false, buildings=false, features=false])
7+
.. py:function:: getOSMData(filename::String)
88
99
Inputs:
1010
* Required:
1111

1212
* ``filename`` [``String``]: Filename of OSM datafile.
13-
* Optional:
14-
15-
* ``nodes`` [``Bool``]: ``true`` to read node data
16-
* ``highways`` [``Bool``]: ``true`` to read highway data
17-
* ``buildings`` [``Bool``]: ``true`` to read building data
18-
* ``features`` [``Bool``]: ``true`` to read feature data
1913

2014
Outputs:
2115
* ``nodes`` [``false`` or ``Dict{Int,LLA}``]: Dictionary of node locations
@@ -29,10 +23,7 @@ These four outputs store all data from the file. ``highways``, ``buildings``, an
2923

3024
.. code-block:: python
3125
32-
nodes, hwys, builds, feats = getOSMData(MAP_FILENAME, nodes=true, highways=true, buildings=true, features=true)``
33-
34-
**Usage Notes:**
35-
Reading data is generally very fast unless your system runs out of memory. This is because LightXML.jl loads the entire xml file into memory as a tree rather than streaming it. A 150 MB OSM file seems to take up about 2-3 GB of RAM on my machine, so load large files with caution.
26+
nodes, hwys, builds, feats = getOSMData(MAP_FILENAME)
3627
3728
Extracting Intersections
3829
------------------------

‎docs/examples.rst

+1-1
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ Read data from an OSM XML file:
55

66
.. code-block:: python
77
8-
nodes, hwys, builds, feats = getOSMData(MAP_FILENAME, nodes=true, highways=true, buildings=true, features=true)
8+
nodes, hwys, builds, feats = getOSMData(MAP_FILENAME)
99
1010
println("Number of nodes: $(length(nodes))")
1111
println("Number of highways: $(length(hwys))")

‎docs/types.rst

+14-14
Original file line numberDiff line numberDiff line change
@@ -92,10 +92,10 @@ Region boundaries include the minimum and maximum latitude and longitude of a re
9292
.. code-block:: python
9393
9494
type Bounds
95-
min_y # min_lat or min_north
96-
max_y # max_lat or max_north
97-
min_x # min_lon or min_east
98-
max_x # max_lon or max_east
95+
min_y::Float64 # min_lat or min_north
96+
max_y::Float64 # max_lat or max_north
97+
min_x::Float64 # min_lon or min_east
98+
max_x::Float64 # max_lon or max_east
9999
end
100100
101101
Point Types
@@ -110,13 +110,13 @@ Used to store node data in OpenStreetMap XML files.
110110
.. code-block:: python
111111
112112
type LLA
113-
lat
114-
lon
115-
alt
113+
lat::Float64
114+
lon::Float64
115+
alt::Float64
116116
end
117117
118118
Because OpenStreetMap typically does not store altitude data, the following alias is available for convenience:
119-
``LLA(lat, lon) = LLA(lat, lon, 0)``
119+
``LLA(lat, lon) = LLA(lat, lon, 0.0)``
120120

121121
Earth-Centered-Earth-Fixed (ECEF) Coordinates
122122
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -126,9 +126,9 @@ Global cartesian coordinate system rotating with the Earth.
126126
.. code-block:: python
127127
128128
type ECEF
129-
x
130-
y
131-
z
129+
x::Float64
130+
y::Float64
131+
z::Float64
132132
end
133133
134134
East-North-Up (ENU) Coordinates
@@ -139,9 +139,9 @@ Local cartesian coordinate system, centered on a reference point.
139139
.. code-block:: python
140140
141141
type ENU
142-
east
143-
north
144-
up
142+
east::Float64
143+
north::Float64
144+
up::Float64
145145
end
146146
147147
Additional Types

‎src/OpenStreetMap.jl

+1
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
module OpenStreetMap
88

99
import LightXML
10+
import LibExpat
1011
import Winston
1112
import Graphs
1213
import Compat

‎src/buildings.jl

-82
Original file line numberDiff line numberDiff line change
@@ -2,88 +2,6 @@
22
### MIT License ###
33
### Copyright 2014 ###
44

5-
### Create list of all buildings in OSM file ###
6-
function getBuildings(street_map::LightXML.XMLDocument)
7-
8-
xroot = LightXML.root(street_map)
9-
ways = LightXML.get_elements_by_tagname(xroot, "way")
10-
11-
buildings = Dict{Int,Building}()
12-
13-
for way in ways
14-
15-
if LightXML.has_attribute(way, "visible")
16-
if LightXML.attribute(way, "visible") == "false"
17-
# Visible=false indicates historic data, which we will ignore
18-
continue
19-
end
20-
end
21-
22-
# Search for tag with k="building"
23-
for tag in LightXML.child_elements(way)
24-
if LightXML.name(tag) == "tag"
25-
if LightXML.has_attribute(tag, "k")
26-
k = LightXML.attribute(tag, "k")
27-
if k == "building"
28-
class = ""
29-
if LightXML.has_attribute(tag, "v")
30-
class = LightXML.attribute(tag, "v")
31-
end
32-
33-
id = int(LightXML.attribute(way, "id"))
34-
buildings[id] = getBuildingData(way, class)
35-
break
36-
end
37-
end
38-
end
39-
end
40-
end
41-
42-
return buildings
43-
end
44-
45-
### Gather highway data from OSM element ###
46-
function getBuildingData(building::LightXML.XMLElement, class::String="")
47-
nodes = Int[]
48-
class = ""
49-
building_name = ""
50-
51-
# Get way ID
52-
# id = int64(LightXML.attribute(building, "id"))
53-
54-
# Iterate over all "label" fields
55-
for label in LightXML.child_elements(building)
56-
57-
if LightXML.name(label) == "tag" && LightXML.has_attribute(label, "k")
58-
k = LightXML.attribute(label, "k")
59-
60-
# If not yet set, find the class type
61-
if isempty(class) && k == "building"
62-
if LightXML.has_attribute(label, "v")
63-
class = LightXML.attribute(label, "v")
64-
continue
65-
end
66-
end
67-
68-
# Check if building has a name
69-
if isempty(building_name) && k == "name"
70-
if LightXML.has_attribute(label, "v")
71-
building_name = LightXML.attribute(label, "v")
72-
continue
73-
end
74-
end
75-
end
76-
77-
# Collect associated nodes
78-
if LightXML.name(label) == "nd" && LightXML.has_attribute(label, "ref")
79-
push!(nodes, int64(LightXML.attribute(label, "ref")))
80-
continue
81-
end
82-
end
83-
84-
return Building(class, building_name, nodes)
85-
end
86-
875
### Classify buildings ###
886
function classify(buildings::Dict{Int,Building})
897
bdgs = Dict{Int,Int}()

‎src/features.jl

-70
Original file line numberDiff line numberDiff line change
@@ -2,76 +2,6 @@
22
### MIT License ###
33
### Copyright 2014 ###
44

5-
### Create list of all features in OSM file ###
6-
function getFeatures(street_map::LightXML.XMLDocument)
7-
8-
xroot = LightXML.root(street_map)
9-
nodes = LightXML.get_elements_by_tagname(xroot, "node")
10-
features = Dict{Int,Feature}()
11-
12-
for node in nodes
13-
14-
if LightXML.has_attribute(node, "visible") &&
15-
LightXML.attribute(node, "visible") == "false"
16-
# Visible=false indicates historic data, which we will ignore
17-
continue
18-
end
19-
20-
# Search for tag giving feature information
21-
for tag in LightXML.child_elements(node)
22-
if LightXML.name(tag) == "tag" && LightXML.has_attribute(tag, "k")
23-
k = LightXML.attribute(tag, "k")
24-
if haskey(FEATURE_CLASSES, k)
25-
id = int(LightXML.attribute(node, "id"))
26-
features[id] = getFeatureData(node)
27-
break
28-
end
29-
end
30-
end
31-
end
32-
33-
return features
34-
end
35-
36-
### Gather feature data from OSM element ###
37-
function getFeatureData(node::LightXML.XMLElement)
38-
39-
class = ""
40-
detail = ""
41-
feature_name = ""
42-
43-
# Get node ID
44-
# id = int64(LightXML.attribute(node, "id"))
45-
46-
# Iterate over all "label" fields
47-
for label in LightXML.child_elements(node)
48-
49-
if LightXML.name(label) == "tag" && LightXML.has_attribute(label, "k")
50-
k = LightXML.attribute(label, "k")
51-
52-
# If empty, find the class type
53-
if isempty(class) && LightXML.has_attribute(label, "v")
54-
v = LightXML.attribute(label, "v")
55-
if haskey(FEATURE_CLASSES, k)
56-
class = k
57-
detail = v
58-
continue
59-
end
60-
end
61-
62-
# Check if feature has a name
63-
if isempty(feature_name) && k == "name"
64-
if LightXML.has_attribute(label, "v")
65-
feature_name = LightXML.attribute(label, "v")
66-
continue
67-
end
68-
end
69-
end
70-
end
71-
72-
return Feature(class, detail, feature_name)
73-
end
74-
755
### Classify features ###
766
function classify(features::Dict{Int,Feature})
777
feats = Dict{Int,Int}()

0 commit comments

Comments
 (0)