Osmotron
Converting Open Street Map Files
Recently, we created a small command line tool in one of our projects to convert Open-Street-Map (OSM) files into other file formats, such as WKT, WKB or GeoJSON. Since we found this tool to be rather helpful, we released it into the wild under the name Osmotron. The name stems from the Osmium lib, which we use under the hood to perform the conversion. We released our project under the boost license, so anybody is welcome to use, fork and modify it!
Usage
Once you have built the tool, you can just point it so an OSM-file and let it do its thing:
./osmotron <path-to-osm-file>
But wait, there is more! You can actually configure many aspects of the conversion with parameters (such as the type of output geometries and formats). As tradition has it, you can actually get a list of all parameters with the following command:
./osmotron --help
So why would you want to use such a tool? As it turns out, the OSM XML format is (at least in my opinion) a bit unwieldy. It contains a set of nodes (=points on the map), ways (=lines on the map) and relations between them. I get it — it’s a graph and graphs are mathematically beautiful. It is also rather easy to convert the input from people walking around with GPS recorders into the OSM format. However, if your data structure is not able to represent polygons as first class citizen, a lot of developers will shudder when they have to use it for geo data. It is also completely unreadable by humans due to its constant referencing of nodes in other parts of the document. Funnily enough, the official wiki lists readability as a “pro” for this file format.
Fortunately, we have other formats such as WKT (Well Known Text), WKB (Well Known Binary) and GeoJSON. WKT and GeoJSON are easily readable by humans. In addition, they are standardized formats supported by many applications and databases. WKB is only machine-readable, but has the benefit of being extremely compact and very performant due to the low parsing overhead.
Performance
The Osmium lib really does a great job of converting even large OSM files in a matter of seconds. As a reference point, converting a 20MB OSM file with Osmotron took under one second on my machine (which includes reading the input and writing the result). As with all performance data on the Internet, you might want to take this with a grain of salt though.
Logging
Osmium provides a nice feature when parsing OSM files: it can report any problems it encounters with the input, such as duplicate nodes or open rings.
So Osmotron not only converts your data, it also validates the data while doing so - what about that?
If you don’t want to know about all the dark corners of your data you can of course turn off the logging inside the provided logging.conf file.
You can also change other aspects of the logging process in that file, so have a look at it when using Osmotron!
Build / Install
To build Osmotron you have to follow these steps:
- Install all the required osmium dependencies
- Download the source code
- Use
cmake -G <your IDE>to generate the project files and fire up your favorite IDE to open the project - OR use
cmakedirectly to create a makefile, e.g. like so$HOME/cmake/bin/cmake -D OSMIUM_INCLUDE_DIR=$HOME/libosmium/include $HOME/build/DisyInformationssysteme/osmotron - Your
osmotronexecutable should then be placed in thebindirectory
We have a public CI build, so if you have any problems you can always have a look in the build config file to see how it’s done.
More Build Stories
If you are still here, gather round the fire and let me tell you a little bit about the monster that is the C++ build process. Most of the time spent on this project was drained into getting various parts and libs of the program to build. First of all, I tried to build a small osmium sample project on windows using CLion with a colorful mix of cmake parameters. After a few hours of trying to fix the various “lib XYZ could not be found” errors I gave up on CLion and tried to build the project with Visual Studio. Funnily enough, Visual Studio produced a completely different set of error messages, but I was still unable to fix them all. This is also the reason why there is currently no Windows build and only the Ubuntu version…
This problem is rather symptomatic for many C++ open source projects: often enough, the projects themselves are not too complex and would compile just fine on any platform. However, many of them are littered with a lot of external dependencies (just look at the insane amount of required libs to build osmium), which you as developer have to hunt down. Making sure that a project with many dependencies is easy to build on all platforms takes a lot of constant effort, so many don’t even bother.
This is also one of the reasons why many C++ libs are delivered as a single “header-only” file with no external dependencies: you just plug it in and it compiles with no hassle — exactly what a frustrated C++ developer is looking for. This is apparently such a big advantage, that header-only libs are very popular besides having a lot of disadvantages (slower compile times, cluttered interface, bad maintainability, etc.).
Another funny part of this project was to get Travis CI to build our project. I have the impression that the guys at travis really don’t like C++ developers, because their build toolchains are ancient. For example, Osmotron requires the very moderate cmake version 3.2, but travis only support cmake up to version 2.8. So, if you have a look at our travis build file, we actually have to download the sources for cmake 3.3 from cmake.org and build cmake from scratch for every single build. The result is that most of the built time is being spent on building cmake instead of Osmotron. :)
The title image Toolbox was published by Florian Richter under CC‑BY 2.0.
Share via
