Making of the Maroc-Topo maps
=============================

outdated as of 31th October 2006... no time to write texts...

  This document briefly describes the process of generating the maps.
The whole map sources are available upon request. They are a lot larger
than the final map package. There are numerous other map-building tutorials
around in the web. There are also infinitelly many possibilities of organizing
a larger map project. I just wanted to write it down "my way" so that it's
easier for someone interested to find his way through my maps...

  One of the main goals was to keep the map as modular as possible, i.e. to
have separate files for separate things. E.g. the altitude lines are
automatically generated with 100m distances currently. If some day more memory
becomes available on the devices, it would be convenient to just create new
contour lines and keep the rest of the map. If someone has an old GPS-device
with few memory, I could easily create a "road only" map. Only as a last step
before exporting, building and releasing the final map, all the sources
are merged into one file and sent to cGPSmapper.

  Also large maps are usually split up into smaller map tiles. This is quiet
natural, similar to having a different paper map for different areas of a
country. So these two criteria were the main basis for structuring the map
of Morocco, which has become quiet large by now.

  Furthermore there were three major data sources which I used for the map.
First this is the SRTM data for contour lines, then VMAP0 for a basemap of
the world, and finally tracklogs and waypoints for refining some portions
of the map. In the remaining document I will go through all the individual
steps necessary for each of these sources, that finally led to the map in
its current form. First the process for generating the land contours
(altitude lines) is described. Then the import process for the publically
available basemaps are explained. Finally the processing of track recordings
from GPS-users is covered. The final map combination, some postprocessing
cosmetics and the compilation steps are concluding the document. Almost.
There are a lot of references to tools, free data and tutorials in the very
end, more like an appendix.

0. Map tiles
  The map is split up into smaller patches at 1 degree boundaries. The
filenames are derived from the center point of the map patch. E.g.
30N-31N 09W-10W will get the name 03050095. This resulted in reasonably
sized tiles and has not made any problems so far. With this grid I currently
have like 50 map tiles, and each of them is completely separate from all the
others. For each of them I have a separate directory, where I performed the
steps from below.


1. Land contours from SRTM data
  Mostly I followed the instructions from the tutorials, for this part
specifically [blauesboot]. The SRTM data used is publicy available for
non-commercial use [ciat-srtm].

  The Shuttle Radar Topography Mission or something created a huge set of
elevation measurements around the whole globe. Well, the poles are not covered,
but for Morocco the SRTM data was a great resource! Basically the data is
stored as some huge TIFF images, each pixel covering an area of 10x10 meter
(or 100x100 or I don't know how large) and the "color" of each pixel is
indicating the altitude at that pixel. In short: it is the most accurate
measurements available for free, and it really *is* accurate!

  But to make a map out of it, I first had to cut the SRTM data into smaller
pieces (my map tiles from above) and then convert it into vector format (as
all GPS-maps are stored in form of vectors). The latter sounds complicated, but
really is the simple part of it! In the tutorials I found the cutting was
usually done with the 3DEM-tool. But as this program never did what I wanted,
I used the long and difficult way by foot.

  The CIAT SRTM data comes in tiles of 5x5 degrees each. And it is "seamless",
which means, the pixels of adjanced tiles do not overlap. This is bad for
vectorization, as in fact only the altitudes at the *center* of the pixels are
stored in the TIFF, while the 10meter or something to the next pixel center
are "empty". If you now want to draw the contour lines between the centers of
the pixels, you will get a gap between each two tiles! It is really important
to understand this pixel-center and gap problem, to understand the way of
cutting the TIFFs correctly. 

  Now I started cutting the TIFFs with some image processing software, as I
wanted to have one small TIFF for each of my map tiles. I used the
ImageMagick-convert program. Any other program might do, but two important
things have to be taken into account. First the simple one: the tool you use
has to read and write 16bit signed pixels. I manually had to patch my
ImageMagick to do so, the Photoshop Elements shipped with my scanner totally
failed... maybe gimp might work? Now for the difficult one, which in fact all
image processing programms mess up with.

  A GeoTiff is a .TIF-image with geographical positioning information included.
These special tags are ignored and thrown away with mostly any standard image
processing tools. However, there is a libgeotiff around on the net, which can
not do anything to the image data and pixels, but read and write these
geographical information. First it can be used to read out the geo-information
to a text file. You can then have a look at that file, edit it, and write it
back into different .TIF files. This is done with the libgeotiff-tools
listgeo.exe (for reading) and geotifcp.exe (for writing).

  I just started with reading the original geo-information from the the
original files shipped by CIAT. Then I took a look at the text files. Included
is a field indicating the the coordinates of the upper left corner pixel and
the size of each pixel. Further there are some map-datum and miraculous stuff
inside, which I simply ignored. So I made a new text file, changing only the
upper left pixel to match the small 1 degree tiles I cropped out of the huge
5 degree tiles. It's rather easy indeed, there is a TiepointTag containing
the position and some zeros and there is a PixelScaleTag, telling how far a
step from one pixel to the next is. Use a simple calculation to get the
"Tiepoint" of pixel (10,10) or whichever else. Note there could also be some
rotation in your GeoTiff-files, which could make things difficult, but the SRTM
GeoTiffs I found were just great.

  Finally I added the .geo-Information to the small tiff cropped out (which
up to now is a simple image-TIFF with no geo information). With geotifcp.exe
(libgeotiff again), I turned off compression and tried anything to make the
file as plain easy readable as possible!

  Now back to the gap-problem. Basically I wanted my 1 degree patches, e.g.
30N-31N 9W-10W. In the original 6000x6000 pixel patches covering 5 degrees,
this meant 1200x1200 pixels per small map tile. In theory! Due to the gap
problem (remember: lines connecting the center points, if the tiles do not
overlap, there is a gap between two rows of pixel center points) I added one
column of pixels to the left and one row of pixels at the bottom, resulting
in 1201x1201 pixels per map tile. The left and bottom (west and south) were
chosen due to the exact alignment of the SRTM-data. Stepping one pixel to the
left also has to be respected when editing the TiepointTag!

  An example set of commandlines:

---

# cut pixels (1199,1200) - (2400,2401) including the border
# i.e. cut 1201x1201 pixels from position (1199,1200)
convert -page 1201x1201+0+0 s_35_07.tif -crop 1201x1201+1199+1200 02850085.tif

# find the geo-information in the big source file
listgeo s_35_07.tif > s_35_07.geo

[ edit the text file moving Tiepoint from (-9.9995833 30.0004166) ]
[ to the new tiepoint of the small map    (-9.0004166,29.0004166) ]
[ save new file as 02850085.geo ]

geotifcp -g 02850085.geo -c none 02850085.tif 02850085_geo.tif

---

  Okay, this was tricky. And for the map tiles at the boundary of the large
5 degree input files, you sometimes have to merge two of them and everything
gets a real mess... it took me weeks to get it really accurate, but now I'm
happy with that solution! One final note about my lazyness: I've not merged
the two large original SRTM tiles for generating tiles 03050045 and 03050035,
therefore these two do have the "gap problem" on the southern boundary. That
one is going far into Algeria and can be ignored for Morocco alone... noone
will ever notice it, I hope! :)

  But the good news is, the remaining work about the contour lines is done
completely automatically. I loaded the GeoTiff DEM files into the great tool
DEM2TOPO. There you can first check the position of the small map tile visually
and then click on "Create .mp file" to vectorize the contours. The numerical
values used in the process:
  - minor lines:        100m intervals
  - intermediate lines: 200m intervals
  - major lines:        400m intervals

  Finally the output .mp's were postprocessed using GPSMapEdit. This is
necessary, as the xyz_geo.mp files are the "main" files in my Morocco-map,
i.e. the levels, IDs, map names and everything is configured in here. So
these values have to be filled with sensible data. First the ID of the
maps was changed to match the tilename (02850085). Without setting the ID
unique, MapSource is in trouble. For saving the file the format
"POI, POLYGON, POLYLINE" was used, resulting in smaller files than RGNx0.
Finally the default levels were changed:
  - Level0: 22bits, mapsource 0: <1.2km
  - Level1: 20bits, mapsource 1: 1.2km-3km
  - Level2: 18bits, mapsource 2: 3km-8km
  - Level3: 17bits, mapsource 3: 8km-12km
  - Level4: 16bits, mapsource 4: 12km-30km
  - Level5: 14bits, mapsource 5: 50km-120km

  This can be done with a standard text editor as well, as .mp files are
plain text-files luckily. Especially for the levels, copy and paste works much
faster that way!

  Congratulations, you have survived the first, and most difficult part, and
now have your contour lines!

2. Basemap
  There is a free basemap of the whole world, called VMAP0. It is kindly
provided by many different sources, I used the version by ESRI, see the
references below. It has rivers, roads, cities, airports and some basic land
coverage (trees, etc.). Altogether the data is not extremely accurate, but it
provides a good basis to start your own map with. And by the way, use any
mapping server on the world and click to a really remote place (Tibet,
Kamtchatka, Patagonia or Morocco), you will get VMAP0-data!

  I downloaded the relevant map tiles from the ESRI-site. Note the selection
tool on the homepage is once again not very accurate and never does what I want.
A little bit of HTML helped me, but otherwise just see the hints far down at
the end, in the postprocessing chapter.

  After importing the ESRI-data into MapEdit I just started clicking around
wildly. Basically there are 4 "layers of information" to be extracted from
the map, and I stored each of these in a different file (remember: modularity,
replace any of them and keep the rest). So for each map tile I created the
following files:
  - 03050095_geo.mp : containing the altitudes and only these
  - 03050095_water.mp : containing oceans, rivers and lakes and nothing more
  - 03050095_cover.mp : containing "orchard", "forest", "scrub" and the like
  - 03050095_city.mp : containing cities, villages, and all the other POIs
  - 03050095_road.mp : containing all kinds of roads and pistes

  The xyz_geo.mp was created above from the SRTM data. It was not touched in
the remaining process.

  The xyz_water.mp was not only created from ESRI-data. In fact during the
SRTM mission also water bodies were identified in some places and are available
as SWBD (SRTM Water Bodies, see link in the references). This data was also
imported into GPSMapEdit and the rivers and water bodies from VMAP0 were merged,
which involved some serious interactive work (looking up river names, assigning
a level to each of the things, deciding whether to trust SRTM or VMAP0, ...).
Also note, several small rivers can be "joined" (not "merged") to bigger
elements, which have the same name and settings. Also you can select several
rivers and then have to "Modify" their settings just once!

  - Really large rivers (those having a name in the basemap even): Level 3
  - Large rivers: level 2
  - Small rivers from ESRI-basemap: level 1
  - Large lakes: typically Level 2, sometimes level 3

  - Type of rivers: 0x1f or 0x26
  - Type of laked: according to size

  The xyz_cover.mp was really the simplest and most useless part. No
postprocessing whatever was applied, just import and accept it.

  - completely residing in level 0 and only there...
  - Trees are 0x15 (the same as in official Garmin maps as well)
  - Scrubs are 0x4f (might make sense to change to green?)
  - Cropland is 0x4e (again: just make it "green"???)

  The xyz_city.mp were in fact not created from the ESRI-version of VMAP0,
but umm... from the DCW version... and it was no fun, converting the .e00
files to anything at all... They just popped out of the nothing, I really hope
to find a better source of VMAP0 some day... :(

  - Capital Cities (Agadir, Marakech,...): Level 7
  - Big cities (Taroudant, Essaouira,...): Level 4
  - Important cities (Ait Baha, Tata,...): Level 2-3
  - remaining cities: Level 1
  - unimportant settlements: Level 0

  - all settlements with name are at least type 0xd00 or larger, so the
    font size in mapsource is not messed up


  Finally the xyz_road.mp were imported from VMAP0. The roads were assigned
names (e.g. N1) and types (e.g. "Major Highway"). All roads I did not find
any information about were kept at level 0, so they can be updated later on
and do not disturb the overall accuracy of the map.

  - Major highways: Level 5 or above (i.e. always visible)
  - not so Major highways: Level 4
  - Main roads: Level 3
  - Collector & arterial roads: Level 2
  - city roads: Level 1
  - undefined roads: Level 0

  - Type "Major Highway": Two lanes per direction or larger
  - Other highway types: Asphalted, red roads in Michelin Map
  - Arterial Road: Mostly asphalted, yellow and large white roads in Michelin
  - Collector Road: Not asphalted
  - "Unpaved Road": Offroad or near-offroad tracks

  Note a better solution for detailed city maps yet has to be found. Probably
creating an transparent overlay would be the best, but I have not thought too
much about this so far.



  Finally note, it is not at all important if the VMAP0 data is going across
the bounds of the map tiles at the moment. There is a neat "Trim-Tool" in
GPSMapEdit: Select any area, right click and go to properties. Now you can
enter the exact 1x1 degree boundaries and remove the outside. This is also
explained in the final postprocessing steps below.


3. Adding Map Data

3.1 Tracks and Trails
  GPSMapEdit allows to conveniently convert tracks to pieces of a vector-map.
But that's pure hand-work, and some automatisation is desirable and possible.

  First I had to convert data from different formats and assign it by the
relevant map-tile. I started doing this manually in MapSource, tried some
different programs and finally gave up. At least cutting the data into handsome
pieces can and should be done automatically. I used gpsbabel, another handy
free tool. It came along with a "polygon filter" allowing to specify a region
to extract waypoints from a large file. It did not work for tracks however. As
I'm a computer scientist (as you might have guessed already) I sat down and
extended this gpsbabel functionality. Now my own personal version did cut the
tracks and to convert all contributed tracklogs and cut them into the map tiles
I just have to run one simple command. The cutting functionallity might become
publically available in some future release of gpsbabel, I don't know yet.

  Now, one more thing I was too lazy to do by hand was to manually merge
multiple tracklogs of the same road. I searched the net and found a programm
called TopoFusion. It provides all the functinality any standard gps-program
does, and one neat extra: "Make Network". The results are not really great to
my opinion, they heavily depend on the quality of the inputs, but at least the
multiple-recording problem and manual interaction was drastically reduced! As
a drawback this programm will throw out all waypoints, only read and write
.gpx-files and some more things that make pre- and postprocessing steps
necessary...

  And then I sat down, clicked the tracks, named them and got hungry... That's
the most time consuming part of the whole process!


3.2 Points of Iterest
  This is really simple with GPSMapEdit. Just convert waypoints to POIs.
Some types I've been using:

Label (0x2800, point)
Parking (0x4d00, point)  (only visible up to 700m)
Picnic area (0x4a00, point)
Scenic area (0x5200, point)
Controlled area (0x5600, point)
Border crossing (0x3006, point) (no symbol in Mapsource :( )
Radio beacon (0x1602, point)
Tide prediction (0x1d00, point)  (river crossing)

Drinking water (0x5000, point)
Camping (0x4800, point)
Restaurant (0x4500, point)   (only visible up to 700m)
Hotel/Motel (0x2b01, point)   (only visible up to 700m)

Auto Repair (0x2f03, point)   (only visible up to 700m)
Gas station (0x4400, point)

Shopping (0x2e00, point)  (only visible up to 700m)
Bank/ATM (0x2f06, point)
First aid station (0x4b00, point)
City hall (0x3003, point) (=Building, only visible up to 700m)
Ground transportation (0x2f08, point) (=Bus station)
Marina (0x4300, point) (=harbour)
Telephone (0x5100, point)
Historical (ghost) town (0x6415, point)

4. Building the map

  I've already mentioned GPSMapEdit several times above, it is just *the* tool
used for creating the map. But it does not create Garmin-compatible maps itself.
Anyway, the first step in building the final, usable map is to merge all the
different source files into one (remember: modularity and so on!)
These are currently the five files mentioned above: geo, water, cover, city,
road. By the way, the levels have to agree in the individual sources, to be
able to merge them in a meaningful way. Otherwise you will notice when trying
to look at the merged map.

  More of these separate files could contain detailed city maps and similar
things. Anyway, load all the files, and DO NOT SAVE! Always be sure to use
"Save As" at this point. Before saving, apply Tools->Generalize (removes
unnecessary points from polylines and polygons, without affecting the accuracy
of the map) and Tools->Remove Duplicates (remove points you accidentally added
twice), and potentially the Trim-Tool (cut out the exact boundary of the map
tile) and then you are ready.

  The preferred file-format of GPSMapEdit is "The Polish Format". This works
great, unless you want to use the map anywhere else than in GPSMapEdit.
Therefore you need cGPSmapper, a commandline-tool to convert "The Polish Format"
into Garmin-compatible maps (Different tools for AlanMap, Magellan etc. might
be around, never tried it so far). cGPSmapper is rather obscure in all its
details, but it can be called completely from within GPSMapEdit, so you don't
have to care all the details. On the other hand, once you know what to do you
can easily call it from the commandline as well.

  Compiling the map is basically done by your computer, its really simple. But,
well, the process might take like 10-20 minutes, as it is not that simple at
all. But you can go and read a book in between, in fact I built some new wheels
for my bicycle while compiling the first few versions of the map.

  The resulting .img can be sent to your map-capable GPS-handheld, in theory. I
was able to send it to my Vista C at least. But it can not be loaded into
MapSource. MapSource needs an index file, a .tdb . This file links arbitrary
many .img files together, e.g. one .tdb for whole Maroc linking to many small
.img files of 10x10km each. For the Morocco map, I introduced the tiling of
1 degree latitude and longitude each, that's like 50x50km. And all the small
tiles are linked together in the index.

  To make a .tdb and a large scale preview map out of the small tiles, a
hand-written xyz_pv.txt file is needed. It is a bit obscure again to write a
working text-file. Drop me a mail, if in doubt. Having the file you simply
invoke cGPSmapper to build the map index and everything else you might possibly
need. And after some tries I finally succeeded, my map showed up in MapSource.

  Commandlines:
  cgpsmapper.exe 03050095.mp     (compile a single tile)
  cgpsmapper.exe pv 03000090.txt (compile the preview and index)

  That's it. A long way, but not really too complicated, is it?



5. Other things, Todos, ...

  - The VMAP0 rivers do not agree to the SRTM contour lines... I'm thinking of
    writing an automatic tool to fix this, butthis is not a promise!
  - There is a GeoNameServer GNS to be found on the internet. It contains
    thousands of names for towns, landmarks and everything. However, up to now
    I found it to be pretty useless, as the acuracy of the points is about 1-2km
    or so. Practicaly and for Morocco this means typically three points with
    three different names are stacked at the same position. I've written a small
    perl-script converting the GNS text files to .mp however, and maybe one day
    I'll make use of it...
  - Citymaps are best created from calibrated paper maps or satelitte images.
    Tracklogs are just not accurate enough for that!
  - I'm looking for a better alternative to TopoFusion, again, maybe I might
    feel like writing one myself one day...
  - I'm tired now and go to bed! Probably you should as well! Good night!


References
==========

  Tools used:

ImageMagick
  available from: http://www.imagemagick.org
  used to cut GeoTiffs correctly

libgeotiff
  available from: http://www.remotesensing.org/geotiff/geotiff.html 
  used to make GeoTiffs from the Tiffs cut with ImageMagick

DEM2TOPO
  available from: http://people.uleth.ch/~brad.gom/dem2topo/
  used to convert GeoTiffs to vector maps in "The Polish Format"

gpsbabel
  available from: http//gpsbabel.sourceforge.net/
  extended the polygon filter to trim the tracks to map tiles

TopoFusion
  available from: http://www.topofusion.com/
  experimentally used for preprocessing tracks (fuse multiple recordings)

GPSMapEdit
  available from: http://www.geopainting.com/en/
  main program used to edit the maps

cGPSmapper
  available from: http://www.cgpsmapper.com/
  used to convert "The Polish Format" into Garmin compatible format

MapSource
  available from: http://www.garmin.com/
  used to test the map


  Free data sources used:

[ciat-srtm]
  Hole-filled seamless SRTM data V1, 2004,
  provided by: International Centre for Tropical Agriculture (CIAT)
  available from: http://gisweb.ciat.cgiar.org/sig/90m_data_tropics.htm
  containing altitudes of the whole world

SWBD: STRM Water Bodies
  provided by: ...
  available from: ftp://e0srp01u.ecs.nasa.gov/srtm/version2/SWBD/
  containing the raster-like waterbodies currently used in the map

ESRI world base map
  provided by: ...
  available from: http://www.esri.com/data/download/basemap/index.html
  containing all the other elements, used as a basis to add more details


  Tutorials and other references:

[blauesboot]
  MapEdit Tutorial
  available at: http://www.blauesboot.de/MapEditManual/howto.html
  basic introduction to all the tools and lots of useful information about
  where to get data and what to do with that data


GeoTiff editing Tutorial
  available at: http://www.3dnature.com/tutorials/NLCD/NLCD4_body.htm
  explains how to access and edit the geo referencing information in GeoTiffs

