By Cal Henderson, June 16th 2012.
TL;DR Version: Check out World of MapCraft
A few years ago, I came across MapWoW, an excellent use of the Google Maps API to show the World of Warcraft map. The map is made of of the WoW minimap images, stitched together to visualize the world. This gives much better resolution than the in-game main map and is really interesting to poke around. The obnoxious ads and inability to use the mouse to zoom are annoying, but not a deal breaker.
Fast forward to December 2011 and the map hasn't been updated to include the changes in the Cataclysm expansion - the old world has changed significantly and new zones have been added. I thought I'd have a go at creating my own version.
When I first came came across MapWoW I had assumed that the images had been screenshots from the game client and then painstakingly glued back together by hand. This is obviously madness. In the intervening time, I had written a bunch of addons for WoW and understood a little about how textures work inside the game - obviously it should be possible to extract the minimap textures from the game client somehow to build the map.
The first relevant thing I found was this excellent article from some time during the Burning Crusade expansion. The author's goal was to measure certain aspects of the game world, but he included extensive information on extracting minimap textures. Appendix 2 is where all the meaty stuff can be found, but the entire thing is worth a read. Go ahead, I'll wait.
While the technical details were super useful and confirmed that I was headed down the right path, a lot of the details have changed as newer expansions have been released; we'll get into those shortly. The two big takeaways were that all of the textures were stored inside the game client's MPQ files and were encoded as BLP files. I had messed around with BLPs in the distant past for Hunter Loot, but MPQs were new to me.
MPQ files, short for Mike O'Brien Pack, are archives used by Blizzard to store game content - graphics, sound, strings, movies - anything that the game client needs. Blizzard games have been using MPQ files for a long time (since at least Warcraft II in 1995) and the format hasn't changed a whole lot. Originally these files were accessed using a file
storm.dll which came with the games. Early on, developers figured out the storm.dll API and used it to read data from MPQ files for their own apps. But storm.dll could only read from MPQs, not write to them. If you wanted to modify a texture in Diablo, you needed to be able to write to an MPQ. LMPQAPI was the first library that was able to do this (used by StarEdit), followed in 2000 by the Stormless MPQ Editor.
Around this time, StormLib was created as a C++ library to read, write and modify MPQ files. It's still maintained today and is the basis for lots of current MPQ viewers and editors.
MPQ files themselves are fairly complex - they contain some headers, special files, a hash table, a block table and the file data. To find a file, you need to hash the name to find the hash table entry, then the block table entries, then the actual data. In practice this is very fast (much faster than traversing a file system) and allows different kinds of compression.
In older games, MPQ files did not contain a file listing - you could only access files if you knew their name already (since they are stored in the hash table as the hash only). This changed before World of Warcraft, and MPQ files contain a special 'list' file, listing all the paths of contained files.
MPQ files also have the ability to be patched by more MPQs. You can load a base MPQ that defines the files 'a', 'b' and 'c'. Loading a patch MPQ on top could modify 'b', remove 'c' and add a file called 'd'. StormLib makes all of this transparent to the caller - after loading the base MPQ and the patch, it just sees 'a', 'b' (new version) and 'd'. Patches can be stored as BSDDiffs or complete replacements. In practice, MPQs are merge-loaded - that is, multiple base (non-patch) MPQs are all loaded into the same 'namespace', as far as the caller is concerned. Internally, StormLib maintains an ordered 'patch chain' of archives. When you request
"world/maps/AhnQiraj/AhnQiraj.tex", the first MPQ in the chain is checked. If the file is not found, the next archive in the chain is checked. Once the file has been found, the next archive is checked for a patch to it. If a delete flag is found in a patch, the process starts over from that point in the chain.
For the current patch level, World of Warcraft (on Windows) has the following MPQ files:
art.MPQ- art assets
base-Win.MPQ- Windows executables (client, launcher, updater, etc)
expansion[1-3].MPQ- art and world files from expansions
world.MPQ- world geometries (mostly WMOs and ADTs)
wow-update-nnnnn.MPQ- patches with a base prefix
wow-update-base-nnnnn.MPQ- patches without a base prefix
enUS/locale-enUS.MPQ- strings and interface textures
enUS/speech-enUS.MPQ- speech mp3s
enUS/expansion[1-3]-locale-enUS.MPQ- strings from expansions
enUS/expansion[1-3]-speech-enUS.MPQ- speech from expansions
enUS/wow-update-enUS-nnnnn.MPQ- patches for locale info
All of the files are divided into two chunks - generic (art, world and sound) and locale-specific (strings, speech). If you're not playing in US-English then your client files will not be called 'enUS'.
Some patch files have a 'base prefix', which means that all files within the patch have a prefix on their path. For instance, if
"Textures/Moon02.blp", then a patch may contain changes to it called
"base/Textures/Moon02.blp". This seems to have been used to allow patching of locale-specific and non-locale-specific data in a single patch, but is not used for newer patches (since rev 13914). You need to tell StormLib about the prefix for a patch when opening it so that it knows how to merge the file indexes together.
When I started out, I found Ladik's MPQ Editor for Windows, which allowed me to poke around in my MPQ archives. I knew I'd need to do some mass-extraction, so I hunted for a command-line solution. I first found mpq-tools which used libmpq. Unfortunately, libmpq's main website has been down for a while. After some wrangling with
pkgconfig to get it built, it worked.
If you're following along at home, I need to set some environment variables to get it to configure and then run:
PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./configure make LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/mpq-extract
Unfortunately, this utility wasn't able to extract files to their correct filename - just to their hash. This was going to be an uphill battle.
After some poking around the guts of libmpq, I stumbled across MPQ Extractor (github search is a wonderful thing), a completly different MPQ utility. It uses StormLib, which is included in the source. The only extra hoop was that it required cmake to build. MPQ Extractor did exactly what I wanted.
After uploading the MPQs that I needed to a server (a mere 12.3 GB), I started to unpack them. The basic unpacking looks like this:
MPQExtractor -e "World\Minimaps\*" -f -p wow-update-1*.MPQ wow-update-base-1*.MPQ -o out art.MPQ
This extracts files matching the path spec
"World\Minimaps\*", maintains the path structure once extracted (
-f), applies some patches (
-p) and sticks the extracted files in a subfolder. Easy.
This worked great, mostly. During extraction, I would get a bunch of errors about files that couldn't be extracted. Some files that I could see in the Windows app were not being extracted. Some files which had been extracted were corrupted.
Much of this was caused by patch application and ordering. Patches have to be applied in the right order to make sure that the diffs are being applied against the right base. Explicitly passing each patch, in order, fixes the ordering problem:
MPQExtractor \ -e "World\Minimaps\*" -f -p \ mpqs/wow-update-13164.MPQ \ mpqs/wow-update-13205.MPQ \ mpqs/wow-update-13287.MPQ \ mpqs/wow-update-13329.MPQ \ mpqs/wow-update-13596.MPQ \ mpqs/wow-update-13623.MPQ \ mpqs/wow-update-base-13914.MPQ \ mpqs/wow-update-base-14007.MPQ \ mpqs/wow-update-base-14333.MPQ \ mpqs/wow-update-base-14480.MPQ \ mpqs/wow-update-base-14545.MPQ \ mpqs/wow-update-base-14946.MPQ \ mpqs/wow-update-base-15005.MPQ \ mpqs/wow-update-base-15050.MPQ \ -o out mpqs/art.MPQ
But this wasn't quite working either. There's an option that allow you to pass a patch prefix (
--prefix xxx), which tells StormLib what the base prefix is for patches. Passing this makes the old patches apply correctly, but not the new ones. Not passing it causes the opposite. There is code inside StormLib that tries to be smart about it, but it wasn't working. I patched the code to allow per-patch prefixes, which makes all patches apply cleanly:
MPQExtractor \ -e "World\Minimaps\*" -f -p \ mpqs/wow-update-13164.MPQ,base \ mpqs/wow-update-13205.MPQ,base \ mpqs/wow-update-13287.MPQ,base \ mpqs/wow-update-13329.MPQ,base \ mpqs/wow-update-13596.MPQ,base \ mpqs/wow-update-13623.MPQ,base \ mpqs/wow-update-base-13914.MPQ \ mpqs/wow-update-base-14007.MPQ \ mpqs/wow-update-base-14333.MPQ \ mpqs/wow-update-base-14480.MPQ \ mpqs/wow-update-base-14545.MPQ \ mpqs/wow-update-base-14946.MPQ \ mpqs/wow-update-base-15005.MPQ \ mpqs/wow-update-base-15050.MPQ \ -o out mpqs/art.MPQ
The patches would cleanly apply and files were no longer corrupt. But I was still missing some files I expected to be there. I tracked it down to files that were added in a patch with a prefix - the search function in StormLib (
SFileFindNextFile()) did not correctly find files added in a prefix patch because it was not stripping the prefix. After patching this, all of the files were extracted correctly.
Well, almost. There is also another issue on non-Windows filesystems. The hashing algorithm used in MPQs (Jenkin's) is case-insensitive, so all files in the archive are essentially case-free.
FoO.bLp are the same file. However, because we get the names from the list file, the case depends on entries there. For whatever reason (I'd guess sloppiness), the case varies across patches and even within a single MPQ. Two files which should be in the same "folder" can have completely differently cased paths:
This causes problems when trying to find which textures belong together. I patched a new option into MPQExtractor to allow all paths to be transformed to lowercase. You can grab my version from github. We can finally extract the files we need. Onwards!
Blizzard games store textures in a format called BLP, which is a proprietry wrapper around S3/DirectX Texture Compression (also known as DXT) texture data. BLPs are a lossy-compressed raster format that WoW uses to store all interface images - every button, dialog and UI element is made from hundreds of BLP textures.
There are a few Windows apps for converting BLPs to TGA and PNG format, but the choices on the command line are a little slim. Since I'll need to convert many thousands of textures, doing them one by one is not a great option. Luckily the author of MPQExtractor also created BLPConverter, a small wrapper around libsquish, which makes bulk conversion trivial. Just point it at your BLP files and let it loose.
BLPConverter -o output_folder -f png input_file.blp
The default options (mip level zero) are fine.
After extraction, we're left with around 22,500 PNGs to sort through. Understanding how they are organized is a bit of a challenge.
Finding the main parts of the world is fairly straight forward. Simply opening up
/world/minimaps/azeroth/ finds 1000 PNG files. Ignoring the ones starting with
noliquid_ for now, they all have the format
mapXX_YY.png. Lay these out in a grid, ignoring the missing images, and you'll see the Eastern Kingdoms (gaps added to show how big each image is).
This isn't the whole of Azeroth, however. Looking through the other top-level folders, there are some good candidates that contain hundreds of PNGs -
Northrend and the mysterious
Expansion01. Laying out the images from the latter in a grid, you get a bit of a spectacle.
This map contains everything added in the Burning Crusade expansion, all mashed into one map; the Outlands, Azuremyst & Bloodmyst Isles and the new zones in the north of the Eastern Kingdoms. These would need to be manually glued together with the existing tiles to form the world map.
It turns out that the offsets of these chunks onto the main world map is stored in the DBC files. I'll get into those in detail later, but they basically act as a simple client-side database, containing all sorts of information about the word map and a million other things.
There are also some map chunks added for Cataclysm that work in a similar way - the Lost Isles, Tol Barad and the Maelstrom (seen briefly during the intro quest to Vash'jir).
Armed with a list of images and their positions on the main map, I went about creating a tileset to use with the Google Maps API. Luckily, I had a big advantage here - Google Maps tilesets consist of 256x256 pixel images, which is exactly what the WoW PNGs were - I needed only to name them with a sensible scheme (merging all the different map image sets into a single set with consistent naming based on position) and glue them to the API.
Different zoom levels (so that you can zoom out) are pretty straight forward in this setup too. Since each zoom level is double/half of the next, creating a zoomed out version just requires taking a square of 4 tiles (2x2), glueing them together and resizing to 50%. I used the swiss army knife of imaging, ImageMagick, to do this with a little PHP glue.
Besides the main world map, Cataclysm introduced some new parts of the world that are "elsewhere" - Deepholm and Vash'jir. The latter was stored slightly differently to every other map (a theme which popped up constantly - WoW internals have a very cobbled together feel), with the tiles being found as
/world/minimaps/azeroth/noliquid_mapXX_YY.blp. This is presumably because Vash'jir is really in Azeroth, just under the ocean (you can see it on the main map - the parts where you can come out of the water).
I considered putting these two (and Outlands) on the main map, as MapWoW has originally done, but they had different background colors and didn't really belong together. At this point I decided that multiple tilesets were probably the way to go.
Of course, nothing is ever that easy. Even with these straight forward maps, there was a certain amount of fiddling to be done. Here is the raw tileset for Deepholm.
The areas of the minimap outside of the navigable space contain lots of noise and junk. In some cases, they contain text left in by the developers, or even other (unused) maps. To clean these up, I picked a background color for each map (a grey, in the case of Deepholm) and cut out chunks from around the outside until the image looked clean. The final version of Deepholm looks a lot better.
After adding a bunch more maps (which we'll talk about in a moment), I decided I needed to add some drop-down menus to make things more manageable. I'm not sure if I've lost my Googling powers, but nearly everything I could find was terrible. I was looking for something like the menus that power Wowhead, but had no luck. In the end I went with a pure CSS solution that works well enough, but I'd like to add some delay in there so that sub-menus don't jump around if your mouse goes outside their area for just a second. If you have any pointers, please let me know in the comments.
So I had the main world maps, but there were a lot more folders full of images. Alongside the maps I'd already found were a handful of battlegrounds, dungeons and raids, but not nearly all of them. Of the ones that were there, some were pretty odd, like this map of the Black Temple.
I went into BT and had a look around; this really wasn't the full minimap. Looking further into the PNGs, there were some other likely BT candidates stored under the path
/world/minimaps/wmo/dungeon/blacktemple. They were, however, a jumble of different small maps, some with only a single image. These were also stored bottom-up, with the Y-values decreasing instead of increasing, so my preview tools showed them wrongly.
It took me a while to figure out what was going on here, but basically any time you went 'indoors', the maps were stored in the 'wmo' folder, all cut up in pieces. This explained why I had some full instances and battelgrounds, some only partially and some missing completely - the Mount Hyjal raid all takes place outside, so was in the main files, while many instances are 100% 'indoors' and only showed up as jumbled chunks. Mixed instances like Black Temple had pieces of "world map" and separate "indoor" chunks.
My first plan was to just manually assemble the chunks into maps. I started with Ragefire Chasm, since it's about the simplest instance in the game. The map is comprised of 12 distinct chunks, two of which had more than one PNG that made them up.
After a few hours of lining up pixels (lots of the bits look very similar and I had put them in the wrong place), I threw up my hands. If the WoW client is able to correctly show minimaps, there must be some information stored somewhere that describes how they fit together.
I mentioned DBC files earlier - that's short for Data Base Client and they contain information about all sorts of things. You can browse the list of all the DBC files on the WoWDev wiki. This contained lots of likely candidate like DungeonMap and AreaTable which I poked around for a few days. I learned a lot about the internals of WoW, but didn't find anything that described the minimap.
The best bet was stored inside the
Map.dbc file, which contained information about places on the world map. Each record pointed to a group of files in the MPQs. For the Ragefire Chasm record (ID 389), it had the value of
OrgrimmarInstance. Inside the MPQ, were the following files:
TEX files appeared in Cataclysm and are poorly understood; probably not what I was looking for. The
WDL file describes a low-resolution heightmap for an area of the game world and is much better understood. Third party tools that preview the world geography use the
WDL files. Looking into the file for RFC showed that it was basically empty. Probably not our candidate either.
WDT file describes how the standard map tiles (like the ones from the main Azeroth map) fit into this map area. Again, the RFC file was basically empty. However they can also point to a
WMO that appears on the map, along with positioning information for it. In the RFC file, it pointed to a
WMO file with a familiar name.
Finally, a clue! Looking into this folder shows a number of similar files.
The 12 numbered files match the 12 map chunks from the RFC tile images. WMO files contain the definition for World Map Objects, which are structures in the world like statues, buildings and entire instances. Each WMO consists of a "root file" describing the entire object and a series of "group files" which describe chunks of the object. A small house might just contain a single group file, while a complex raid could contain hundreds. Each group file describes a 3D box of data - vertexes, textures, lighting and so on.
If you're interested in how Blizzard build out the world, this video gives a little insight into their toolkit.
The interesting piece of information here is stored in the MOGI chunk of the root file - it describes the position of the group within the whole WMO, using two 3D coordinates for opposite corners of the bounding box. Luckily for me, groups are not rotated at all and are square to the WMO itself (even if the WMO is rotated when placed in the world). While WMOs are a binary format, the chunked structure has been fairly well documented by people picking apart the files. Parsing it is a simple case of reading the bytes and converting them into something we can use. The relevant code is in wmo_build.php in the
By extracting the bounding box for each group, dropping the height coordinate and then mapping these to the PNGs found in wmo folders earlier, I was able to combine the PNGs into a coherent map. Here's the merged map for RFC, with the bounds for each group within the WMO outlined and numbered.
You can see that the size and shape of each group varies wildly - some contain four 256x256 pixel tiles, while some are tiny. In other maps, some groups contain no minimap images at all - pieces of the world which are never represented on the minimap.
With a single merged image for the instance (called a 'flat' in my source), creating a tileset was straightforward. I cut the image into 256x256 squares, then created multiple zoom levels by combining groups of 4 tiles. Easy! We're almost there.
Many of the instance maps were pretty straightforward to assemble once I'd figured all this out, so I set about building a map for every battleground, dungeon, raid and major indoor location in the game. Of course, there were still some annoying issues to deal with. These ended up sucking up more time than the entire project up to this point. Urgh.
As with the world tile sets, the WMO tiles sometimes contained random junk. The main difference was that some of these 'garbage' chunks were huge, taking many minutes to render the flat images from. Here's the flattened image for Karazhan.
No idea what that giant mess is - it's not shown in game, nor is the blue rock junk around the main hall (a few people have written in to point out that it's Netherspace, seen while fighting Prince Malchezaar. However, you only ever see the small square at the very bottom edge of it in-game). A few other instances had similar pieces of junk attached. Sometimes cleaning it up was as easy as removing a single WMO group, but often it involved trimming pieces from a single WMO group, much like the world maps.
To help it take the crown of most annoying instance to deal with, Karazhan also suffers from typos. Some files are labelled Karazhan, some Kharazan and some Karazan. Fuck that place.
Some instances and raids include both indoor and outdoor segments. To deal with these, I flattened a portion of the world map that represented the outdoor portion, then combined them with the interior segments. Unfortunately, indoor and outdoor minimaps use a hugely different scale. The outdoor maps are all of the same scale, while interior minimaps vary from WMO to WMO. In the case of the Magister's Terrace it's a difference of about 4 to 1. Rather than sizing down the interiors, which would suck since they're so nicely detailed, I just showed the two pieces alongside one another. The effect is a little weird, but probably the best I can hope for.
This is most noticeable in the map for Ulduar, where the first 4 bosses are outdoors, then you head inside for the rest, apart from Freya who is outside again.
When I assembled the map for the Black Temple, it took a long time and then looked like this.
There is the now familiar indoor/ourdoor issue to deal with, but something new too. The main temple interior consists of many overlapping floors and the image shows them all on top of each other. I wanted the maps to be actually useful (and god knows I got lost in BT enough times), so that wasn't going to work.
I started building assembled images with all the WMO groups labelled with their number, so I could pick out which pieces belonged on which layers.
The caverns of time are mostly simple, but you can see many overlapping pieces in the bottom left corner where the ramp spirals up. This allowed me to pinpoint which of the groups I wanted to turn on or off for rendering a given 'floor'. For the Black Temple, I then created seven separate flat images for the different floors. Once I had the different floors (and the exterior yard for BT), I layed those out into a single large flat image and used that as the base for the tileset.
This involved a ton of trial and error, figuring out which groups were needed. In some cases, a group is shown on multiple levels to help show how the levels fit together. I was helped a little by going back to the MOGI chunk of the WMO root file - by sorting chunks by their vertical position I was able to easily cull chunks which were obivously far above/below the floor I was dealing with. You can see that the final map is pretty understandable compared to the single squished image.
The maps were all ready and working, but they felt sluggish. A quick
du showed that the main tileset for Azeroth was a whopping 709 MB. No single user would ever download the entire tileset (that would mean zooming all the way in, then panning around endlessly - the world is big), but that's still huge.
I have done some experimentation with PNG compression in the past, so I turned first to PNGOUT to shave off some bytes. The first layer of the main map went well, running for a few hours. When the conversion script hit the first zoom level, however, it halted. Because of the way I had assembled the zoomed layers, many of the PNGs has an alpha channel. The PNGStore code contains a script called
peek.pl which lets you quickly check which of the 14 subtypes a PNG is. The ones I was producing were type 6 with 8-bit channels.
In the case where a zoomed-in tile did not have 4 images on the layer below (blank pieces of the map did not have images, so this occured at island edges) I had left a quarter (or more) of the tile transparent. WMO-assembled tilesets failed completely, as the assembly process had created transparent flats to start with, as the individual groups inside a WMO were of course transparent (they overlap a lot). PNGOUT is unable to work with alpha channel images, for whatever reason, but this also pointed to an easy optimization - alpha channels take up a lot of space that we could save, simply by replacing those pixels with the solid color of the map's background color (the ocean, in the case of the main map).
Just removing the alpha channel made a huge saving, for no difference in appearance. After that, running PNGOUT shaved a further 10% from the total size of the main map.
|Original||49 MB||709 MB|
|Opaque||29 MB||391 MB|
|Crushed||27 MB||353 MB|
The main map took about 5 hours to crush, with the other instances and raids taking a further 5 or so - a task to run overnight.
The final piece of the puzzle was to stick all the tiles onto a CDN and have the served with sensible caching headers. Luckily I already had S3 and Cloudfront configured, so it was just a case of uploading a few tens of thousands of images.
With all of the tilesets generated, the site was complete. At least until Mists of Pandaria comes out.
In the course of poking around the MPQ files and reading the DBC database, I learned a bunch of things about how the WoW world was originally arranged. This points to some interesting facts that I haven't seen talked about very much. If you're a WoW fan, you might find these interesting.
LD_prefix), Khaz Modan in the middle (
KZ_prefix) and Azeroth at the bottom (
More of these odd/changed names can be found by browsing the WMO BLP index.
After many hours of looking at PNGs, I've never located the tiles for the Dalaran Sewers. If you visit them in-game, there are definately distinct mini-map images when in the sewers, not just the regular Dalaran tiles. I've considered writing an addon to try and pull the path to the textures from the client, but haven't quite bitten the bullet on that yet.
If you can find them, please let me know so I can include them in the 'Misc' list!
Some people pointed out that this has been available for some time in-game through Addons like Carbonite and Cartographer. I'm well aware of that! It's a much easier task to use game textures insde the game. You can also visit every part of the world and walk around.
Others have pointed to other Google-maps style WoW maps that have been around for a while, like mapwow (wyrimaps & Marlamin are my faves). None of the others have battleground, dungeon, raid or city maps. They also tend to be clunky and pretty slow. I'd like to take some of the decent features (like zone labels and points of interest) and merge them into mapcraft.
The biggest requested feature is street view. This is a significant challenge, but probably possible. It will require digging into the 3D terrain models and rendering some panoramic images (many thousands of them). An interesting challenge though!
If you spot any glaring mistakes or omissions, drop me an email and teach me: cal [at] iamcal.com
I play a Night Elf Hunter called Bees, who currently has a collection of 196 vanity pets. I am the creator of Hunter Loot, wrote and maintained the Hunter module for Rawr during BC and helped rewrite WarcraftPets.
Copyright © 2012 Cal Henderson.
The text of this article is all rights reserved. No part of these publications shall be reproduced, stored in a retrieval system, or transmitted by any means - electronic, mechanical, photocopying, recording or otherwise - without written permission from the publisher, except for the inclusion of brief quotations in a review or academic work.
All source code in this article is licensed under a Creative Commons Attribution-ShareAlike 3.0 License. That means you can copy it and use it (even commerically), but you can't sell it and you must use attribution.
Comments have been disabled