iamcal.com

37

PNGStore - Embedding compressed CSS & JavaScript in PNGs

By Cal Henderson, August 22nd 2010.

Updated 2010-08-24: There are now some additional results.

Alex Le used a really interesting technique (originally proposed by Jacob Seidelin) for pushing the limits of the 10k Apart Contest - storing JavaScript and CSS in PNG files. The idea is pretty simple - since the contest limits the size of the final files, you can include more code by putting text inside PNGs and taking advantage of the PNG's internal compression.

So I wondered - can this technique be used to make regular web sites faster for very slow connections?

The Variables

The main way in which the real world differs from the contest is that we care about how many bytes go over the wire, not how many are stored on disk. This means that we can use GZip compression to serve compressed text (CSS & JavaScript) anyway. If this technique is going to be any good, it needs to beat GZipped content for size.

While Alex was using a fairly straight forward PNG-8 encoding, I wondered what would happen if I tried to put more bytes per pixel - storing 3 bytes per pixel in a PNG-24 or even 4 bytes per pixel in a PNG-32. This would avoid needing the PLTE (palette) block at the start of the image.

Alex used a very tall image, one pixel wide. I seemed to remember something about image dimensions mattering for PNG size, so try wide, tall and square images and checking for differences seems like a good idea.

Since JavaScript & CSS both tend to only use 7-bit ASCII, can we compress things down either further? With 7 bits of data per character, we could pretty easily fit 8 characters into 7 bytes, by just spreading out the bytes:

11111112 22222233 33333444 44445555 55566666 66777777 78888888

With all of these changes, the issue isn't information density, but final output size. Because of the way internal PNG compression works, some kinds of data will just compress better than others.

The Implementation

I hacked together a quick image generator using PHP and GD. For various input files, create output images with a combination of shape, bit depth and encoding. It's available on GitHub. The file bake.php creates all of the test images, while unpack.htm checks that you can decode them correctly using Canvas. I calculate MD5 sums of the data returned to check that it's exactly as expected.

As further bit depth and compressed encodings are used, the image dimensions shrink. It's not practical to view the wide and tall images, since they are several thousand pixels in a single dimension. Here are some of the square image encodings of the jQuery library:


ASCII 8bit

ASCII 24bit

ASCII 32bit

8-in-7 8bit

8-in-7 24bit

8-in-7 32bit

The visual representation tells you some interesting things about the code. The 8bit images are red because they only store data in the red channel. Data is more uniformly distributed in the 8-in-7 images, since the ASCII images never set the high bit on any of the bytes, so only really use half of the possible values for each pixel (the dark ones). The 32bit images are harder to see, because the transparency makes the whole image faint.

The image dimensions are reduced as the bit depth and encoding packing are increased.

As a final step, I passed the images through Yahoo's Smush.it to remove extra cruft that's not needed to be able to decode the data inside the files.

The Results

Here are the full results for the jQuery library, in table form:

Mode Dimensions Raw Size Smushed Notes
Raw 71,807  
GZipped 24,281  
ASCII 8bit Wide 71807 x 1 25,195 - Failed to load in browser
ASCII 8bit Tall 1 x 71807 29,838 - Failed to load in browser
ASCII 8bit Square 267 x 269 25,826 - No extra compression possible
ASCII 24bit Wide 23936 x 1 42,615 24,487 Smallest, still bigger than GZip
ASCII 24bit Tall 1 x 23936 56,433 34,670  
ASCII 24bit Square 154 x 156 64,588 24,847  
ASCII 32bit Wide 17952 x 1 55,897 34,149 Failed
ASCII 32bit Tall 1 x 17952 62,415 36,666 Failed
ASCII 32bit Square 133 x 135 67,974 34,482 Failed
Seq8 8bit Wide 62732 x 1 42,923 42,335  
Seq8 8bit Tall 1 x 62732 52,529 52,392  
Seq8 8bit Square 250 x 252 43,517 42,863  

Analysis

As expected, the image dimensions don't map directly onto final file size.

The very wide and tall 8bit images failed to load correctly. PNGs have an upper dimension limit of 65535 (0xFFFF), so for input files larger than 64k, the smaller dimension will need to be increased. This limit applies to the dimensions, not the pixels, since the square 8bit image worked fine.

The 32bit images failed to work. It's unclear if this is because PHP-GD doesn't allow them to be set correctly (it only accepts values on a 0-127 scale instead of 0-255) or if there's something special about extracting the values in canvas (compositing mode seems to be set correctly). Regardless, these images came out larger than the 8bit or 24bit versions.

The 8-in-7 packing produced the images with the smallest dimensions, but the largest final files. This is because the highly uniform data is the hardest to compress; the files are the smallest if you turn off PNG compression.

Wide images always did better than tall ones, with square ones a little bit behind wide ones. For large files, square makes more sense since it doesn't break at large input. I suspect this might be related to the way pixels are stored in the IDAT block and perhaps making sure lines are aligned to 8 byte boundaries might be optimal.

The smallest output file was the Wide 24bit ASCII version. However, it was still a couple of hundred bytes larger than the GZipped original and that doesn't take into account the JavaScript needed to extract the code from the image. PNGs use Deflate/zLib compression which works differently to GZip, but it seems as though there is no big saving to be made here.

I additionally tried each PNG with every different combination of the PNG compression filters - Sub, Up, Average and Paeth. The results were for some smaller file sizes for the larger files, but no improvement at the low end.

Conclusion

Storing CSS & JavaScript data in PNGs is a neat hack, but GZipping your source files will get you a larger improvement and greatly simplify your production build process. But still, pretty cool.

Updated 2010-08-24: There are now some additional results.

Post-Script

I did this late one night, so there are probably mistakes. If you spot any of them, drop me an email and teach me: cal [at] iamcal.com

All of the source code can be downloaded from the GitHub repo. Bonus points for fixing the 32bit image encoding. This whole thing was inspired by Alex Le.

Copyright © 2010 Cal Henderson.

The text of this article is all rights reserved. No part of these publications shall be reproduced, stored in a retrieval system, or transmitted by any means - electronic, mechanical, photocopying, recording or otherwise - without written permission from the publisher, except for the inclusion of brief quotations in a review or academic work.

All source code in this article is licensed under a Creative Commons Attribution-ShareAlike 3.0 License. That means you can copy it and use it (even commerically), but you can't sell it and you must use attribution.

37 people have commented

Paul
# August 23, 2010 - 2:48 am PST
This png compression technique is great when your using a server that only has basic hosting (eg no scripting etc) like virgins isp's free hosting as gzipping wouldn't work here.
Mark Fowler
# August 23, 2010 - 4:35 am PST
So, over the wire size isn't the only thing that's important. On the iPhone files over a certain size (25K on older models) aren't cached, and apparently[1] this is depenant on the size of the file when delivered, not the size of the file over the wire.

[1] according to <a href="http://www.yuiblog.com/blog/2008/02/06/iphone-cacheability/">www.yuiblog.com/blog/2008/02/06/iphone-cacheability</a> at least
Cal
# August 23, 2010 - 9:22 am PST
Paul: I had thought about that, but forgotten it somewhere along the line. For modern browsers and limited web hosts it can provide an alternative to gzip. This only make sense for slow connections though, since the code extraction can be slow (around a second for jQuery on my PC).

Mark: Very interesting! I wonder if the unpacking is fast enough to use in iPhone WebKit. Need to do some benchmarking.
Joel Lee
# August 23, 2010 - 10:41 am PST
I recompress the image by PNGOUT and got a 24,074 bytes jquery_ascii_24b_square.png. That is smaller than the gzipped one! Here is the file: <a href="http://imgur.com/W8I9K.png">imgur.com/W8I9K.png</a>
Ryan Grove
# August 23, 2010 - 4:22 pm PST
@Mark: That iPhone caching article is outdated, actually. I published the results of more recent research here: <a href="http://www.yuiblog.com/blog/2010/07/12/mobile-browser-cache-limits-revisited/">www.yuiblog.com/blog/2010/07/12/mobile-browser-cache-limits-revisited</a>
Jos Hirth
# August 24, 2010 - 11:13 am PST
>I recompress the image by PNGOUT and got a 24,074 bytes [...]

Still bigger than gzip. I compressed jquery-1.4.2.min.js with GZRepack (kzip+7z+deflopt) and got it down to 23,866 bytes.

By the way, running Deflopt over that PNG produced by PNGOUT removes another 10 bytes. But then it's still 198 bytes bigger.
Andy Davies
# August 24, 2010 - 2:38 pm PST
Of course the place where this wins over gzip is the scenarios in which gzip doesn't work e.g. something mangles the Accept-Encoding header for "security" reasons.

I seem to remember Steve Souders did a list somewhere.
Thomas Powell
# August 24, 2010 - 3:21 pm PST
Nice post Cal, you clearly explore the motivation of text encoded in binary (this case PNG) for performance this technique this technique and other more insidious ones are of keen interest to malware authors. Smuggling payloads around due to MIME type hijinks or obfuscating in various ways like this allows JS based malware to be difficult to spot without execution of the page. The most ridiculous one I have seen is the encoding of script code in the very white space of a DOM tree. So while we can now see clearly some of the inappropriateness of this for performance it is something of interest outside of that domain as well and worth exploring further.
Cal
# August 24, 2010 - 3:22 pm PST
I've tried a few new images formats and have got some even better numbers: <a href="http://iamcal.github.com/PNGStore/">iamcal.github.com/PNGStore</a>

Patches for any of the outstanding issues would be great.
Jos Hirth
# August 24, 2010 - 4:58 pm PST
>Of course the place where this wins over gzip
>is the scenarios in which gzip doesn't work [...]

Ye, one should compare it with Packer for example. The PNG method should be smaller and the decoding should be also quicker (since the decompression itself is handled by a highly optimized native library).
jpvincent
# August 25, 2010 - 10:22 am PST
there is 15% of people coming to website and not sending a Gzip header, see this PDF, that's a Velocity conference presentation : <a href="http://bit.ly/9K23ZQ">bit.ly/9K23ZQ</a>

so, to summarize :
- there is no size gain over gzip
- it can take a lot of CPU
- if CSS is encoded, it spares one HTTP request
- it could be used where gzip is not used : some cheap hosting company and the 15% of people that come to our sites without Gzip headers
- there is bugs to be aware of

what if we encode with base64 images in CSS, then encode the CSS in the PNG ? :)
do we have a whole site in one single file ?
ChrisS
# August 25, 2010 - 11:02 am PST
PNGs always store an extra byte at the beginning of each row of pixels. This would explain why wide images are better than square or tall images: with tall images you're storing 1 extra byte per byte in the original file. I wonder if the PNG lib you're using is creating full palettes for the 8 bit PNGs, which, if reduced in size (you shouldn't need to actually have all those palette entries), you should get smaller sizes. Even so, the basic PNG algorithm just runs deflate on the image data, so I'm not surprised to see similar sizes.
gonchuki
# August 26, 2010 - 7:34 am PST
Amazing! someone really took this seriously and made a well thought experiment.
My only suggestion would be to test this stuff:
a) other JS files, you shouldn't take conclusions from just one single case.
b) brute force test all widths for the best candidate (either 1b gray or 24b). The jQuery test didn't trigger any PNG filter on PNGOut, but maybe other files or widths can benefit from filtering a few lines.

@ChriS:
PNG recompressors optimize the palettes by default, so if only 1 color out of the 255 entries is used, then the output file will only have that one color.
Joel Lee
# August 26, 2010 - 11:05 am PST
I just got this a few more bytes smaller. The original file is jquery_ascii_t4_8b_wide.png and the recompressed file is <a href="http://imgur.com/LfqiN.png">imgur.com/LfqiN.png</a> (23,725 bytes).
Joel Lee
# August 26, 2010 - 11:37 am PST
Original: jquery_ascii_8b_gray_wide.png
Recompressed: <a href="http://i.imgur.com/KZZv5.png">i.imgur.com/KZZv5.png</a> (23,722 bytes)
Joel Lee
# August 26, 2010 - 11:46 am PST
But 7-zip could compress jquery-1.4.2.min.js to 23,679 bytes easily. PNG loss.
Dathan Pattishall
# August 26, 2010 - 2:30 pm PST
Wow your awesome cal. Great work!
Cal
# August 26, 2010 - 4:46 pm PST
Joel Lee: yeah, it's ultimately not going to beat gzip for most things - both deflate and gzip are based on LZ77 and huffman, and PNGs has extra header overhead.

However, PNG filters are 2D compressors that work for certain patterns of pixels, so changing the file dimensions is likely to yield a couple of combinations that win out. However, that requires you generate every possible size of PNG (about 70k versions for some of these encodings) optimize them all individually and see what you get.

But in situations where you can't use gzip or want to do something else fancy, it gets pretty close to gzip in size.
AVKASH
# September 7, 2010 - 9:03 pm PST
hey..this is amazing technique...
Jan Wikholm
# September 29, 2011 - 11:42 pm PST
I think that at first your mind thinks "binary ==> save space" but over the web I would argue that saving or losing a hundred bytes in either direction is insignificant compared to the amount of overhead saved from HTTP traffic.

Consider having a sprite png with your site's pixel graphics (icons, logos etc) and then appending to that your site's JS and CSS.

You will be saving considerable amount of resources (esp. time) from just the HTTP headers not sent due to them all being shipped as one.

It would be interesting to actually benchmark this.
Ben White
# December 5, 2011 - 6:35 am PST
I'm doing something similar to this, but I'm simply using grayscale and have added pngcrush to the process to really optimize the file size of the PNG. My process can be found here. <a href="http://bwirl.blogspot.com/2011/11/optimize-web-apps-with-png.html">bwirl.blogspot.com/2011/11/optimize-web-apps-with-png.html</a>
PAEz
# March 8, 2012 - 10:55 am PST
Thats really cool!
Really looking forward to when Chrome gets WebP with the new lossless format (recently introduced) and trying this.....the lossless mode gets better compression than a png put through pngout/crush.
Cal
# March 8, 2012 - 11:49 am PST
A good point - will be interesting to see if WebP makes this technique actually useful for size reduction
cb
# May 24, 2012 - 12:11 am PST
Your tests are interesting but maybe you could use various JS files for them. Because of some randomness in the compression chain, I have better results sometimes with RGB, sometimes with Gray format. You might find the tool I made useful : <a href="http://pouet.net/topic.php?which=8770">pouet.net/topic.php?which=8770</a>
Regards
Maurice Svay
# December 14, 2012 - 6:02 am PST
Why not store everything in a zTXt PNG chunk? <a href="http://www.libpng.org/pub/png/spec/1.2/PNG-Chunks.html#C.zTXt">www.libpng.org/pub/png/spec/1.2/PNG-Chunks.html#C.zTXt</a>
Serrurier Montpellier
# July 30, 2014 - 5:09 am PST
thank you
Serrurier Grabels
# August 4, 2014 - 4:36 am PST
A good point - will be interesting to see if WebP makes this technique actually useful for size reduction
serrurier sete
# September 26, 2014 - 3:28 am PST
nice article
Matt Cutt
# March 6, 2015 - 11:21 pm PST
Thats really cool!
Really looking forward to when Chrome gets WebP with the new lossless format (recently introduced) and trying this.....the lossless mode gets better compression than a png put through pngout/crush.
explore quotes
# March 8, 2015 - 10:48 pm PST
I think that at first your mind thinks "binary ==> save space" but over the web I would argue that saving or losing a hundred bytes in either direction is insignificant compared to the amount of overhead saved from HTTP traffic.

Consider having a sprite png with your site's pixel graphics (icons, logos etc) and then appending to that your site's JS and CSS.

You will be saving considerable amount of resources (esp. time) from just the HTTP headers not sent due to them all being shipped as one.

It would be interesting to actually benchmark this.
tonytong
# December 11, 2015 - 6:58 pm PST
i find a free online service to minify js <a href="http://www.online-code.net/minify-js.html">www.online-code.net/minify-js.html</a> and compress css <a href="http://www.online-code.net/minify-css.html">www.online-code.net/minify-css.html</a>, so it will reduce the size of web page.
rohit
# March 30, 2016 - 8:50 am PST
<...>april fool day 2016 jokes<...>
<...>april fool day pranks<...>
<...>april fool day 2016 images<...>
Fan Box Office Collection
# March 30, 2016 - 9:57 am PST
This blog provided all stuff related to Fan Box office collection, total earning, business report
Hostgator Coupons 2016
# April 4, 2016 - 10:11 pm PST
Here you can find all the hostgator offers and deals to take the domain name and web hosting at cheaper cost.
google drive login
# September 12, 2016 - 1:05 am PST
I like your amazing post.thanks for share this with us.
Good and full of knowledge
Programming Assignment Help
# April 5, 2017 - 11:34 pm PST
I really wanted to send a small word to say thanks to you for the fantastic points you are writing on this site.
janadhar
# April 25, 2017 - 2:18 am PST
<...>Janadhar Election Management Software<...>??
<...>Election Campaign in india<...>??
<...>Election Digital Election Managament<...>??

Leave your own comment

Your name: Required
Email address: Optional, will be hidden