Comments on: Zip files and Encoding – I hate you.

By: gobi

gobi — Mon, 24 Aug 2009 10:49:17 +0000

The unzip command has -O and -I options to specify source filename encodings.

If archive was done on Windows u use -O option some thing like:

unzip -O sjis yourarchive.zip

-I option is used if you archived it on Linux/Unix with diffirent option.

By: O. Andersen

O. Andersen — Fri, 24 Jul 2009 14:15:55 +0000

Windows-1252 is actually a superset of ISO 8859-1 (disregarding control characters which will not appear in file names anyway), so your example is technically incorrect: encoding as ISO 8859-1 and decoding as Windows-1252 would work perfectly fine.

You later mention CP437 as an encoding used on Windows machines, but also say that “everyone” ignores the specification, which says that only CP437 or UTF-8 should be used. (CP437 is effectively incompatible with ISO 8859-1 and Windows-1252.) I am confused as to what encoding Windows actually uses/assumes. Do some versions use Windows-1252 and others CP437? Please clarify. (Obviously, other encodings must be used for non-Western demographics, as touched upon by another commentator, but let us leave that for now.)

By: Christopher Warner

Christopher Warner — Fri, 08 May 2009 20:38:42 +0000

Blame all the programmers who think that encoding doesn’t matter and refuse to get on the UTF-8 bandwagon even though the rest of the world has long since been on the bus.

By: Todd Morrison

Todd Morrison — Wed, 08 Apr 2009 16:15:27 +0000

Thank you for this research.. i have been facing this problem recently.

By: noep

noep — Mon, 08 Dec 2008 18:19:25 +0000

Serves people right for using languages other than english.

By: Cary Clark

Cary Clark — Mon, 08 Dec 2008 17:18:56 +0000

Future zip executables (compressors) should assume filenames are in the system encoding (a very reasonable assumption in my opinion) and convert them to UTF-8 in the created zip files.

By: WAHa.06x36

WAHa.06x36 — Mon, 08 Dec 2008 13:01:27 +0000

Also, tar is a very unflexible and limited format, and not much good for any platform with filesystem metadata of any kind, which is pretty much all of them these days.

By: WAHa.06x36

WAHa.06x36 — Mon, 08 Dec 2008 12:59:20 +0000

There’s nothing “weird” or “nonstandard” about the OS X Unicode decomposition, it’s just plain NFD as far as I know. Now, if unicode decomposition was the only problem, this would all be trivial.

But the real problem is the already mentioned ISO-8859-1, Windows-1252, CP437, and the as-yet unmentioned Shift_JIS, EUCKR, Big5, ISO-8859:s 2 through 15 or however many there are, and so on, and so on.

Really, the only way to reliably open a zip file is to either ask the user for the character encoding (and he probably doesn’t know), or to try and autodetect it.

I’ve had some success using Mozilla’s universalchardet to open Zip files in http://code.google.com/p/theunarchiver/. A friend is currently also helping getting some of the core code to run on Linux. It’s all Objective-C, though, which will probably scare people off from using it.

By: Mike Seth

Mike Seth — Mon, 08 Dec 2008 12:49:22 +0000

Use tar(1)?

By: kL

kL — Mon, 08 Dec 2008 12:15:48 +0000

I haven’t got much success with tar.bz2 either.

There’s The Unarchiver for Mac OS X which tries to guess encoding of filenames.

Since UTF-8 can be mostly-reliably distinguished from 8-bit encodings, I think it should be required for all decompressors.

And NFD is Mac OS X’s problem, not ZIP’s. If some app tries to use bytes in filenames that system simply does not allow by definition, then that’s bug in the app, and app should be fixed.

I think going forward, all ZIP-dependent specs should require filenames in UTF-8 and forbid applications from relying on any particular Unicode normalization.