hi, planet && iso8859-15 to utf8 with netatalk magic
Hi, Planet!
Welcome to my blog, and thanks to Martin for subscribing me. I m not going to bother you with any details about me, if you have any questions feel free to ask me on freenode or oftc (in case you re wondering: my nick is bzed).
After a longer time of planning we ve upgraded our main file server at
work: New machine, new storage (7TB - woohoo!), Etch instead of Sarge and
UTF-8 for file names instead of
ISO8859-15. Everything not a too complicated thing, except for renaming the about 10,000,000 files from ISO8859-15 to UTF-8 in a proper way.
In theory this is easy, as
convmv happily takes care of the problem. But this did not work for us unfortunately, as we re not only providing NFS and Samba services, but also AFP (
Apple_Filing_Protocol) via netatalk. Years ago you had to choose ISO8859-15 if you wanted to allow your Apple and Windows users to see the same, non-broken file names - not only because AFP didn t support UTF8 before version 3, which came out with OSX
1). Normally you d think that a server would not allow a client to write a file with a file name which includes not supported characters - again, OSX and netatalk go their own way here:
Characters which are not supported by the underlying file system are character(!)-wise encoded using the
CAP encoding. So we had to find a way to
- convert file names from ISO8859-15 to UTF-8
- replace CAP encoded characters by the appropriate UTF-8 character
Luckily netatalk ships the
uniconv(1) util
2) which is able to take care of this problem. It also updates the CNID
3) Database at the same time. There s two things you need to take care of in case you re using the dbd
CNID backend:
- uniconv assumes that cnid_metad is listening on port 4700, you need to start it first. Don t run afpd at the same time!
- If you ve specified external CNID dbpaths in your AppleVolumes.default (normally they re located in the root of each volume, prone to be deleted by users accidentally), you need to move the .AppleDB directory into your volume s root directory, otherwise uniconv will create a new CNID database. Just move it back as soon as uniconv is finished.
Be careful: uniconv assumes that all colons in a file name are part of a CAP encoded character. That s not a problem as long as you have only Mac and Windows clients accessing your server, as both systems do not allow colons in file names. In our case we found a number of cache files from Nautilus (which is running on our sunray/Solaris web-surf terminals as default browser) with colons in the file name. I hope no user will miss them

Including a final synchronisation of all files to the new storage, about two hours it took uniconv to convert the file names and a aunch break, we were finished after about 14 hours of work. Until now it seems that our users didn t notice the change at all, except for those who realized that they have much more free space now. Looks like a successful weekend so far!
1)
and uses a NFD variant of Unicode normalization (
UTF-8#Mac_OS_X), but netatalk takes care of this and translates it into the NFC form, which is used on most other platforms
2)
it s called netatalk-uniconv in Debian and shipped with
netatalk
3)
Catalog Node ID, another crust from old Mac
OS versions