This is the mail archive of the cygwin mailing list for the Cygwin project.

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: [1.7] Invalid UTF8 while creating a file -> cannot delete?

2009/9/10 Lapo Luchini:
> But the real problem with that test is not really what shows and how,
> the biggest problem is that it seems that filenames created with a
> "wrong" filename are quite limited in usage and can't seemingly be deleted.
> % export LANG=en_EN.UTF-8
> % cat t.c
> #include <stdio.h>
> int main() {
> Â Âfopen("a-\xF6\xE4\xFC\xDF", "w"); //ISO-8859-1
> Â Âfopen("b-\xC3\xB6\xC3\xA4\xC3\xBc\xC3\x9F", "w"); //UTF-8
> Â Âreturn 0;
> }
> % gcc -o t t.c
> % mkdir test ; cd test ; ../t ; cd ..
> % ls -l test
> ls: cannot access test/a-âââ: No such file or directory
> total 0
> -????????? ? ? Â Â? Â Â? Â Â Â Â Â Â Â Â? a-âââ
> -rw-r--r-- 1 lapo None 0 2009-09-10 21:19 b-ÃÃÃÃ
> % find test
> test
> test/a-???
> test/b-ÃÃÃÃ
> % find test -delete
> find: cannot delete `test/a-\366\344\374': No such file or directory

Hmm, we've lost the \xDF somewhere, and I'd guess it was when the
filename got translated to UTF-16 in fopen(), which would explain what
you're seeing:

'find' reads the filename correctly, invokes remove() on it, which
translates it to UTF-16 again, whereby we lose a second byte, so we're
down to a-\366\344, which can't be deleted because it doesn't exist.

> Â Âremove("a-\xF6\xE4\xFC\xDF");

Now here we start with the full name again, so if we lose the last
byte we get what's actually on disk, hence the call succeeds.

Bytes that don't contribute to valid UTF-8 characters get mapped to a
certain subrange of UTF-16 low surrogates at 0xDC80, which is a clever
trick for encoding such bytes into UTF-16 and get them out again after

I stared at the code for this in sys_cp_mbstowcs for a bit, but
haven't spotted where those missing byte might have gone.


Problem reports:
Unsubscribe info:

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]