is this a bug in glibc or readpst?

[Paul Wise]

Hi folks,

I have a strange bug that might be an issue in glibc or it might be a
bug in how readpst works and or uses freopen in multi-process mode.

This is a summary of how the debugging process went up until now:

readpst from Debian buster in multi-process mode works but readpst from
Debian bullseye randomly loses some data. Current readpst works on
Debian buster but not Debian bullseye. The problem isn't related to the
GCC optimisation level. The problem isn't compiler related, clang
exhibits the problem too. Upgrading libc6 from 2.28-10 to 2.29-1 caused
the issue. Bisecting glibc pointed at commit 0b727ed4d, which is titled
"libio: Flush stream at freopen (BZ#21037)" and looks legitimate as it
aligns glibc freopen with POSIX specifications. readpst is using
freopen() after fork() to get new *.pst FILE pointers for child
processes. Both the parent and child FILE are opened read-only. The
FILE position is 0 after freopen for both scenarios. readpst seems to
be skipping some PST file blocks in the broken scenario. The debug logs
seem to indicate that in the broken scenario it reads data from a wrong
location, even though the file position is 0 after freopen. Switching
the readpst code to use fclose()+fopen() after fork() instead of
freopen() after fork() fixes the issue.

Here are the glibc freopen commit and the initial libpst bug report.

https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=0b727ed4d605d9318cb0d323c88abb0d5a441a9b
https://github.com/pst-format/libpst/issues/7

The readpst forking and freopen functions are here:

https://github.com/pst-format/libpst/blob/main/src/readpst.c#L203
https://github.com/pst-format/libpst/blob/main/src/libpst.c#L395

I have been debugging this using the attached scripts on a Debian
system. The outside script sets up a chroot using Debian schroot and
then runs the inside script to do the test inside the chroot. If you
want to run the inside script you may want to customise the paths it
uses so the script doesn't put files in your home directory.

I am hoping someone can help give me an idea if this is likely to be a
problem in glibc or readpst/libpst and or what other debugging strategy
might be useful to figure that out and pinpoint where the problem is.

-- 
bye,
pabs

https://bonedaddy.net/pabs3/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: outside
Type: application/x-shellscript
Size: 698 bytes
Desc: not available
URL: <https://sourceware.org/pipermail/libc-help/attachments/20221123/f2fdd1b2/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: inside
Type: application/x-shellscript
Size: 1244 bytes
Desc: not available
URL: <https://sourceware.org/pipermail/libc-help/attachments/20221123/f2fdd1b2/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL: <https://sourceware.org/pipermail/libc-help/attachments/20221123/f2fdd1b2/attachment.sig>