[Bug libc/17322] New: Unexpected results from using posix_fallocate with nfs target

["jferlan at redhat dot com" <sourceware-bugzilla at sourceware dot org>]

https://sourceware.org/bugzilla/show_bug.cgi?id=17322

            Bug ID: 17322
           Summary: Unexpected results from using posix_fallocate with nfs
                    target
           Product: glibc
           Version: 2.18
            Status: NEW
          Severity: normal
          Priority: P2
         Component: libc
          Assignee: unassigned at sourceware dot org
          Reporter: jferlan at redhat dot com
                CC: drepper.fsp at gmail dot com

Created attachment 7761
  --> https://sourceware.org/bugzilla/attachment.cgi?id=7761&action=edit
Sample posix_fallocate program (cc -o posix_fallocate posix_fallocate.c)

Working on a bug for libvirt which resulted in one of the reviews indicating
perhaps I had found a glibc bug.  Hopefully I explain things well enough - at
the very least there are a couple of links which can show a bit of history. If
you need more data - I'd be happy to oblige... I did try googling to see if
something like this had been reported before, but didn't see anything.

I'm running an f20 environ which I try to keep "up to date". A "yum list
installed | grep glibc" returns:

glibc.i686                     2.18-12.fc20       @updates                      
glibc.x86_64                   2.18-12.fc20       @updates                      
glibc-common.x86_64            2.18-12.fc20       @updates                      
glibc-devel.x86_64             2.18-12.fc20       @updates                      
glibc-headers.x86_64           2.18-12.fc20       @updates                      

First the libvirt bug:

https://bugzilla.redhat.com/show_bug.cgi?id=1077068

Which led to me submitting some patches that got some questions where I thought
perhaps I had answered what I had seen well enough:

http://www.redhat.com/archives/libvir-list/2014-August/msg00367.html

However, after making some changes as a result of my first patch effort, I got
the following review:

http://www.redhat.com/archives/libvir-list/2014-August/msg01074.html

Which led me down the path of create a test program and try to explain what I
think the bug is and see what I get.

Essentially the problem is that for an NFS target on libvirt some tester was
trying to create specifically sized files and then used a libvirt command which
essentially does a stat() on the resulting file to ensure the size specified
was the size created.  Worked quite well for all types, except NFS.

This issue may be related to NFS, but I'm not quite sure. As it turns out NFSv4
will allow "larger" 'wsize' values and will do some autonegotiation on your
behalf if you don't provide a size - so in my case - that's going to be a wsize
of 1MB.

It seems as though this "wsize" is used in some matter during the
posix_fallocate() in the math that would generate the "sized" file that the
tester wanted.

In the example from the above bz, it was a 10 MiB file which would return an
'st_size' from stat() of "88".  The libvirt code would then say the size of the
volume was 88*512 or 45056 bytes/44.0 KiB.  Not quite what was expected. For a
non NFS file, the 'st_size' was 20480 which ends up being the right size.  If I
modify the nfs server startup to change the "wsize" - I can get different
results all the way down to 4096 in which case I get the "expected" results.

Using the attached (hopefully) program - I've more or less reproduced the
sequence of calls libvirt makes and results in a similar trace that is shown in
one of the links above:

% strace ./posix_fallocate /home/nfs_pool/target/bug10mib.img 10
...
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7fabd997d000
write(1, "Open '/home/nfs_pool/target/bug1"..., 74Open
'/home/nfs_pool/target/bug10mib.img' for fallocate of 10485760 bytes
) = 74
open("/home/nfs_pool/target/bug10mib.img", O_RDWR|O_CREAT|O_EXCL,
03017576725020) = 3
write(1, "Before fallocate_posix\n", 23Before fallocate_posix
) = 23
fstat(3, {st_mode=S_IFREG|S_ISUID|S_ISVTX, st_size=0, ...}) = 0
write(1, "stat st_blocks=0 st_blksize=1048"..., 46stat st_blocks=0
st_blksize=1048576 st_size=0
) = 46
write(1, "size in libvirt KiB would be 0\n", 31size in libvirt KiB would be 0
) = 31
fallocate(3, 0, 0, 10485760)            = -1 EOPNOTSUPP (Operation not
supported)
fstat(3, {st_mode=S_IFREG|S_ISUID|S_ISVTX, st_size=0, ...}) = 0
fstatfs(3, {f_type="NFS_SUPER_MAGIC", f_bsize=1048576, f_blocks=137675,
f_bfree=101444, f_bavail=94428, f_files=8962048, f_ffree=8174933, f_fsid={0,
0}, f_namelen=255, f_frsize=1048576}) = 0
pwrite(3, "\0", 1, 1048575)             = 1
pwrite(3, "\0", 1, 2097151)             = 1
pwrite(3, "\0", 1, 3145727)             = 1
pwrite(3, "\0", 1, 4194303)             = 1
pwrite(3, "\0", 1, 5242879)             = 1
pwrite(3, "\0", 1, 6291455)             = 1
pwrite(3, "\0", 1, 7340031)             = 1
pwrite(3, "\0", 1, 8388607)             = 1
pwrite(3, "\0", 1, 9437183)             = 1
pwrite(3, "\0", 1, 10485759)            = 1
write(1, "fallocate ret=0 errno=0\n", 24fallocate ret=0 errno=0
) = 24
write(1, "After fallocate_posix\n", 22After fallocate_posix
) = 22
fstat(3, {st_mode=S_IFREG|S_ISVTX, st_size=10485760, ...}) = 0
write(1, "stat st_blocks=88 st_blksize=104"..., 54stat st_blocks=88
st_blksize=1048576 st_size=10485760
) = 54
write(1, "size in libvirt KiB would be 44\n", 32size in libvirt KiB would be 44
) = 32
close(3)                                = 0
exit_group(0)                           = ?
%

My NFS Server is yes on the same host as the client program.  I didn't do
anything special to set it up - used the following sequence of steps:

mkdir /home/nfs_pool
mkdir /home/nfs_pool/nfs-export
mkdir /home/nfs_pool/nfs

edit /etc/idmapd.conf to change/add the uncommented line:

Domain = localhost.localdomain

edit /etc/exports and add:

/home/nfs_pool/nfs-export localhost(rw,no_root_squash)

Then use 'service nfs restart'

% exportfs -v
/home/nfs_pool/nfs-export
       
localhost(rw,wdelay,no_root_squash,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
%

I used libvirt to start the pool, but all it does is essentially mount things,
resulting in a mount -v showing:

% mount -v
...
localhost:/home/nfs_pool/nfs-export on /home/nfs_pool/target type nfs4
(rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp6,port=0,timeo=600,retrans=2,sec=sys,clientaddr=::1,local_lock=none,addr=::1)
...
%

FWIW: 

Compared to a write on just my /home partition:

%  strace ./posix_fallocate /home/bug10mib.img 10
...
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7fcb8c3eb000
write(1, "Open '/home/bug10mib.img' for fa"..., 58Open '/home/bug10mib.img' for
fallocate of 10485760 bytes
) = 58
open("/home/bug10mib.img", O_RDWR|O_CREAT|O_EXCL, 03017576725020) = 3
write(1, "Before fallocate_posix\n", 23Before fallocate_posix
) = 23
fstat(3, {st_mode=S_IFREG|S_ISUID|S_ISVTX, st_size=0, ...}) = 0
write(1, "stat st_blocks=0 st_blksize=4096"..., 43stat st_blocks=0
st_blksize=4096 st_size=0
) = 43
write(1, "size in libvirt KiB would be 0\n", 31size in libvirt KiB would be 0
) = 31
fallocate(3, 0, 0, 10485760)            = 0
write(1, "fallocate ret=0 errno=0\n", 24fallocate ret=0 errno=0
) = 24
write(1, "After fallocate_posix\n", 22After fallocate_posix
) = 22
fstat(3, {st_mode=S_IFREG|S_ISUID|S_ISVTX, st_size=10485760, ...}) = 0
write(1, "stat st_blocks=20480 st_blksize="..., 54stat st_blocks=20480
st_blksize=4096 st_size=10485760
) = 54
write(1, "size in libvirt KiB would be 102"..., 35size in libvirt KiB would be
10240
) = 35
close(3)                                = 0
exit_group(0)                           = ?
%

-- 
You are receiving this mail because:
You are on the CC list for the bug.