[PATCH] io: allow filesystem st_blksize-directed buffer sizes up to 128k instead of 8k

[Collin Funk]

Hi наб,

наб <nabijaczleweli@nabijaczleweli.xyz> writes:

> The only difference between old and new being
>   setvbuf(input, (char *)malloc(128*1024), _IOFBF, 128*1024);
> with 128k matching zfs's st_blksize:
>   Benchmark 1: < f  out/cmd/head -n 290955
>     Time (mean ± σ):     153.4 ms ±   3.9 ms    [User: 78.0 ms, System: 75.3 ms]
>     Range (min … max):   146.5 ms … 159.2 ms    19 runs
>
>   Benchmark 2:  < f ./head.old -n 290955
>     Time (mean ± σ):     259.7 ms ±   5.7 ms    [User: 86.3 ms, System: 173.3 ms]
>     Range (min … max):   249.1 ms … 269.3 ms    11 runs
>
>   Summary
>     < f  out/cmd/head -n 290955 ran
>       1.69 ± 0.06 times faster than  < f ./head.old -n 290955
> (where the output is 290955 lines, 230427757 bytes and the program
>  reduces to a getdelim()/fwrite() loop which reduces to a read()/write()
>  loop).
>
> A value around 128k gets us around 60% of the performance back
> without being so large it should affect the memory pressure.
>
> Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
> ---
>  libio/filedoalloc.c | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
>
> diff --git libio/filedoalloc.c libio/filedoalloc.c
> index f360d96a9b..3a3d6f52d3 100644
> --- libio/filedoalloc.c
> +++ libio/filedoalloc.c
> @@ -61,6 +61,13 @@
>  #include <stdlib.h>
>  #include <unistd.h>
>  
> +/* nfs/cifs/cephfs request 512k/1M/4M. That is a /lot/ for an iostream.
> +   However, many other filesystems request reasonable values,
> +   and clamping them to BUFSIZ of 8k throws away a lot of performance:
> +   for a getdelim()/fwrite() loop at zfs's requested st_blksize of 128k,
> +   over a 220М file, completes in 60% of the time of when the buffer size is 8k. */
> +#define MAX_FS_BUFSIZ (128 * 1024)
> +
>  /* Allocate a file buffer, or switch to unbuffered I/O.  Streams for
>     TTY devices default to line buffered.  */
>  int
> @@ -84,7 +91,9 @@ _IO_file_doallocate (FILE *fp)
>  	    fp->_flags |= _IO_LINE_BUF;
>  	}
>  #if defined _STATBUF_ST_BLKSIZE
> -      if (st.st_blksize > 0 && st.st_blksize < BUFSIZ)
> +      if (st.st_blksize > MAX_FS_BUFSIZ)
> +        st.st_blksize = MAX_FS_BUFSIZ;
> +      if (st.st_blksize > 0)
>  	size = st.st_blksize;
>  #endif
>      }

Adding Pádraig and Paul to CC, who certainly know more about reasonable
buffer sizes than I do.

However, with some trivial testing I recall doing in Coreutils, large
buffers don't always have great throughput.

Here is a trivial example that was run once, so obviously not perfect:

    $ dd if=/dev/random of=input bs=1G count=10 status=none
    $ dd iflag=fullblock if=input bs=$(numfmt --from=iec 8k) \
        status=none | pv -r > /dev/null
    [ 639MiB/s]
    $ dd iflag=fullblock if=input bs=$(($(stat -c %o input) * 512)) \
        status=none | pv -r > /dev/null
    [ 576MiB/s]

That test was done on btrfs without any compression, for reference.

Collin