This is the mail archive of the
libc-help@sourceware.org
mailing list for the glibc project.
read() cant read files larger than 2.1 gig on a 64 bit system
- From: glide creme <glidecreme at gmail dot com>
- To: libc-help at sourceware dot org
- Date: Fri, 25 Dec 2009 17:34:18 -0500
- Subject: read() cant read files larger than 2.1 gig on a 64 bit system
- References: <431f8c1a0912251330l266dd23blae9d425b65e04162@mail.gmail.com>
Sorry if I'm asking on the wrong mailinglist,
but I've asked on various forums,mailinglist and I havn't found a solution yet.
The problem:
I can't read files larger than 2 gig on my ubuntu64bit using posix
'read()', the problem is not related to the allocation itself, which
works.
Attached is a sample program that illustrates this.
I've tried with various, compiler flags like
-D_FILE_OFFSET_BITS=64
According to man 2 read, the maximum is limited by SSIZE_MAX
which is defined in
/usr/include/bits/posix1_lim.h
# define SSIZE_MAX LONG_MAX
And LONG_MAX is defined in /usr/include/limits.h as
# if __WORDSIZE == 64
# define LONG_MAX 9223372036854775807L
# else
# define LONG_MAX 2147483647L
# endif
# define LONG_MIN (-LONG_MAX - 1L)
Can anyone clarify if this is a bug in libc, in my build setup etc.
Thanks
edit:
by the way
$ls -l bigfile.dat
-rw-r--r-- 1 tmp tmp 2163946253 2009-12-23 07:57 bigfile.dat
$ ./a.out
LONG_MAX:9223372036854775807
SSIZE_MAX:9223372036854775807
a.out: Read only 2147479552 of 2163946253 bytes: Succes
readelf -h ./a.out
ELF Header:
?Magic: ? 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
?Class: ? ? ? ? ? ? ? ? ? ? ? ? ? ? ELF64
?Data: ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?2's complement, little endian
?Version: ? ? ? ? ? ? ? ? ? ? ? ? ? 1 (current)
?OS/ABI: ? ? ? ? ? ? ? ? ? ? ? ? ? ?UNIX - System V
?ABI Version: ? ? ? ? ? ? ? ? ? ? ? 0
?Type: ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?EXEC (Executable file)
?Machine: ? ? ? ? ? ? ? ? ? ? ? ? ? Advanced Micro Devices X86-64
?Version: ? ? ? ? ? ? ? ? ? ? ? ? ? 0x1
?Entry point address: ? ? ? ? ? ? ? 0x400750
?Start of program headers: ? ? ? ? ?64 (bytes into file)
?Start of section headers: ? ? ? ? ?5312 (bytes into file)
?Flags: ? ? ? ? ? ? ? ? ? ? ? ? ? ? 0x0
?Size of this header: ? ? ? ? ? ? ? 64 (bytes)
?Size of program headers: ? ? ? ? ? 56 (bytes)
?Number of program headers: ? ? ? ? 9
?Size of section headers: ? ? ? ? ? 64 (bytes)
?Number of section headers: ? ? ? ? 37
?Section header string table index: 34
ldd ./a.out
? ? ? ?linux-vdso.so.1 => ?(0x00007fff689ff000)
? ? ? ?libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007ffee433e000)
? ? ? ?libm.so.6 => /lib/libm.so.6 (0x00007ffee40ba000)
? ? ? ?libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007ffee3ea3000)
? ? ? ?libc.so.6 => /lib/libc.so.6 (0x00007ffee3b34000)
? ? ? ?/lib64/ld-linux-x86-64.so.2 (0x00007ffee464e000)
$uname -a
Linux bronco 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC
2009 x86_64 GNU/Linux
-------------------------------------------------------------
#include <stdio.h>
#include <stdlib.h>
#include <err.h>
#include <fcntl.h>
#include <sysexits.h>
#include <unistd.h>
#include <sys/stat.h>
#include <limits.h>
#include <bits/posix1_lim.h>
#include <sys/types.h>
// get bytesize of file
size_t fsize(const char* fname){
?struct stat st ;
?stat(fname,&st);
?return st.st_size;
}
int main() {
?const char *infile = "bigfile.dat";
?int fd;
?size_t bytes_read, bytes_expected = fsize(infile);
?char *data;
?printf("LONG_MAX:%lu\n",LONG_MAX);
?printf("SSIZE_MAX:%lu\n",SSIZE_MAX);
?if ((fd = open(infile,O_RDONLY)) < 0)
? err(EX_NOINPUT, "%s", infile);
?if ((data =(char *) malloc(bytes_expected)) == NULL)
? err(EX_OSERR, "data malloc");
?bytes_read = read(fd, data, bytes_expected);
?if (bytes_read != bytes_expected)
? err(EX_DATAERR, "Read only %lu of %lu bytes",bytes_read, bytes_expected);
?/* ... operate on data ... */
?free(data);
?exit(EX_OK);
}