Re: Sparse files?

[Corinna Vinschen <corinna-cygwin at cygwin dot com>]

On May 13 10:50, Bill C. Riemers wrote:
> OK.  Then there must be a problem in the way cygwin creates sparse files,
> since the files created by seeking past the end in cygwin always take up the
> full amount of disk space.  I've verified this with both Windows and cygwin
> tools.  I'll try taking it up on the cygwin list to see what I can learn.  I
> would like to understand why dd works perfectly for creating sparse files
> under unix, but not under cygwin.

Dunno how dd actually works but I reckon it sets the file size first
(ftruncate() or so) and then seeks to the position it should write to.

Sparseness is a transparent builtin feature of ext2 or ext3 drivers while
sparseness on Windows is, as usual, non-transparent but instead has to be
controlled by the application.  The technique used by Cygwin is rather
simple.  If the file size is X bytes and the application seeks to a file
position X + 128K, *then* a file is made sparse by Cygwin(*). 

If you want to see sparse files created by Cygwin, just try the below
sparse-test.c file, which is what I hacked while experimenting with
that feature and trying to find bugs.


Corinna

(*) By calling DeviceIoControl (h, FSCTL_SET_SPARSE, ...).  The value
    of 128K is what I found as smallest useful value, evaluated on XP.

======= SNIP ======
#include <stdio.h>
#include <sys/types.h>
#include <sys/fcntl.h>
#include <sys/mman.h>
#include <errno.h>
#include <unistd.h>
#include <windows.h>
#include <sys/cygwin.h>

int
main(int argc, char **argv)
{
  off_t seek = 184;
  int buf[4096];
  char *map;
  off_t len;
  DWORD hlen;

  if (argc > 1 && atol (argv[1]) > 0)
    seek = atol (argv[1]);
  printf ("Creating file of size %luK\n", seek + 8);

  int fd = open ("sparse.test", O_WRONLY | O_CREAT | O_TRUNC, 0644);
  if (fd < 0)
    {
      printf ("open failed: %d \"%s\"\n", errno, strerror(errno));
      return 1;
    }
  // Write first block
  memset (buf, 1, 4096);
  write (fd, buf, 4096);

  // Seek 184K
  lseek (fd, seek * 1024, SEEK_CUR);

  // Write second block
  memset (buf, 2, 4096);
  write (fd, buf, 4096);
  
  // Print size values
  len = GetFileSize ((HANDLE)get_osfhandle (fd), &hlen);
  len += ((off_t) hlen << 32);
  printf ("Size:         %10lldK\n", len >> 10);
  len = GetCompressedFileSize ("sparse.test", &hlen);
  len += ((off_t) hlen << 32);
  printf ("Size on disk: %10lldK\n", len >> 10);

  close (fd);

  // Reopen 
  fd = open ("sparse.test", O_RDWR);
  if (fd < 0)
    {
      printf ("open 2 failed: %d \"%s\"\n", errno, strerror(errno));
      return 1;
    }

  map = mmap (NULL, (seek + 8) * 1024, PROT_READ | PROT_WRITE,
	      MAP_SHARED, fd, 0);
  if (map == MAP_FAILED)
    {
      /* Write a byte somwhere in the middle */
      if (lseek (fd, seek * 512, SEEK_SET) == (off_t) -1)
        {
	  printf ("seek to %lld failed: %d \"%s\"\n", seek >> 1, errno, strerror(errno));
	  return 1;
	}
      if (write(fd, "\003", 1) <= 0)
        {
	  printf ("write 2 failed: %d \"%s\"\n", errno, strerror(errno));
	  return 1;
	}
    }
  else
    {
      off_t i;

      // Check contents
      for (i = 0; i < 4096; ++i)
        if (map[i] != 1)
	  printf ("first page doesn't contain 1 in byte %lu\n", i);
      for (i = 4096; i < (seek + 8) * 1024 - 4096; ++i)
        if (map[i] != 0)
	  printf ("sparse pages don't contain 0 in byte %lu\n", i);
      for (i = (seek + 8) * 1024 - 4096; i < (seek + 8) * 1024; ++i)
        if (map[i] != 2)
	  printf ("last page doesn't contain 3 in byte %lu\n", i);

      // Write a 3 in the middle
      map[((seek + 8) * 1024) / 2] = 3;

      munmap (map, (seek + 8) * 1024);

    }
  // Print new size values
  len = GetFileSize ((HANDLE)get_osfhandle (fd), &hlen);
  len += ((off_t) hlen << 32);
  printf ("Size:         %10lldK\n", len >> 10);
  len = GetCompressedFileSize ("sparse.test", &hlen);
  len += ((off_t) hlen << 32);
  printf ("Size on disk: %10lldK\n", len >> 10);
  close (fd);
  return 0;
}
======= SNAP ======
-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Co-Project Leader          mailto:cygwin@cygwin.com
Red Hat, Inc.

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/