[cygwin] DD bug fails to wipe last 48 sectors of a disk

[Jason Pyeron]

> -----Original Message-----
> From: Brian Inglis
> Sent: Tuesday, December 29, 2020 12:55 PM
> 
> On 2020-12-28 19:41, Jason Pyeron wrote:
> > On Monday, December 28, 2020 7:46 PM, Hashim Aziz wrote:
> >> On 23 June 2020 8:33 PM, Brian Inglis wrote:
> >>> I don't have the facilities to test, and there appear to be *NO* Windows
> >>> documentation details on error condition handling, but my suspicion is
> >>> that Unix reads and writes fail only *AFTER* reading or writing at the
> >>> end of the device, but Windows reads and writes extents may be checked
> >>> and failed *BEFORE* reading or writing any data near the end of the
> >>> device.
> >>> If the actual Windows error code returned is generic, Cygwin would need
> >>> to pre-check the device size as Windows does, and reduce read and write
> >>> sizes to the allowed maximum at the end of the device.
> 
> >> That's very helpful, thank you. Do you know if any more work has been done
> >> to attempt to fix this bug, and whether it's likely to be fixed anytime
> >> soon? It's crazy that such a commonly used command leaves so much data
> >> unwiped unbeknown to so many users, it's a very serious security hole and
> >> the sooner it can be fixed the better.
> 
> > Have you tried iflag=fullblock ? This causes special handling.
> 
> >> I didn't previously see this email, but the point is that this is a bug -
> >> dd should not require first making calculations based on the size of each
> >> drive or using the smallest possible block size (and hence taking a
> >> ridiculous amount of time) in order to do what
> 
> > Do you have any metrics that it is faster, by any meaningful amount? If so I
> > would be very interested in mitigating it, but I suspect not the actual
> > case.
> 
> >> it's meant to do. It should always wipe the last sector of the drive
> >> regardless, just as it does on other UNIX-like systems. This is why this
> >> behaviour is a bug that needs to be fixed.
> 
> > This does not appear to be a bug, but user error. Per the DD source "Some
> > devices require alignment on a sector or page boundary"
> > DD has never "dealt with error handling" except when conversion were in play.
> > When no conversions are in play it
> 
> >              {
> >                /* Write any partial block. */
> >                exit_status = EXIT_FAILURE;
> >                break;
> >              }
> 
> > On windows the block devices require respecting block device boundaries, any
> > change would be an upstream patch - not a Cygwin patch.
> 
> Your dd output appears to be ambiguous, relative to your claim that the last 48
> sectors are not written, and may appear to indicate that all sectors of the
> drive may have been written, assuming that you mean 512 byte sectors.
> 
> > 1000182120448 bytes (1.0 TB, 931 GiB) copied, 8284 s, 121 MB/s
> 
> 1000182120448 == 238462*4*1024^2
> 
> > dd: error writing '/dev/sda': No space left on device
> > 238468+0 records in
> 
> 1000207286272 == 238468*4*1024^2
> 
> > 238467+0 records out
> 
> 1000203091968 == 238467*4*1024^2
> 
> > 1000204861440 bytes (1.0 TB, 932 GiB) copied, 8284.89 s, 121 MB/s
> 
> 1000204861440 == 238467*4*1024^2 + 27*64*1024
> 
> None of these numbers +/-48*512 bytes, which have odd factors, make a lot of
> sense as a disk size.
> 
> Could you please state explicitly, how many bytes/sectors/blocks/pages/clusters
> of what size you expect to get written, and how many
> bytes/sectors/blocks/pages/clusters of what size are actually written?
> 
> If anyone has access to a Linux system which has write access to a Windows drive
> over the network (e.g. Samba, NFS) where this can be reproduced, we can try to
> take this upstream, get their take, suggest an incremental reseek and write half
> buffer size patch, if they agree this is an issue and could be tackled in this
> manner.

I do, and I am even willing to spend a few dollars in testing this too.

BUT, I want a clear test plan first, with clearly articulated issues BECAUSE I do not believe there are any issues actually existing.

The best I can glean from the thread is 

1. Cygwin is agedly breaking dd, but is also broken in the windows native dd [1]
2. Using "correct" (as I have previously defined it e.g. [4]) values is both
2.1. too slow on the write [2]
2.2. inappropriately complicated of a process for the user
3. It has been well investigated 
3.1. with a root cause was identified as a windows OS issue [3,5,6]
3.2. with a mitigation [4,5]

I am happy to spend time and money on 2.1.
I will not spend my time on dealing with 2.2 - except providing an example and documentation update for upstream.

Respectfully,

Jason Pyeron

1: > From: Hashim Aziz
> Sent: Saturday, June 20, 2020 1:31 PM
> To: The Cygwin Mailing List
> Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> Message-ID: <DB7PR01MB5193CC1C7630FB13B81B9DBAD5980@DB7PR01MB5193.eurprd01.prod.exchangelabs.com>

2: > From: Hashim Aziz
> Sent: Tuesday, June 23, 2020 11:29 AM
> To: cygwin 
> Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> Message-ID: <DB7PR01MB519396195ADB3E3E1217B22BD5940@DB7PR01MB5193.eurprd01.prod.exchangelabs.com>

3: > From: Christian Franke
> Sent: Sunday, June 28, 2020 10:35 AM
> To: cygwin@cygwin.com
> Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> Message-ID: <5c9c5b44-c8a6-a075-705e-1761533cc966@t-online.de>

4: > From: Jason Pyeron
> Sent: Sunday, June 28, 2020 1:50 PM
> To: cygwin@cygwin.com
> Subject: RE: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> Message-ID: <00f601d64d74$959826d0$c0c87470$@pdinc.us>

5: > From: Brian Inglis
> Sent: Sunday, June 28, 2020 4:28 PM
> To: cygwin@cygwin.com
> Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> Message-ID: <9614fca7-4006-06f2-9956-b30dd14cc49e@SystematicSw.ab.ca>

6: > From: Jason Pyeron
> Sent: Monday, December 28, 2020 9:42 PM
> To: cygwin@cygwin.com
> Subject: RE: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> Message-ID: <121001d6dd8c$1dc8b0e0$595a12a0$@pdinc.us>