This is the mail archive of the libc-help@sourceware.org mailing list for the glibc project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: Help: shutdown(..., SHUT_WR) on TCP sockets


Florian Weimer <fweimer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> writes:

> * Sergey Organov:
>
>> Florian Weimer <fweimer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> writes:
>>
>>> * Sergey Organov:
>>>
>>>> Maybe, but honestly, I fail to see how adding 'shutdown(fd, SHUT_WR)'
>>>> anywhere before 'close(fd)' can do things worse from the POV of data
>>>> delivery to the other end.
>>>>
>>>> What I observe is that either:
>>>>
>>>> sleep(1);
>>>> close(fd);
>>>> exit(0);
>>>>
>>>> or:
>>>>
>>>> sleep(1);
>>>> shutdown(fd, SHUT_WR);
>>>> close(fd);
>>>> exit(0);
>>>>
>>>> deliver all the data, while:
>>>>
>>>> shutdown(fd, SHUT_WR);
>>>> sleep(1);
>>>> close(fd);
>>>> exit(0);
>>>>
>>>> cuts some of the data (read() returns 0 on the other end indicating
>>>> closed socket).
>>>
>>> Wait, you get less data with the last sequence?  And you do not write to
>>> fd after the shutdown call?
>>
>> Yes, and this is single-threaded application so I'm very sure I don't
>> write anything myself.
>>
>> However, as I already mentioned, ioctl(fd, TIOCOUTQ, &v) gives 0 before
>> shutdown() and 1 right  after shutdown(), that suggests shutdown()
>> itself writes something. Dunno if it's expected.
>
> It needs to queue the FIN segment, for the connection teardown
> handshake.
>
>>>> Another mystery is that 'ioctl(fd, TIOCOUTQ, &v)' gives 0 in all the
>>>> above cases when put at the beginning of the above sequences[*],
>>>> indicating that there are no pended data, so there should be nothing to
>>>> loose in the first place, one way or another.
>>>
>>> Yes, I agree that data loss should not happen in these cases.
>>
>> It looks like shutdown() does something that is transferred to receiver
>> and makes it drop some amount of recently received data that receiving
>> application didn't read yet. I.e., it's probably receiving end that
>> drops the data?
>
> Do you see any RST segments in packet captures?  As I said, RST is not
> expected to be ordered with respect to data delivery, so if an RST
> segment is generated, then it would explain the data loss.

I've got time to get back to the issue, and what I see is the following.

When shutdown(fd, SHUT_WR) is called without delay on the sending end, the sending
terminates early like this:

[...]
08:14:25.741054 IP (tos 0x0, ttl 64, id 23482, offset 0, flags [DF], proto TCP (6), length 53)
    omega-51da5d.gnss.ru.8002 > osv.43076: Flags [P.], cksum 0x5e17 (correct), seq 540:541, ack 16504, win 1215, options [nop,nop,TS val 188980857 ecr 1249457135], length 1
08:14:25.741331 IP (tos 0x0, ttl 64, id 23483, offset 0, flags [DF], proto TCP (6), length 53)
    omega-51da5d.gnss.ru.8002 > osv.43076: Flags [P.], cksum 0x5e16 (correct), seq 541:542, ack 16504, win 1215, options [nop,nop,TS val 188980857 ecr 1249457135], length 1
08:14:25.741728 IP (tos 0x0, ttl 64, id 23484, offset 0, flags [DF], proto TCP (6), length 53)
    omega-51da5d.gnss.ru.8002 > osv.43076: Flags [P.], cksum 0x5e15 (correct), seq 542:543, ack 16504, win 1215, options [nop,nop,TS val 188980857 ecr 1249457135], length 1
08:14:25.743013 IP (tos 0x0, ttl 64, id 23485, offset 0, flags [DF], proto TCP (6), length 52)
    omega-51da5d.gnss.ru.8002 > osv.43076: Flags [F.], cksum 0x641b (correct), seq 543, ack 16504, win 1215, options [nop,nop,TS val 188980858 ecr 1249457135], length 0
08:14:25.743644 IP (tos 0x0, ttl 64, id 23486, offset 0, flags [DF], proto TCP (6), length 52)
    omega-51da5d.gnss.ru.8002 > osv.43076: Flags [.], cksum 0x6414 (correct), seq 544, ack 16511, win 1215, options [nop,nop,TS val 188980858 ecr 1249457135], length 0

whereas in the "good" case, where shutdown(fd, SHUT_WR) is called after
1-second delay, sending terminates much later, like this:

[...]
08:04:50.446074 IP (tos 0x0, ttl 64, id 34857, offset 0, flags [DF], proto TCP (6), length 53)
    omega-51da5d.gnss.ru.8002 > osv.43048: Flags [P.], cksum 0xc795 (correct), seq 697:698, ack 16503, win 1385, options [nop,nop,TS val 188923385 ecr 1249313311], length 1
08:04:50.446339 IP (tos 0x0, ttl 64, id 34858, offset 0, flags [DF], proto TCP (6), length 53)
    omega-51da5d.gnss.ru.8002 > osv.43048: Flags [P.], cksum 0xc794 (correct), seq 698:699, ack 16503, win 1385, options [nop,nop,TS val 188923385 ecr 1249313311], length 1
08:04:50.446738 IP (tos 0x0, ttl 64, id 34859, offset 0, flags [DF], proto TCP (6), length 53)
    omega-51da5d.gnss.ru.8002 > osv.43048: Flags [P.], cksum 0xc793 (correct), seq 699:700, ack 16503, win 1385, options [nop,nop,TS val 188923385 ecr 1249313311], length 1
08:04:50.459717 IP (tos 0x0, ttl 64, id 34860, offset 0, flags [DF], proto TCP (6), length 64)
    omega-51da5d.gnss.ru.8002 > osv.43048: Flags [.], cksum 0x0fed (correct), ack 16510, win 1385, options [nop,nop,TS val 188923386 ecr 1249313311,nop,nop,sack 1 {16509:16510}], length 0
08:04:51.384289 IP (tos 0x0, ttl 64, id 34861, offset 0, flags [DF], proto TCP (6), length 52)
    omega-51da5d.gnss.ru.8002 > osv.43048: Flags [F.], cksum 0xcd35 (correct), seq 700, ack 16510, win 1385, options [nop,nop,TS val 188923479 ecr 1249313311], length 0

To me it looks like it's in fact the sending end of the connection that
drops a lot of yet unsent packets (packets 543--700) when shutdown() is
called.

What is also surprising is that TIOCOUTQ requests put like this:

  int v1 = -1, v2 = -1;
  ioctl(fd, TIOCOUTQ, &v1);
  shutdown(fd, SHUT_WR);
  ioctl(fd, TIOCOUTQ, &v2);
  DPRINT("$$OUT %d %d\n", v1, v2);
  sleep(1);
  close(fd);
  exit(0);

result in

$$OUT 3 2

debug output in the "bad" case that doesn't match the number of dropped
data at all.

Any thoughts?

Thanks,
-- Sergey


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]