Jon Turney [Tue, 7 Jul 2020 19:54:27 +0000 (20:54 +0100)]
Cygwin: Use MEMORY_WORKING_SET_EX_INFORMATION in dumper
Use the (undocumented) MEMORY_WORKING_SET_EX_INFORMATION in dumper to
determine if a MEM_IMAGE region is unsharable, and hence has been
modified.
After this, we will end up dumping memory regions where:
- state is MEM_COMMIT (i.e. is not MEM_RESERVE or MEM_FREE), and
-- type is MEM_PRIVATE and protection allows reads (i.e. not a guardpage), or
-- type is MEM_IMAGE and attribute is non-sharable (i.e. it was WC, got
written to, and is now a RW copy)
Jon Turney [Sun, 5 Jul 2020 13:42:59 +0000 (14:42 +0100)]
Cygwin: Don't dump non-writable image regions
After this, we will end up dumping memory regions where:
- state is MEM_COMMIT (i.e. is not MEM_RESERVE or MEM_FREE), and
-- type is MEM_PRIVATE and protection allows reads (i.e. not a guardpage), or
-- type is MEM_IMAGE and protection allows writes
Making this decision based on the current protection isn't 100% correct,
because it may have been changed using VirtualProtect(). But we don't
know how to determine if a region is shareable.
(As a practical matter, anything which gets us the stack (MEM_PRIVATE)
and .data/.bss (RW MEM_IMAGE) is going to be enough for 99% of cases)
Ken Brown [Mon, 20 Jul 2020 12:59:09 +0000 (08:59 -0400)]
Cygwin: mmap: fix mapping beyond EOF on 64 bit
Commit 605bdcd410384dda6db66b9b8cd19e863702e1bb enabled mapping beyond
EOF in 64 bit environments. But the variable 'orig_len' did not get
rounded up to a multiple of 64K. This rounding was done on 32 bit
only. Fix this by rounding up orig_len on 64 bit, in the same place
where 'len' is rounded up.
Rounding up is needed to make sigbus_page_len a multiple of the
allocation granularity.
In addition, failing to round up could cause orig_len to be smaller
than len. Since these are both unsigned values, the statement
'orig_len -= len' could then cause orig_len to be huge, and mmap would
fail with errno EFBIG.
I observed this failure while debugging the problem reported in
Cygwin: pty: Fix a bug on redirecting something to /dev/pty*.
- After commit 0365031ce1347600d854a23f30f1355745a1765c, key input
becomes not working by following steps.
1) Start cmd.exe in mintty.
2) Open another mintty.
3) Execute "echo AAA > /dev/pty*" (pty* is the pty opened in 1.)
This patch fixes the issue.
Ken Brown [Mon, 13 Jul 2020 11:01:57 +0000 (07:01 -0400)]
Cygwin: FIFO: allow take_ownership to be interrupted
Use cygwait in take_ownership to allow interruption while waiting to
become owner. Return the cygwait return value or a suitable value to
indicate an error.
raw_read now checks the return value and acts accordingly.
Ken Brown [Sat, 11 Jul 2020 19:20:45 +0000 (15:20 -0400)]
Cygwin: FIFO: make certain errors non-fatal
If update_my_handlers fails to duplicate one or more handles, just
mark the corresponding handlers as being in an error state.
But if update_my_handlers is unable to open the process of the
previous owner, it's likely that something serious has gone wrong, so
we continue to make that a fatal error.
Ken Brown [Sat, 11 Jul 2020 18:52:55 +0000 (14:52 -0400)]
Cygwin: FIFO: improve taking ownership in fifo_reader_thread
When a reader takes ownership in fifo_reader_thread, it now goes
directly to the part of the main loop that listens for a connection.
Previously it went back to the beginning of the loop.
Also, if the reader has to delay taking ownership because the previous
owner has not finished updating the shared fifo_client handlers, it
now checks to see if cancel_evt has been set. Previously it might
have had to spin its wheels unnecessarily only to eventually find that
its thread had been canceled.
Ken Brown [Sat, 11 Jul 2020 18:34:24 +0000 (14:34 -0400)]
Cygwin: FIFO: reduce I/O interleaving
Add a bool member 'last_read' to the fifo_client_handler structure,
which is set to true on a successful read. This is used by raw_read
as follows.
When raw_read is called, it first locates the writer (if any) for
which last_read is true. raw_read tries to read from that writer and
returns if there is input available. Otherwise, it proceeds to poll
all the writers, as before.
The effect of this is that if a writer writes some data that is only
partially read, the next attempt to read will continue to read from
the same writer. This should reduce the interleaving of output from
different writers.
Ken Brown [Sat, 11 Jul 2020 18:05:23 +0000 (14:05 -0400)]
Cygwin: FIFO: keep a writer count in shared memory
When a reader opens, it needs to block if there are no writers open
(unless is is opened with O_NONBLOCK). This is easy for the first
reader to test, since it can just wait for a writer to signal that it
is open (via the write_ready event). But when a second reader wants
to open, all writers might have closed.
To check this, use a new '_nwriters' member of struct fifo_shmem_t,
which keeps track of the number of open writers. This should be more
reliable than the previous method.
Add nwriters_lock to control access to shmem->_nwriters, and remove
reader_opening_lock, which is no longer needed.
Previously only readers had access to the shared memory, but now
writers access it too so that they can increment _nwriters during
open/dup/fork/exec and decrement it during close.
Add an optional 'only_open' argument to create_shmem for use by
writers, which only open the shared memory rather than first trying to
create it. Since writers don't need to access the shared memory until
they have successfully connected to a pipe instance, they can safely
assume that a reader has already created the shared memory.
For debugging purposes, change create_shmem to return 1 instead of 0
when a reader successfully opens the shared memory after finding that
it had already been created.
Remove check_write_ready_evt, write_ready_ok_evt, and
check_write_ready(), which are no longer needed.
When opening a writer and looping to try to get a connection, recheck
read_ready at the top of the loop since the number of readers might
have changed.
To slightly speed up the process of opening the first reader, take
ownership immediately rather than waiting for the fifo_reader_thread
to handle it.
Ken Brown [Wed, 15 Jul 2020 13:46:42 +0000 (09:46 -0400)]
Cygwin: FIFO: fix problems finding new owner
When the owning reader closes and there are still readers open, the
owner needs to wait for a new owner to be found before closing its
fifo_client handlers. This involves a loop in which dec_nreaders is
called at the beginning and inc_nreaders is called at the end. Any
other reader that tries to access shmem->_nreaders during this loop
will therefore get an inaccurate answer.
Fix this by adding an nreaders method and using it instead of
dec_nreaders and inc_nreaders. Also add nreaders_lock to control
access to the shmem->_nreaders.
Make various other changes to improve the reliability of finding a new
owner.
If __HAVE_LOCALE_INFO__ is not defined, then the locale in the
locale-specific ctype functions is ignored. In the previous
implementation this resulted in compiler warnings. For example:
gcc -Wall main.c
main.c: In function 'main':
main.c:6:11: warning: variable 'locale' set but not used [-Wunused-but-set-variable]
6 | locale_t locale;
| ^~~~~~
Marc Hoersken [Wed, 15 Jul 2020 18:53:21 +0000 (20:53 +0200)]
Cygwin: make sure failed sockets always signal writability
Since FD_CONNECT is only given once, we manually need to set
FD_WRITE for connection failed sockets to have consistent
behaviour in programs calling poll/select multiple times.
Example test to non-listening port: curl -v 127.0.0.1:47
Brian Inglis [Mon, 13 Jul 2020 12:58:56 +0000 (06:58 -0600)]
Cygwin: FAQ 1.5: Clarify "What version is this"
Patch to:
https://sourceware.org/git/?p=newlib-cygwin.git;f=winsup/doc/faq-what.xml;a=blob
as a result of thread:
https://cygwin.com/pipermail/cygwin/2020-July/245442.html
and comments:
https://cygwin.com/pipermail/cygwin-patches/2020q3/010331.html
Relate Cygwin DLL to Unix kernel,
add required options to command examples,
differentiate Unix and Cygwin commands;
mention that the cygwin package contains the DLL,
replace setup.exe reference by Cygwin Setup program wording.
testsuite: Fix iconv tests to use new encoding config defines
_ICONV_CONVERTER -> ICONV_FROM_ENCODING. It's not perfect, as the
library can support different from/to encodings now, but at least in
the default configurations the tests now work.
libc/iconv: find_alias was mis-computing remaining alias table length
This caused the strnstr to walk off the end of the alias array and
fetch invalid data. Instead of attempting to update 'len', just
re-compute it based on the table end pointer that is already known.
Cygwin: convert sys_wcstombs/sys_mbstowcs wrapper to inline functions
This should slightly speed up especially path conversions,
given there's one less function call rearranging all function
arguments in registers/stack (and less stack pressure).
For clarity, rename overloaded sys_wcstombs to _sys_wcstombs
and sys_cp_mbstowcs to _sys_mbstowcs.
Re: CPU microcode reported wrong in /proc/cpuinfo
https://sourceware.org/pipermail/cygwin/2020-May/245063.html
earlier Windows releases used different registry values to store microcode
revisions depending on the MSR name being used to get microcode revisions:
add these alternative registry values to the cpuinfo registry value lookup;
iterate thru the registry data until a valid microcode revision is found;
some revision values are in the high bits, so if the low bits are all clear,
shift the revision value down into the low bits
sys_mbstowcs is called with the destination buffer length
set to MaximumLength from the receiving UNICODE_STRING buffer.
This is twice as much as the actual size of the buffer in
wchar_t units, which is the unit expected by sys_mbstowcs.
sys_mbstowcs always attaches a NUL, within the destination
buffersize given. But if the string is exactly one wchar_t
less than the actual buffer, and the buffersize is given too
large, sys_mbstowcs writes a NUL one wchar_t beyond the buffer.
This has only been exposed with Cygwin 3.1.5 because alloca
on newer gcc 9 apparently allocates more tightly. The alloca
buffer here is requested with 16 bytes, which is exactly the
number of bytes required for the string L"cmd.exe". Older gcc
apparently allocated a few more bytes on the stack, while gcc 9
allocates in 16 byte granularity...
Fix this by giving the correct destination buffer size to
sys_mbstowcs.
Cygwin: tcp: Support TCP_KEEPIDLE, TCP_KEEPCNT, TCP_KEEPINTVL
Use WSAIoctl(SIO_KEEPALIVE_VALS) on older systems.
Make sure that keep-alive timeout is equivalent to
TCP_KEEPIDLE + TCP_KEEPCNT * TCP_KEEPINTVL on older systems,
even with TCP_KEEPCNT being a fixed value on those systems.
Corinna Vinschen [Tue, 30 Jun 2020 11:33:57 +0000 (13:33 +0200)]
Cygwin: tcp: fix IPPROTO_TCP option handling
- Drop definitions from <cygwin/sockets.h>
- Drop options only available on BSD
- Fix value of TCP_MAXSEG. It was still defined as the BSD value
while WinSock uses another value
- Handle the fact that TCP_MAXSEG is a R/O value in WinSock
Cygwin: pty, termios: Unify thoughts of read ahead beffer handling.
- Return value of eat_readahead() is redefined. The return values
of fhandler_termios::eat_readahead() and fhandler_pty_slave::
eat_readahead() were little bit different. This patch unifies
them to number of bytes eaten by eat_readahead().
- Considerration for raixget() is added to fhandler_pty_master::
accept_input() code.
- Transfering contents of read ahead buffer in
fhandler_pty_master::transfer_input_to_pcon() is removed since
it is not necessary.
- fhandler_pty_slave::eat_readahead() ckecks EOL only when ICANON
is set.
- Guard for _POSIX_VDISABLE is added in checking EOL.
Cygwin: pty: Discard CSI > Pm m sequence from native windows apps.
- If vim is started from WSL (Ubuntu) which is executed in pseudo
console in mintty, shift key and ctrl key do not work. Though
this issue is similar to the issue resolved by commit 4527541ec66af8d82bb9dba5d25afdf489d71271, that commit is not
effective for this issue. This patch fixes the issue by discarding
"CSI > Pm m" in fhandler_pty_master::pty_master_fwd_thread().
Cygwin: pty: Fix screen distortion after less for native apps again.
- Commit c4b060e3fe3bed05b3a69ccbcc20993ad85e163d seems to be not
enough. Moreover, it does not work as expected at all in Win10
1809. This patch essentially reverts that commit and add another
fix. After all, the cause of the problem was a race issue in
switch_to_pcon_out flag. That is, this flag is set when native
app starts, however, it is delayed by wait_pcon_fwd(). Since the
flag is not set yet when less starts, the data which should go
into the output_handle accidentally goes into output_handle_cyg.
This patch fixes the problem more essentially for the cause of
the problem than previous one.
Cygwin: pty: Fix screen distortion after using less for native apps.
- If the output of non-cygwin apps is browsed using less, screen is
ocasionally distorted after less exits. This frequently happens
if cmd.exe is executed after less. This patch fixes the issue.
Cygwin: pty: Revise the code which prevents undesired window title.
- In current pty, the window title can not be set from non-cygwin
program due to the code which prevents overwriting the window
title to "cygwin-console-helper.exe" in fhandler_pty_master::pty_
master_fwd_thread(). This patch fixes the issue.
Cygwin: console: Discard some unsupported escape sequences.
- If the cygwin vim is started from a non-cygwin process which is
executed in pseudo console, shift key and ctrl key do not work.
In this case, vim is executed under /dev/cons*. If vim outputs
escape sequence which is not supported by pseudo console, the
escape sequence is leaked into the parent pty. This causes
unexpected results. This patch fixes the issue by discarding
"CSI > Pm m". "OSC 10;? BEL/ST" and "OSC 11;? BEL/ST" are
discarded as well.
Cygwin: pty: Prevent garbage remained in read ahead buffer.
- After commit 29431fcb5b14d4c5ac3b3161a076eb1a208349d9, the issue
reported in https://cygwin.com/pipermail/cygwin/2020-May/245057.html
occurs. This is caused by the following mechanism. Cygwin less
called from non-cygwin git is executed under /dev/cons* rather
than /dev/pty* because parent git process only inherits pseudo
console handle. Therefore, less sets ICANON flag for /dev/cons*
rather than original /dev/pty*. When pty is switched to non-cygwin
git process, line_edit() is used in fhandler_pty_master::write()
only to set input_available_event and read ahead buffer is supposed
to be flushed in accept_input(). However, ICANON flag is not set
for /dev/pty*, so accept_input() is not called unless newline
is entered. As a result, the input data remains in the read ahead
buffer. This patch fixes the issue.
Ken Brown [Thu, 28 May 2020 17:05:32 +0000 (13:05 -0400)]
Cygwin: stat: fix st_mode of fifos again
This partially reverts commit f36262d56ac78f04de147746ce4a85c6155e4a23. That commit incorrectly
made the st_mode of a fifo reflect the Windows permissions of the disk
file underlying the fifo.
Corinna Vinschen [Mon, 25 May 2020 15:40:27 +0000 (17:40 +0200)]
Cygwin: revamp localtime.o build rule
Rename localtime.c.patched to localtime.patched.c to keep the correct
language suffix.
Create localtime.patched.c in the build dir rather than in the source
dir. Decouple the build rule for creating localtime.patched.c from
the rule to build localtime.o, so we don't have to rebuild the patched
source file all the time.
Mark Geisert [Fri, 22 May 2020 09:32:53 +0000 (02:32 -0700)]
Cygwin: tzcode resync: details
Add tz_posixrules.h with data generated from most recent Cygwin tzdata
package. Establish localtime.cc as primarily a wrapper around a patched
copy of localtime.c. See README for more information.
Mark Geisert [Fri, 22 May 2020 09:32:51 +0000 (02:32 -0700)]
Cygwin: tzcode resync: basics
Modifies winsup/cygwin/Makefile.in to build localtime.o from items in
new winsup/cygwin/tzcode subdirectory. Compiler option "-fpermissive"
is used to accept warnings about missing casts on the return values of
malloc() calls. This patch also removes existing localtime.cc and
tz_posixrules.h from winsup/cygwin as they are superseded by the
subsequent patches in this set.
Cygwin: pty: Stop counting number of slaves attached to pseudo console.
- The number of slaves attached to pseudo console is used only for
triggering redraw screen. Counting was not only needless, but also
did not work as expected. This patch removes the code for counting.
Cygwin: pty: Revise code to make system_printf() work after close.
- After commit 0365031ce1347600d854a23f30f1355745a1765c, the issue
https://cygwin.com/pipermail/cygwin-patches/2020q2/010259.html
occurs. This patch fixes the issue.
There is no need to explicitly take ownership in fixup_after_exec; if
ownership transfer is needed, it will be taken care of by
fhandler_fifo::close when the parent closes. Moreover, closing the
parent's fifo_reader_thread can cause problems, such as the one
reported here:
Cygwin: pty: Call FreeConsole() only if attached to current pty.
- After commit 071b8e0cbd4be33449c12bb0d58f514ed8ef893c, the problem
reported in https://cygwin.com/pipermail/cygwin/2020-May/244873.html
occurs. This is due to freeing console device accidentally rather
than pseudo console. This patch makes sure to call FreeConsole()
only if the process is attached to the pseudo console of the current
pty.
Szabolcs Nagy [Tue, 19 May 2020 09:52:49 +0000 (10:52 +0100)]
Reimplement aligned_alloc
The original implementation had multiple issues:
- Only worked when posix_memalign was available (Linux, RTEMS).
- Violated C11 link namespace rules by calling posix_memalign.
- Failed to set errno on error.
These can be fixed by essentially using the same implementation
for aligned_alloc as for memalign, i.e. simply calling _memalign_r
(which is always available and a "more reserved name" although
technically still not in the reserved link namespace, at least
code written in c cannot define a colliding symbol, newlib has
plenty such namespace issues so this is fine).
It is not clear what the right policy is when MALLOC_PROVIDED is set,
currently that does not cover aligned_alloc so it is kept that way.
Windows Explorer shows a warning with Cygwin-created DACLs, but putting
the text of the warning into Google doesn't lead to the relevant Cygwin
docs. Let's copy the warning text into the docs in the hopes of helping
confused users. Most of the credit for the wording belongs to Yaakov
Selkowitz.
Ken Brown [Mon, 11 May 2020 13:03:37 +0000 (09:03 -0400)]
Cygwin: FIFO: improve the interruptibility of raw_read
During a blocking read, we sleep for 1 ms after each iteration through
the connected writers. Currently we do this by calling Sleep (1).
Remove this call to Sleep and instead change the timeout in the
cygwait call from 0 to 1, so that raw_read can be interrupted while
sleeping.
Ken Brown [Sat, 9 May 2020 21:25:39 +0000 (17:25 -0400)]
Cygwin: FIFO: code simplification
There are currently three functions that call NtQueryInformationFile
to determine the state of a pipe instance. Do this only once, in a
new fifo_client_handler::set_state () function, and call that when
state information is needed.
Remove the fifo_client_handler methods pipe_state and get_state, which
are no longer needed.
Make fhandler_fifo::get_fc_handler return a reference, for use in
select.cc:peek_fifo.
Make other small changes to ensure that this commit doesn't change any
decisions based on the state of a fifo_client_handler.
The tricky part is interpreting FILE_PIPE_CLOSING_STATE, which we
translate to fc_closing. Our current interpretation, which is not
changing as a result of this commit, is that the writer at the other
end of the pipe instance is viewed as still connected from the point
of view of raw_read and determining EOF.
But it is not viewed as still connected if we are deciding whether to
unblock a new reader that is trying to open.
Cygwin: fifo: fix type of fifo_reader_id_t operators
fifo_reader_id_t::operator == and != have been defined without type
accidentally. For some weird reason, only x86 gcc complains about
this problem, not x86_64 gcc.
Ken Brown [Tue, 14 Apr 2020 13:45:44 +0000 (09:45 -0400)]
Cygwin: FIFO: update commentary
The beginning of fhandler_fifo.cc contains a long comment giving an
overview of the FIFO implementation. This is now updated to describe
the support for multiple readers.
Ken Brown [Wed, 6 May 2020 15:31:39 +0000 (11:31 -0400)]
Cygwin: FIFO: support opening multiple readers
Although we can have multiple readers open because of dup/fork/exec,
the current code does not support multiple readers opening a FIFO by
explicitly calling 'open'.
The main complication in supporting this is that when a blocking
reader tries to open and there's already one open, it has to check
whether there any writers open. It can't rely on the write_ready
event, whose state hasn't changed since the first writer opened.
To fix this, add two new named events, check_write_ready_evt and
write_ready_ok_evt, and a new method, check_write_ready().
The first event signals the owner's reader thread to call
check_write_ready(), which polls the fc_handler list to check for
connected writers. If it finds none, it checks to see if there's a
writer in the process and then sets/resets write_ready appropriately.
When check_write_ready() finishes it sets write_ready_ok_evt to signal
the reader that write_ready has been updated.
The polling is done via fifo_client_handler::pipe_state(). As long as
it's calling that function anyway, check_write_ready() updates the
state of each handler.
Also add a new lock to prevent a race if two readers are trying to
open simultaneously.
Ken Brown [Sat, 25 Apr 2020 13:54:18 +0000 (09:54 -0400)]
Cygwin: FIFO: allow any reader to take ownership
Add a take_ownership method, used by raw_read and select.cc:peek_fifo.
It wakes up all fifo_reader_threads and allows the caller to become
owner. The work is done by the fifo_reader_threads.
For synchronization we introduce several new fhandler_fifo data
members and methods:
- update_needed_evt signals the current owner to stop listening for
writer connections and update its fc_handler list.
- shared_fc_handler() gets and sets the status of the fc_handler
update process.
- get_pending_owner() and set_pending_owner() get and set the reader
that is requesting ownership.
Finally, a new 'reading_lock' prevents two readers from trying to take
ownership simultaneously.
Ken Brown [Fri, 24 Apr 2020 13:05:12 +0000 (09:05 -0400)]
Cygwin: FIFO: find a new owner when closing
If the owning reader is closing, wait for another reader (if there is
one) to take ownership before closing the owner's pipe handles.
To synchronize the ownership transfer, add events owner_needed_evt and
owner_found_evt, and add methods owner_needed and owner_found to
set/reset them.
Modify the fifo_reader_thread function to wake up all non-owners when
a new owner is needed.
Make a cosmetic change in close so that fhandler_base::close is called
only if we have a write handle. This prevents strace output from
being littered with statements that the null handle is being closed.
git://sourceware.org / newlib-cygwin.git / log