Cygwin malloc tune-up status

[Teemu Nätkinniemi]

ti 29. syysk. 2020 klo 5.23 Mark Geisert (mark@maxrnd.com) kirjoitti:
>
> Johannes Schindelin wrote:
> > Hi Mark,
> > On Thu, 24 Sep 2020, Mark Geisert wrote:
> >> I've been looking into two potential enhancements of Cygwin's malloc operation
> >> for Cygwin 3.2.  The first is to replace the existing function-level locking
> >> with something more fine-grained to help threaded processes; the second is to
> >> implement thread-specific memory pools with the aim of lessening lock activity
> >> even further.
> >>
> >> Although I've investigated several alternative malloc packages, including
> >> ptmalloc[23], nedalloc, and Windows Heaps, only the latter seems to improve on
> >> the performance of Cygwin's malloc.  Unfortunately using Windows Heaps would
> >> require fiddling with undocumented heap structures to enable use with fork().
> >> I also looked at BSD's jemalloc and Google's tcmalloc.  Those two require much
> >> more work to port to Cygwin so I've back-burnered them for the time being.
> >
> > I am just a lurker when it comes to your project, but I wonder whether you
> > had any chance to look into mimalloc
> > (https://github.com/microsoft/mimalloc)? I had investigated it in Git for
> > Windows' context for a while (because nedmalloc, which is used by Git for
> > Windows, is no longer actively maintained).
>
> Hi Johannes,
> Great minds think alike!  Yours is the 3rd pointer I've received on- and off-list
> towards mimalloc.  I had not heard of it before.  I've looked into it and have now
> added it to my back-burnered list.
>
> mimalloc looks promising.  It's fairly small.  It has at least one issue it shares
> with jemalloc and tcmalloc: initialization code that needs to run before the
> Cygwin DLL has completely set up a new process' environment.  A chicken-and-egg
> problem, if you will.  A solution to that (which I'm pondering) will allow me to
> test all three malloc alternatives in the future.
>

Hi!

I encounter a problem with Cygwin's malloc and remembered this thread.
Sorry if this is off-topic.

I have been trying to port bwa aligner to Cygwin. Initially everything
seemed to work but for some reason in some cases threading didn't seem
to work properly. I got a fix recently from a third party which was to
force bwa to use rpmalloc.

This got me thinking if there is a problem with Cygwin's malloc in
some cases and if there were people in this list who might be
interested in knowing that the problem exists.

Here's a link to the rpmalloc fix.

https://github.com/WGSExtract/bwa/commit/3087fa876b079fcb6a0a58f1e01757f4820094a8

Here's a test case:

bwa_original mem -t 10 bwa_reference/hs37d5.fa ERS4238880_1.fastq > test.sam

bwa_working mem -t 10 bwa_reference/hs37d5.fa ERS4238880_1.fastq > test.sam

Files:

https://drive.google.com/drive/folders/1waICih51f4mHZEyWY1onyEcKqm0kj3Yt?usp=sharing

Thanks,
Teemu