Extreme slowdown due to malloc?

[Achim Gratz]

Mark Geisert writes:
> The page fault numbers are comparable to what you've shown for Cygwin
> on your system.  The long pause after zstd prints "Constructing
> partial suffix array" is because zstd is cpu-bound in qsort() for a
> long time.  No paging during that time.

Then there's probably another performance bug in qsort since that step
finishes much faster on Linux too.

> Then when the statistics start being printed out, that's when the
> paging insanity starts.

OK, so you see the same thing.  Good.

> What I discovered is that zstd is repeatedly asking malloc() for large
> memory blocks, presumably to mmap files in, then free()ing them.

Oh, so it's not recycling those… maybe that's not very hard to change.

> Any malloc request 256K or larger is fulfilled by mmap() rather than
> enlarging the heap for it.  But crucially, there is no mechanism for
> our malloc to hang on to freed mmap()ed pages for future use.  If you
> free an mmap()ed block, it is unmap()ed immediately.  So for zstd's
> usage pattern you get an incredible number of page faults to satisfy
> the mmap()s and Windows seems to take a non-trivial bit of time for
> each mmap().

It probably puts backing store behind it even when there's enough
memory.  Come to think of it I should try to move the page file to the
NVMe drive (off the SSD it's on currently) and see if that changes
things.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds