Re: Excessive memory usage by associative arrays

[Sergey Klyaus <myautneko at gmail dot com>]

Hello.

> Especially declare "global foo[SIZE]" if you know a static size limit for a particular array.
Thanks for that note, I reduced per-map size, that works for me, now
script consumes only ~150 Mbytes.
For the case, here it is: https://gist.github.com/myaut/d31f6a2a6f2472c860b3

> How many CPUs does your machine actually have?
My VM has only 2 vCPUs. Entire machine has only 24 hardware threads.
I've noticed that dyninst runtime actually uses number of online CPUs
(_stp_runtime_num_contexts)

Also, I find situation when OOM need to interfere when SystemTap is
working is frightening.
STP_MEMORY is not defined by default, and after I do it, stap fails
with nicer error message:
ERROR: global variable 'a' allocation failed


BR, Sergey.

2015-01-19 23:28 GMT+03:00 Josh Stone <jistone@redhat.com>:
> On 01/19/2015 12:07 PM, Frank Ch. Eigler wrote:
>> Sergey Klyaus <myautneko@gmail.com> writes:
>>
>>> [...]
>>> - Each string is stored statically and MAP_STRING_LENGTH is 256 (i
>>> have 2 keys so its 256)
>>> - Plus statistics data gives us 1088 bytes per entry
>>> - SystemTap initializes map on per-cpu basis and do it with
>>> for_each_possible_cpu() (128 in my case)
>>
>> Yes, unfortunately all those factors multiply.  You can control
>> MAXMAPENTRIES and MAXSTRINGLEN directly via -D options or other ways.
>> How many CPUs does your machine actually have?
>>
>> See also
>> https://sourceware.org/systemtap/wiki/TipExhaustedResourceErrors
>
> Especially declare "global foo[SIZE]" if you know a static size limit
> for a particular array.
>
> On stats, if you're just using @count and you don't expect this value to
> have highly-concurrent updates, then a simple "+=" counter will save
> memory by not reserving per-cpu space.
>
> Also if your probe has any other global writes that aren't stats, then
> you'll be taking an exclusive write lock anyway, so stats don't help.
> That is, a probe is only concurrent if all its globals are concurrent.
>