Exhausted resources

Systemtap attempts to assure that probe handlers run within strict time/space limits. A typical script producing a modest amount of trace data will not encounter these limits, but maybe your script does. What can you do?

A blunt workaround is to use the --suppress-handler-errors flag. It lets most errors be quietly absorbed by the systemtap runtime, so the script will soldier on and on, until it runs exit(), or a user interrupts it. It'll print a single warning message then.

It depends on what resource ran out.

error message

exhausted resource

possible remedy

MAXACTION exceeded

time -- number of statements executed by probe handler

recompile with larger -DMAXACTION=NN or -DMAXACTION_INTERRUPTIBLE=NN

impose limits on iteration - foreach (... limit NN) or break)

MAXNESTING exceeded

time/space -- function recursion too deep

reduce recursion

recompile with larger -DMAXNESTING=NN

division by zero

time/space -- universe is too small to represent exact value

try dividing by one instead

aggregation overflow

space -- number of distinct index tuples in aggregation array

recompile with larger -DMAXMAPENTRIES=NN

declare that array only with larger size -- global my_array[NN]

delete unneeded elements, or use % array global declaration suffix

Array overflow

space -- number of distinct index tuples in ordinary array

as above

string truncation

space -- strings are silently limited to a maximum length

recompile with larger -DMAXSTRINGLEN=NN

WARNING: ... skipped probes: NN

time -- cross-processor contention over global script variables

reduce number of globals

maybe recompile with larger -DMAXTRYLOCK=NN

use aggregates and @count() instead of integer counters

various tolerable transient conditions

retry with larger -DMAXSKIPPED=NN

space -- too many return probes may be pending

add larger .maxaction(NN) to .return probe point

space -- too little kernel stack available for probe handler run

risky, but maybe recompile with smaller -DMINSTACKSPACE=NN

probe overhead exceeded threshold

time -- probe handlers are taking too high fraction of total real time

shrink probe handler code

recompile with larger -DSTP_OVERLOAD_THRESHOLD

disable this heuristic with -g --suppress-time-limits

There were NN transport failures

space -- amount of data printed by script exceeded staprun's capability to copy it to the output file

print less stuff

run with larger trace buffers: stap -s NN

out-of-memory during or after startup

space -- large arrays?

recompile with smaller -DMAXSTRINGLEN, if string indexes

recompile with smaller -DMAXMAPENTRIES or global foo[SIZE]

join index columns into single string with sprintf()

use smaller kernel boot maxcpus=NNN if kernel reserves excess hotplug slots

use scalar (++) rather than aggregate (<<<1; @count) arrays

See also TipSkippedProbesOptimization.


Tips

None: TipExhaustedResourceErrors (last edited 2015-05-19 18:51:49 by FChE)