| Summary: | Controlling gmon.out file creation in HPC environment | ||
|---|---|---|---|
| Product: | glibc | Reporter: | Pidad <pidsouza> |
| Component: | libc | Assignee: | Not yet assigned to anyone <unassigned> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | neleai |
| Priority: | P2 | Flags: | fweimer:
security-
|
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Host: | Target: | ||
| Build: | Last reconfirmed: | ||
There is already a functionality to support this. You need to interpose a _mcleanup by compiling shared library containing function like one below and then LD_PRELOAD it for your program.
void _mcleanup (void)
{
if (user condition)
{
void (*p)() = dlsym (RTLD_NEXT, "_mcleanup");
p ();
}
}
|
In the High Performance Computing (HPC) the parallel(Message Passing Interface - MPI) applications run as large number of tasks (100s, 1000s to 1 million tasks). Each of these tasks may be run on different nodes of a HPC cluster. Each task is a process by itself, which are executed in parallel using the same binary (SPMD or MPMD models). When these MPI applications are profiled using -pg option, depending on the number of tasks those many separate gmon.out files will be created in the same shared file system. This leads to bottleneck on to the application both from file system space, large amount disk operation during process exit. HPC users might be interested in limited set of gmon.out files which meet certain defined criteria and not all the gmon.out files created. However the set is not known until after the application has completed execution and the gmon.out files are about to be written. In the view to address the above said bottleneck and give the user only the gmon.out files of interest, following approach can be thought about: 1. Disable the generation of gmon.out file at exit(). 2. Obtain access to all gmon data in memory prior to application exit using public interfaces so that data can be processed for the subset of tasks that are of interest. The above approach was tried on Redhat 6.1 (Santiago) 2.6.32-131.0.15.el6.ppc64. With moncontrol(0) could not stop the creation of gmon.out file during exit. It only switches off the profiling. The atexit handler gmon.c:_mcleanup() creates the gmon.out. There is no way to stop it creating. Even if one succeeds to stop creation of gmon.out file, the data variable which holds the profile data is declared as hidden. Ref - gmon.c:struct gmonparam _gmonparam attribute_hidden = { GMON_PROF_OFF }; Due to this, the applications will not be able to read the collected in-memory data from the tasks to apply the filtering criteria. We need mechanisms to stop the creation of gmon.out file and access to read the collected gmon data using which gmon.out file is generated at will.