Re: Slow pthread_create() under high load

["Albert D. Cahalan" <acahalan at cs dot uml dot edu>]

Stephen C. Tweedie:
> Maybe, but although a lot of the POSIX threads are reasonable,
> things like requiring uid/gid updates to be instantly effective
> across all threads in the process are just insane.  There ain't
> no way that's going into the kernel any time soon: it would
> require adding semaphore locking to all of the credential
> information that is currently task-local.

If we had a task-group notion in the kernel, we could send some
sort of signal to all tasks in the group. This gets everything
done in a task-local manner. The signal handler need not be a real
user-visible one in user address space; it could be kernel-only.

(gee, reminds me of MTRR on SMP -- Intel won't share registers)

Khimenko Victor:
> AFAIK there are two issues: clone in 2.2 can not create sibling
> (just child) and it'll lead to nasty problems since "parent"
> thread can exit before "child" thread. In late 2.3.x it IS fixed.

Parent exit is blocked? I don't think so, but this is important.
Without this you can not get the right PID behavior.

> And then you need special handling of signals: sometimes signals
> should be delivered to GROUP of threads. AFAIK there was some

Hey, I need a "task group" concept for correct ps operation anyway.

Ulrich Drepper:
> How do you think this should help?  If you make all the threads
> siblings (of the initial thread), how should terminating one of them
> effect the others?  If one threads dies of a SEGFAULT all the others
> must, too.  Being siblings, they just continue to run.  In the case of
> a normal multi-threaded application the parent will be the shell (or
> whatever program started the MT program).  This program does not go on
> a reap the children.

Well, of course there must be some way to link the tasks.
I'll describe one method below.

Ulrich Drepper:
> But then it's not worth it.  I want to get rid of the manager thread
> since it is what makes the implementation slow.  And in your list of
> requirements you haven't listed anything which could solve the problem.
> ...
> So, you know everything better because...?  The manager makes things
> horribly slow because the communication is awkward and the
> serialization it enforces on the implementation is keeping many things
> from running in parallel.

I think you need at least a placeholder thread. This would be a manager
thread that doesn't actually do anything. Without this, you can not let
threads exit as desired -- the POSIX parent (your shell) will notice.

Richard Gooch:
> Because the kernel doesn't make this easy. The kernel has the concept
> of "tasks", which don't map well to POSIX threads. The kernel doesn't
> distinguish between "processes" and "threads" like POSIX does.
> Everything is just a task. Tasks can share some things (FS, files,
> VM), but they can't (yet) share PIDs (probably never), signal queues
> and other things required for POSIX compliance.

It is easy to share PIDs. Rename the old getpid() call to gettid()
and introduce a new getpid() call. The directories in /proc should
be seen as tasks, not processes.

The new PID is:
1. set equal to TID in a new fork child
2. set equal to TID during exec
3. unchanged for non-fork clone calls

There, now you have a Linux-compatible POSIX-compatible PID
without forcing all thread packages to follow POSIX (mis-)features.

For normal threads, the initial task must exit last. It would not
be unreasonable to enforce this by making a zombie out of the
initial task; alternately the kernel could hide the reserved task
slot in some other way.

> You could probably write a native Linux threads library (lthread_*()
> anyone?) which was just as easy to use and powerful, and didn't
> require buggering the kernel. So from the point of view of the
> kernel community, there is reluctance to throw complex and ugly
> POSIX-specific support code into the kernel, for the sake of
> compliance with an ill-considered standard. Leave the problems to
> user-space, and hence the performance problems people mention.

If you have no problems with embrace-and-extend, yes.