Serialization of AIO requests to the same fd in posix-aio

[Anthony Liguori]

Hi Ulrich,

In sysdeps/pthread/aio_misc.c:__aio_enqueue_request(), there is a check 
to see if an existing thread is working on the same file descriptor as 
the request being enqueued.  If it is, it queues it to run after the 
existing request instead of possibly spawning a new thread.

The net effect is that submitting multiple requests to the same file 
descriptor are always serialized.  This significantly affects performance.

The comment beneath this check says:

      /* The current file descriptor is worked on.  It makes no sense
     to start another thread since this new thread would fight
     with the running thread for the resources.  But we also cannot
     say that the thread processing this desriptor shall immediately
     after finishing the current job process this request if there
     are other threads in the running queue which have a higher
     priority.  */

I don't understand what "fight with the running thread" is supposed to 
suggest.  Is this suggesting that there is some fundamental race 
condition?  Is it assuming that two IO requests cannot happen at the 
same time at the kernel level?  Since the threads end up using 
pread/pwrite, it should be safe to have two request executing 
simultaneously.

Currently, we use posix-aio for disk IO in QEMU/KVM.  To achieve good 
performance, we have to implement a fd pool using dup() to work around 
this check.  The difference between using the pool and not using the 
pool is around 400% in write performance.

Can you further elaborate on what this check is trying to prevent?

Thanks,

Anthony Liguori