Currently (and since commit 647eb037f from 2001), ldconfig will unlink any file it considers a stale library symlink. It will do so even if the -X option is given on the command line, and it will also do so no matter what error occurred while executing stat64 on the real file name.

One problem with this is that people might reasonably expect unlink -X -N to have no side effects except its output. This is how I found the issue: the configure script of the xapian library calls “/sbin/ldconfig -N -X -v”, and does so only to parse its output, with no modification intended.

Another problem is that the errno resulting from the stat64 call is not checked. So any error accessing that file leads to removal of the symlink. In particular, insufficient permissions are affected by this as well. So if a group of admins were allowed to write /usr/lib and someone installed a symlink there to a library which is group-readable but not world-readable, then an admin not in that group might accidentially delete said symlink. With only root accessing these directories, access should normally not be a problem, but I guess the problem might reappear with e.g. nfs mapping root to non-root.

So on the whole, I suggest that you change the unlink condition to
if (errno == ENOENT && do_remove && strstr (direntry->d_name, ".so."))
or something like this.