htdig and sources.redhat.com loadavg

[Jason Molenda]

Hi all, sorry I was in meetings all afternoon - just catching
up.


On Mon, Apr 05, 2004 at 05:14:20PM -0400, David Edelsohn wrote:

> 	I am guessing that htdig pushed sourceware into a thrashing mode.
> Because CVS doesn't time out or service requests sequentially, people are
> just hanging around with long CVS operations while they do other things.
> sourceware needs workload management.


The schedule of crontabs is rather carefully considered.  The
service load on sourceware variest considerably by hour-of-the-day --
Monday mornings EST happen to be the highest load of the week.
Sunday is the lowest load.

The htdig index update jobs are scheduled so they finish before
the weekday morning load happens.  When something goes wrong - as
it did in this case - and htdig runs into the morning rush, the
system trashes for quite a long time.


Matthew Galgoci wrote;

> Ideally the searching should be offloaded to another machine that
> is dedicated to the purpose of indexing and running the search
> database. 

The reason for keeping the search engine on the main system is
that the search engine has direct access to the files; it doesn't
have to go through httpd.  NFS could be used to access the files
from a different system, but you're still introducing a slowdown
by not having local access.

I'm not trying to preclude such a change, I'm just pointing out
the thinking behind the current arrangement.

(well, and the fact that we only had one computer allocated
for the original sourceware system.)



No one has mentioned my favorite possibility:  Not archiving older
e-mail notes.  Or having multiple search archives, divided by time
period.  e.g. epoch - 2001.  2001 - 2003.  2004 - ...

I can't remember if there's a good reason to not do this.  It seems
like a good idea to me, with the obvious caveat that this complicates
the web search engine UI.

J