htdig and sources.redhat.com loadavg

[Hans-Peter Nilsson]

On Mon, 5 Apr 2004, Phil Edwards wrote:
> On Mon, Apr 05, 2004 at 03:36:34PM -0400, Hans-Peter Nilsson wrote:
> > looked and there was; a DB file is over the 2G limit once again,
> > and the last two updates (done every other day for gcc htdig)
> > have failed.
>
> Can we stick
>
>   if test -n `find /wherever/htdig/lives -type -f -size +2g`; then
>      echo 'Ah crap, out of space again.'
>      kill htdigpid
>   fi
>
> in a crontab and be done with this once and for all?

No.  The limit isn't related to "out of diskspace"; there's
enough of that.  The limit is of indexing; 2G for files indexed
by a signed integer and no, it doesn't help using the cute
-D_FILE_OFFSET_BITS=64 (sp?).  Feel free to fix it if you care,
but please see older posts this list before further suggestions.

It's not like a local search engine is as needed now as it was
when htdig was first installed, and htdig-3.1.5 just can't cope
with the current amount of data.  A later version (3.2 series)
has been tried, but was prohibitively slow at indexing.  Maybe
mnogosearch is an option.  Sorry, I'm just out of time and
interest.

brgds, H-P