newscache: Thread: cache size limit

Hi,

i'm trying to set up NewsCache on a FreeBSD 6.1 Stable machine. (2800
Mhz Pentium 4 / 1024M RAM). If I try to run newscache as news/news
user and group the .db file never grows larger than 2048M (even if i
set the max cache size to 40gig+). If i run newscache as root/wheel
the same thing happens... but with a segfault, too...
Here's what I see in the log:

---
NewsCache[9219]: updating the overview database
failed(alt.binaries.blah.blah): NVContainer::make_current(): mmap
failed
kernel: pid 9219 (newscache), uid 0: exited on signal 11 (core dumped)
NewsCache[9192]: 9219 caught signal 11
---
I'm using the FreeBSD port version (Makefile states PORTVERSION=    1.2.r.6)


I tried to get a cvs snapshot, but it seems that the cvs password
listed on site doesn't work anymore.

Anyone could give me a hint, why should i do in order to get a full
update / bigger than 2048M?


PS
The file system is UFS (/dev/ad0s1a on / (ufs, local, soft-updates)

Thank You!

Please try to use the following compile options:

 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE 

Here a short testprogram:

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h> /* mmap() is defined in this header */
#include <fcntl.h>
#include <error.h>


int main (int argc, char *argv[])
{
         int fdin, fdout;
         char *src, *dst;
         struct stat statbuf;

        if ((fdin = open (argv[1], O_RDONLY)) < 0) {
                perror ("open error\n");
                return(1);
        }

        if (fstat (fdin,&statbuf) < 0) {
                printf ("fstat error\n");
                return(1);
        }

        if ((src = mmap (0, statbuf.st_size, PROT_READ, MAP_SHARED, fdin, 0)) == (caddr_t) -1) {
                perror ("mmap error for input\n");
                return(1);
        }

        close (fdin);
        return (0);
}


gcc -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE   -o test_mmap test_mmap.c

Please report the situation.

-----Original Message-----
From: Cristian Avram ####@####.####
Sent: Tue 7/25/2006 10:37 AM
To: ####@####.####
Subject: cache size limit
 
Hi,

i'm trying to set up NewsCache on a FreeBSD 6.1 Stable machine. (2800
Mhz Pentium 4 / 1024M RAM). If I try to run newscache as news/news
user and group the .db file never grows larger than 2048M (even if i
set the max cache size to 40gig+). If i run newscache as root/wheel
the same thing happens... but with a segfault, too...
Here's what I see in the log:

---
NewsCache[9219]: updating the overview database
failed(alt.binaries.blah.blah): NVContainer::make_current(): mmap
failed
kernel: pid 9219 (newscache), uid 0: exited on signal 11 (core dumped)
NewsCache[9192]: 9219 caught signal 11
---
I'm using the FreeBSD port version (Makefile states PORTVERSION=    1.2.r.6)


I tried to get a cvs snapshot, but it seems that the cvs password
listed on site doesn't work anymore.

Anyone could give me a hint, why should i do in order to get a full
update / bigger than 2048M?


PS
The file system is UFS (/dev/ad0s1a on / (ufs, local, soft-updates)

Thank You!

---------------------------------------------------------------------
To unsubscribe, e-mail: ####@####.####
For additional commands, e-mail: ####@####.####

Thanks for the quick reply. I had to comment the line with error.h in
your source in order to get a clean compile.

gcc -v
Using built-in specs.
Configured with: FreeBSD/i386 system compiler
Thread model: posix
gcc version 3.4.4 [FreeBSD] 20050518


I've tried to load a big file (8gig) this is the output:

./test_mmap bigfile.iso
mmap error for input

Can you help me, what should I do next?

Thanks!

On 7/25/06, Straub Herbert ####@####.#### wrote:
>
>
>
> Please try to use the following compile options:
>
>   -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE
>
>  Here a short testprogram:
>
>  #include <stdio.h>
>  #include <sys/types.h>
>  #include <sys/stat.h>
>  #include <sys/mman.h> /* mmap() is defined in this header */
>  #include <fcntl.h>
>  #include <error.h>
>
>
>  int main (int argc, char *argv[])
>  {
>           int fdin, fdout;
>           char *src, *dst;
>           struct stat statbuf;
>
>          if ((fdin = open (argv[1], O_RDONLY)) < 0) {
>                  perror ("open error\n");
>                  return(1);
>          }
>
>          if (fstat (fdin,&statbuf) < 0) {
>                  printf ("fstat error\n");
>                  return(1);
>          }
>
>          if ((src = mmap (0, statbuf.st_size, PROT_READ, MAP_SHARED, fdin,
> 0)) == (caddr_t) -1) {
>                  perror ("mmap error for input\n");
>                  return(1);
>          }
>
>          close (fdin);
>          return (0);
>  }
>
>
>  gcc -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE   -o test_mmap test_mmap.c
>
>  Please report the situation.
>
>
>  -----Original Message-----
>  From: Cristian Avram ####@####.####
>  Sent: Tue 7/25/2006 10:37 AM
>  To: ####@####.####
>  Subject: cache size limit
>
>  Hi,
>
>  i'm trying to set up NewsCache on a FreeBSD 6.1 Stable machine. (2800
>  Mhz Pentium 4 / 1024M RAM). If I try to run newscache as news/news
>  user and group the .db file never grows larger than 2048M (even if i
>  set the max cache size to 40gig+). If i run newscache as root/wheel
>  the same thing happens... but with a segfault, too...
>  Here's what I see in the log:
>
>  ---
>  NewsCache[9219]: updating the overview database
>  failed(alt.binaries.blah.blah): NVContainer::make_current(): mmap
>  failed
>  kernel: pid 9219 (newscache), uid 0: exited on signal 11 (core dumped)
>  NewsCache[9192]: 9219 caught signal 11
>  ---
>  I'm using the FreeBSD port version (Makefile states PORTVERSION=
> 1.2.r.6)
>
>
>  I tried to get a cvs snapshot, but it seems that the cvs password
>  listed on site doesn't work anymore.
>
>  Anyone could give me a hint, why should i do in order to get a full
>  update / bigger than 2048M?
>
>
>  PS
>  The file system is UFS (/dev/ad0s1a on / (ufs, local, soft-updates)
>
>  Thank You!
>
> ---------------------------------------------------------------------
>  To unsubscribe, e-mail:
> ####@####.####
>  For additional commands, e-mail: ####@####.####
>
>
>
>
>

Quoting Cristian Avram ####@####.####

> gcc -v
> Using built-in specs.
> Configured with: FreeBSD/i386 system compiler
> Thread model: posix
> gcc version 3.4.4 [FreeBSD] 20050518
>
>
> I've tried to load a big file (8gig) this is the output:

I think, the CPU cannot address more than 2 GB via mmap. I try this on  
a 64bit Athlon an there is no problem with a 8GB file. But i did'nt  
try this with NewsCache and a databasefile greater than 2GB.

> Can you help me, what should I do next?

Another question is: how many articles are in the newsgroup with the  
big database file? An example with at.linux and where the files are  
exists:

ls -lha /var/cache/newscache/at/linux
total 720K
drwxr-xr-x 2 news news 4.0K Aug 17 19:44 .
drwxr-xr-x 3 news news 4.0K Aug 17 19:44 ..
-rw-r--r-- 1 news news 768K Aug 17 19:45 .db

Articles, which are greater than 15kB are stored as files in the  
directory (example: .art128009). Can you verify this situation?

You can try to run newscacheclean -t -p -s

Best Regards
Herbert Straub