Memory usage 6.80RC2 while importing headers

[Roel1976]

Hi,

after I have downloaded all the headers from a large newsserver. Newsbin uses more than 26GB memory to import the headers.

What I see is the following on my storage:

DownloadsV2.db3-wal is 1.54GB and growing.
There is one Storage.db3-wal in a subdirectory (21.4MB)
In the header import directory there are 2600 files (in total 97GB)

When I wait to newsbin finished one of the import files, the DownloadsV2.db3-wal is not truncated or closed.

The question is when newsbin does a commit while importing the directory .\import (containing 2600 files). Is it after everything processed or after every file?

[Quade]

DownloadsV2.db3-wal is 1.54GB and growing.
There is one Storage.db3-wal in a subdirectory (21.4MB)
In the header import directory there are 2600 files (in total 97GB)

None of this is memory though. It's disk space.

Depending on what groups you hit, headers will easily exceed 1 TB of disk space. If that's what you want that's fine but it's not going to be quick. I expect it'll takes at least a month to get them all downloaded and imported. Newsbin compresses the data too so, you're looking a multiple TB's worth of network download.

DownloadsV2 is your download list. I consider 1.5 gigs to be excessive for the download list. It sounds like you're not only downloading headers but feeding the header download output to the download list. If you're not actually downloading anything but headers, I'd probably delete downloadsv2 which wipes the download, failed and wish lists. Then re-start the header download (don't "download all" again, just update headers normally).

97 GB of GZ files in the import folder, represents about a TB of headers to process. The GZ files are compressed and 10:1 compression isn't unusual.

[Roel1976]

Thanks for the explanation

But I have still having problems to run Newsbin without crashes.

Before I reinstall Windows and newsbin to some unexpected dependency of a system dll. How could I help to solve the Newsbin crash? I could make a adplus crash dump if you want.

I have currently cleared the download database and the import directory and disabled the autodownload. So far I could determine this is running stable after some days importing the headers.

When I enable the autodownload so it downloads the lastest files from some newsgroups within a range of a few days. Newsbin crashed with "NewsbinPro Newsgroup Reader has stopped working" from Windows. Newsbin shows only 68 entries in the Downloading Files tab. In the Logfile.txt nothing is logged as a error within the last few minutes before the crash.