0 "new files"

[n4aof]

I'm running version 6.80B9: 00 72 06 C3 BF 8E

Lately it's been acting very strangely. Everything works fine when I feed it a NZB file (which is how most of my downloads are done), but I also download the headers for about 130 groups from three servers (Astraweb, Astraweb EU, and XLned). I normally download the latest headers about every day or two, then I look through the new posts to see if there is anything that catches my eye, like ebooks from a favorite author or a music album or classic movies or TV shows that aren't on my regular watch lists. This has worked well for me... until recently.

For the last few days, newsbin had downloaded all the new headers (anywhere from a few hundred to over a million for each group), but after the header download finishes the "new files" column shows 0 new files, and of course nothing shows up when I show the posts (with the Hide Old filter).

If I leave newsbin running in the background waiting for whatever my indexer turns up, several hours later news bin will occasionally show some number of "new files" for some groups (without doing another header download).

I tried right clicking on each group and selecting Post Storage, Use Download Age to see if resetting the groups would help -- it didn't make any difference.

Several of the groups where newsbin is telling me there are 0 new files are groups that I know are very active, and where the header download shows downloading several hundred thousand new headers.



Quick add on:

While I was typing another post, I closed and started newsbin twice. The second time I started newsbin, I noticed that it now showed a few hundred new files for some of the groups (without having downloaded any new headers).

[Quade]

1 - Are you using any kind of header download filters?

2 - If you look down at the status bar do you see a number in parens near "Cache" (XXX) is what it would look like.

My first guess is that header import is stuck and that you have a largish number in the parens which indicates headers aren't importing.

[danrobin]

I'm having that problem. My Cache:400/400(0) Is this messing with the header downloads and if so is there a way to clear it out?

[Quade]

My Cache:400/400(0)

This says there's no headers to import. What group are you downloading headers from? Are you seeing them actually download?

[n4aof]

1 - Are you using any kind of header download filters?

2 - If you look down at the status bar do you see a number in parens near "Cache" (XXX) is what it would look like.

My first guess is that header import is stuck and that you have a largish number in the parens which indicates headers aren't importing.

No header download filters that I am aware of. I do have a couple of filters that I sometimes use to filter the posts being displayed in some groups when there are a lot of posts - but (at least so far as I know) those don't filter the headers being downloaded, only the posts being displayed.

Cache says 189/200 (888)

[danrobin]

It's running ok I watched the header d/ls there were none for those groups.

[Quade]

Cache says 189/200 (888)

Yeah, you have 888 chunks of headers trying to import. That's why you're not seeing any new headers. With Newsbin running, I might move all the ".gz" files out of the import folder. One of them probably won't move. Exit Newsbin, move this
remaining GZ to some other folder, then move the remaining GZ's back. Start Newsbin. Don't download headers and watch if that number shrinks. If you have automatic header updates enabled, disable it for now till you catch up.

In the options, "Open Data Folder" will take you to the Import folder (well the folder above the import folder).

[wiggins09]

Same issue.

I added some groups, no filters applied & have set storage to download age.
I have watched NB d/l hdrs using 100% of my bandwidth. Hdrs number from 10k to 2m+ per group, cache is working - xxx/xxx (xx) is changing regularly.
After 3 days I have one visible post in a non-filtered pane, & the NB session d/l monitor shows gbs of hdrs per day leeched. (5gb hdrs while I typed this...) But spool folders have c.56k each.

These groups are new to me (I've not used them before) but comments on forums say the posts in them are OK.
Why does all this get garbaged?


BTW Thanks for all the work you guys do on NB, it is a great prog.

[Quade]

Why does all this get garbaged?

1 - Some groups are NZB only these days and downloading headers from them is sort of pointless.

2 - Some groups are so large that you can fill your hard disk with just the header downloads.

3 - If your machine isn't pretty damn fast, it can take a long time to feed a massive group. Even at the fastest a 40 meg GZ file will probably take 12-15 seconds to feed into the header database.

If you have 5 GB of GZ files, that mean you downloaded 50 GB worth of headers. I've been thinking about ways to double the speed of the header feed but still that means it's going to take a long time to feed 50 gb of headers into the DB.

[wiggins09]

OK Thanks Quade.
I have left NB running for 7 hours, all activity has stopped & I have 1 header shown after 3 days with 0 filters & no age restriction....
I have decided that it must be a lost cause at the moment.

[n4aof]

3 - If your machine isn't pretty damn fast, it can take a long time to feed a massive group. Even at the fastest a 40 meg GZ file will probably take 12-15 seconds to feed into the header database.

If you have 5 GB of GZ files, that mean you downloaded 50 GB worth of headers. I've been thinking about ways to double the speed of the header feed but still that means it's going to take a long time to feed 50 gb of headers into the DB.

My machine isn't blazingly fast (not my the latest standards, anyway, Win7, 6GB RAM, G620 CPU at 2.60GHz) but it WAS fast enough for NewsBin until recently. So far as I can tell from other programs, the computer hasn't gotten slower and I don't think the groups have gotten that much more active for there to be a major increase in daily headers. I'm not running out of disk space (the working drive is 1TB with 400+GB free).

Since my initial post, I have started letting newsbin run continuously until it finishes processing the gz files. On good days that means downloading headers in the evening (which takes way under an hour), getting most of the posts to show up in a couple of hours, then waiting until after lunch the next day for the last couple of gz files to feed into the db. Other days (like right now) there are 50+ GZ files in the cache that slowly make it to the db perhaps one or two every hour or so (with, of course, more posts appearing after each gz file finishes processing).

Bottom line here is that the headers still download quickly, but that the last couple of betas seem to take longer to finish processing them. Whenever the cache finishes, I'll try installing B10 to see if that does anything.

By the way, the CACHE line on the status shows 165/200 (54) but there are only 28 GZ files in the gz folder.

Those 28 GZ files range in size from 9KB to just over 42MB and they total 297MB all together.

[Quade]

If you leave Newsbin running and set it to download headers every 30 minutes, you'll never have to wait for them to finish importing. You might want to revisit what groups actually deliver you meaningful files and prune the ones that don't too.