Emergency pause fails to protect

[RayMark]

When the free space on the hard drive(s) where NewsBin data and downloads are located (DataPath, DownloadPath) runs out, NewsBin pauses, supposedly until the user can free more space - so that no downloaded files are lost or corrupted.
Unfortunately, it is not what happens.

I have DataPath and DownloadPath pointing to different hard drives.
If either of them runs out of free space, things go very wrong even though NewsBin "pauses" (still working in the background!!!) - many bad or zero length files are generated as a result.


When the DataPath drive is full, even when NewsBin is "paused", corrupt and mostly 0 length chunks are still generated, even though NewsBin is not downloading. I assume, these new chunks are created as a result of a background assembling/decoding. But they are 0 length because there is no room for them.
So when the user makes more free space on the drive, those 0 length chunks then are used to create files on the DownloadPath hard drive, resulting in corrupt or 0 length files.
I recently had such an episode with 6.80 beta 4 when because of a problem with password protected posts, there was a bottleneck and the DataPath drive was filled.
As the result many rar parts on the DownloadPath drive were damaged or zero length (correctly indicated in all uppercase letters) in many different posts - hundreds of bad files were generated.

When the free space runs out on the DownloadPath drive, things are not much better.
These situations perhaps are easier to avoid, but they happen from time to time. It happened again to me a couple of days ago: I found NewsBin paused, but when I made more free space, NewsBin generated (on the DownloadPath drive) many damaged or 0-length rar parts in two posts - even though they were downloaded correctly to the DataPath drive, then they were damaged while moving to the DownloadPath drive. So I had to download those damaged rar parts again unnecessarily.
BTW, Retry or Resume Download do not work in such situations. The correct approach is:
1. to close NewsBin.
2. to delete the damaged (uppercase) rar parts manually.
3. to start Newsbin again.

I think Newsbin should be fixed in such a way that no bad chunks or bad files in the download location were ever created - a better pausing is needed, without background processes. At least for such automatic emergency situations.
But perhaps better also when the user pauses NewsBin: if I pause it because I need all the computer resources for some critical task and NewsBin is still using cpu and data bandwidth for hard drive activities - it is wrong.

Also, once NewsBin pauses automatically for low free space emergency, it should not resume by itself after only one small file is moved by the user, because the same situation repeats again and again, The User has to pause NewsBin anyway.
After an emergency pause, NewsBin should resume only when the user decides it can.

BTW, is there a way to set some custom free space limit: to tell NewsBin to stop downloading when a specified free space threshold is reached?
I would like to leave some reserve to avoid such emergency situations in the first place.

[Quade]

When the free space on the hard drive(s) where NewsBin data and downloads are located (DataPath, DownloadPath) runs out, NewsBin pauses, supposedly until the user can free more space - so that no downloaded files are lost or corrupted.
Unfortunately, it is not what happens.

My advice to you...is stop running out of disk space.

[Filename]
MinFreeSpace=20000

In the NBI file, you can crank up the disk free threshold. I'd probably set it to 20-50 gigs. Maybe even 100 gigs if you keep having problems. The issue is that your download gets way ahead of your file assembly. It's the file assembly that's running you clean out of space. File assembly can't be paused. Pause just stalls download. 20000 is 20 gigs. It's in megs.

[RayMark]

Four questions:

1. (not really a question)

That is exactly my point - file assembly can't be paused - that is exactly the bug!
File assembly should be paused when it clearly makes no sense to continue with it because it only generates garbage when there is no free space.
Besides, it gives a false hope that NewsBin is able to handle it - it "pauses" after all.

2.

Is it always safe to close or even to kill NewsBin - after restarting the file assembly would continue as if nothing happened?


3.
--- The threshold issue

[Filename]
MinFreeSpace=20000

Is it for DataPath drive or for DownloadPath drive, or for both the same and cannot be set different?


4.

Perhaps I would like to have chunks on the DownloadPath drive, using the DataPath drive mostly for the spool with headers.
Because the DownloadPath drive is changed frequently, and the drive with headers should remain the same.
If i can put the Chunks folder on the DowwloadPath drive, then the issue with DataPath drive running out of free space would be eliminated.

Yes, I would really like to try with chunks on the same drive as assembled downloads. To see how it affects the speed. In my case the speed may even improve, because my DowloadPath drive usually has the fastest connection: it is usually inserted into the docking station integrated into the PC case itself (on the top of it) connected directly to the fastest internal SATA.
I keep replacing the DowloadPath drives very frequently (hot swap is supported), therefore the spool with headers must reside on a different, more permanent drive.

I hope there is a way to place the Chunks folder onto the DownloadPath drive instead of DataPath drive?
I vaguely remember seeing such an option in older versions of NewsBin but I can't find it now.
In .nbi?

[Quade]

1 - Don't run out of disk space.

2 - As long as you don't wait too long to run it again. The chunks sit in the chunks folder but the chunks folder gets auto-cleaned when the chunks get 2 days old.

3 - Both. Either one running low will cause it to pause.

4 - You could probably try mounting a drive or folder on another drive to the chunks folder. Keep in mind, if it's not a monitored drive then you can run out of space on this other drive and Newsbin won't notice. This goes back to "don't run out of disk space". I recommend turning off the recycle bin for the drives Newsbin is using too. This will improve the speed of cleanup after unrar significantly.

[RayMark]

4.
Oh, so there is no straightforward way to put chunks at a different location? Perhaps you could add such a possibility. ChunksPath = .. in NBI file
Or simply an option to use the DownloadPath for Chunks folder instead of DataPath.
It should not be too difficult to do?
Chunks is part of downloads after all, nothing to do with headers or other data.

So NewsBin uses the recycle bin? It never occurred to me. I do everything with FAR and I have recycle bin disabled in it. But I haven't noticed any increase of the size of the recycle bin on those drives - it is usually about 130 bytes.

[RayMark]

OK, so I did this:

- closed NewsBin

- MinFreeSpace=20000

- moved the Chunks folder with its content from G:\NBData to L:\NNTP\

- mklink /j G:\NBData\Chunks L:\NNTP\Chunks
Junction created for G:\NBData\Chunks <<===>> L:\NNTP\Chunks

started NewsBin

So far so good, seems to be working
Still, it would be preferable if NewsBin itself allowed to have Chunks on the download drive instead of using a junction point.

[Quade]

I'm unlikely to make the chunks folder an option. It's important for Newsbin to always know where the chunks are.

Another reason is that you can work around it if you really want to like you just did here.

[RayMark]

Well, as long as workarounds are working...
The data drive running out of space problem solved, and the speed increased, I now regularly see speeds around 430 Mbps. At least 100 Mbps more than before.
And the extracting does not fall behind, no waiting downloaded posts.

Much better NewsBin now
Maybe it is a sign for you to allow Chunks on the DownloadPath drive after all...

Actually, with some larger files, download speed, once it picks up, never goes below 470 Mbps, reaching even 494. I probably missed > 500
The number of threads and servers used (I am using up to 5 different servers) might further increase the speed.