Download list progress falls BADLY behind

[RayMark]

6.80 beta 6 build 4932, but with earlier versions as well

"Current Progress", "Status", "Size" columns (everything that has changing values) in the download list fall badly behind.
For example, when already several items are downloaded, they are still shown in white. After hitting the (global) Pause button - they suddenly become blue (downloaded) - instead of showing progress smoothly for each item when it is actually being downloaded.
Also, after pausing downloads, the values in Status column are finally updated for an item being assembled / par checked: "Par: 616 of 1113" - the first number suddenly jumps to its actual value 616 when otherwise was lacking behind.

More frequent updating of the changing values in the visible part of the download list would not have any noticeable effect on the download speed.
I can't tell from which version it began to become a problem, but now I keep hitting "Pause" every time I want to see the actual state of the download list.
Maybe it is more noticeable when downloading to a slower drive. With faster drives the number of already downloaded items stays low, so the problem is less glaring.
The problem occurs even when the download list has very few items in it, for example, now: only 135 items. So it seems, it does not depend on the number of items in the list.

However, when the download list contains many items (I tried with additional 4000 items - but they were all paused) - then an additional problem occurs: "Display Update Pending"
These Pending messages are not resolved quickly enough. After scrolling through the list to such messages and even selecting one, it stays "pending" for several minutes, sometimes maybe more than 10 minutes. Sometimes seems like forever. Even when most of the items in the download list are paused.
After hitting the global Pause button - visible pending items are finally quickly resolved.
So this is also wrong, the pending items should be resolved if not immediately, then at least in seconds, not in minutes.

BTW. when expanding the item currently being downloaded - I get only "Display Update Pending" lines instead of rar parts. After waiting for quite some time - close to a minute - the pending lines finally become resolved one-by-one, but in seemingly accidental order, - turns out - they are already downloaded, they are resolved from Pending to downloaded. No chance to see their progress.
I do remember that earlier it was possible to see the actual progress in such cases. Maybe it was before "Pending" were introduced? Not sure.
Now I scrolled down - on the same expanded single item - and again many pending lines. And the whole item is shown only as a half downloaded (half blue / half white). When, after hitting Pause I see that it is downloaded completely, and the next item in the download list is also almost downloaded.

You see the picture? The updating of the download list is not done in the timely manner.

UPDATE: it seems that this lack of updating of the download and assembly progress becomes very bad when the download destination drive is slow. With a faster drive I am able to actually see the progress indicator moving, and also, when expanding the item being downloaded, with a faster drive there are no "Pending" lines, everything is resolved at once and progress bars are moving.

If I connect a WD Blue drive directly to the internal SATA III (6 Gb/s) (as I most often do) - the progress is shown pretty timely.
If I specify the download destination on an identical WD Blue drive connected via USB 2.0 - Newsbin is not able to handle it, progress indication not working at all until PAUSE is applied.
Is it because Newsbin gets confused with some caching issues? I have the caching switched off on such external hard drives.
Sure, the downloads to such a drive are much slower, but the progress indication should still work correctly.

[Quade]

If I specify the download destination on an identical WD Blue drive connected via USB 2.0 - Newsbin is not able to handle it, progress indication not working at all until PAUSE is applied.

Yeah, this isn't recommended.

My guess is what you're reporting is the download "getting behind" because your download speed is fast and your disk is slow. I have a partial solution in the works. I decouple file assembly from file repair and unrar so, assembly and unrar can run simultaneously but it won't solve the problem where you can still get a bunch of chunks in the chunks folder because your disks are slow.

Someone suggested capping the number of chunks in the chunks folder to a number and simply stalling the download when you exceed that number. Another possible solution is to make the assembly happen in the download thread so, while files are assembled, the download is stalled. Either way there's no good solution for when your download speed exceeds the ability of the disks to accept the data

[RayMark]

The download speed of course exceeds the ability of disks to accept data.
Unless you download to SSD - but it makes no sense, because SSDs are very expensive and you still have to move all the downloaded stuff from SSD to regular drives. And the time required for this operation should really be included into the total time required for downloading.

So, yes, slowing down the downloading to the speed at which the hard drive can accept data is a very reasonable solution.
If possible, such a slowing down could be switched on or off - then everybody would be happy. Although I don't see any harm in it - when downloading to an SSD the download speed would still be much higher accordingly.

Such a solution would solve other issues as well. For example, today I was downloading (as an exception) to a slow disk and suddenly something showed up that I wanted to download and watch at once.
So I moved it to the top, I even paused all the other downloads individually, but they still continued to be assembled and unrarred and I had to wait more than 3 hours for the thing I wanted to have in 3 minutes.
Sure, the solution is to start a second copy of NewsBin with a different data path. I could have done so

But still, it is not very clear why slower disk causes the progress indication to stop working. Although who cares, if the slowing down will fix the problem.

But do not overdo it: uTorrent, when downloading to slower drives without speed limit, sometimes shows "disk overload: 100%" and stops downloading for a very long time. It seems that there is something wrong with uTorrent, those pauses last way too long.

[Quade]

I think downloading internal and unraring external is the best way to avoid problems with using USB2. Sure the unrar will be slow but there's a bunch of back and forth processing between the the downloaded files and Newsbin.

SSD is the best place for download and data folders too. I've never had an SSD fail and I've had them for years.

[RayMark]

Downloading internal, unrarring external?
I wonder if it would be possible (to implement?) to pause downloading when the free space on the download path drive drops to a predefined by the user level, and also to pause unrarring when the free space on the unrar path drive drops to a predefined by the user level, possibly a different value, as the space is less valuable than on the internal drive.

BTW, my download speed with bigger files (but repairing /unrarring when paused) to a slow WD Blue drive connected directly to the internal SATA 6 Gb/s (hot-swappable) is about 460 Mbps. At least 450. It stays for a long time above 470-480, then drops to 430 for a while, then goes up again over 470 or over 480. Stays over 470 longer than over 430. At the same time uTorrent is also doing something, so difficult to judge.
Also, more adjustments of the number of threads could be done, and which usenet server goes through which ISP could be changed. So perhaps the average speed could be further increased.
This does not require any moving of files afterwards and there are no issues with unrarring or progress indication falling behind (to try without pausing!). Chunks are also on the same drive.
I am pretty happy with this setup, the problems occur when on occasion I need to download to USB 2.0 - Newsbin is not able to handle slower drives, it seems.
I guess, I could experiment with internal->external and see if it is much faster.
The thing that I don't want even to try - to unrar to internal and then move to external - when the drive limits the speed, moving to external might be similar or even slower than downloading itself.

[Quade]

I wonder if it would be possible (to implement?) to pause downloading when the free space on the download path drive drops to a predefined by the user level, and also to pause unrarring when the free space on the unrar path drive drops to a predefined by the user level, possibly a different value, as the space is less valuable than on the internal drive.

Pretty sure I already explained how to do that for the download path. on the other hand the unrar path is unchecked.

USB simply isn't as good as SATA which isn't as good as NVME. Even USB3 isn't as good as internal. When you're talking about speeds as high as this, everything has to be in place to maximize the speed. It's all usable, it's just when you start optimizing for speed, you usually won't want USB in the mix.

[RayMark]

Yes, I know how to do it for download path. But unrar path is also important: if there is not enough room, the rar will not be extracted correctly. I just hope that an error code would be generated in such a case and the rar would not be deleted.
You see, even when the unrar path is on the same drive as the download path, unrar might run out of free space - I guess, I could set the limit taking into account the largest rars I download.
Still, it is easy to make a mistake in anticipation, sometimes i download pretty large rars not even noticing the size: > a 494 GB rar just recently, for example. But setting 1 TB threshold would be pretty impractical if such rars do not happen often.
But even more important is when unrar path points to a different drive.
Surely, when unrarring runs out of space, NewsBin should pause even if it does not check the space before starting unrarring. But to check the space might be much easier to implement than to pause and resume, especially if unrar is run as a separate process.