Assemble incompletes + simultaneous downloads

[megalomando]

Two questions.

1. Is there a switch to have "assemble incompletes" do so automatically? I am often finding several downloads have stalled and not gotten to the par stage because the incomplete likes were not assembled and there were not enough pars to assemble the completed ones. Finding one of these, I have to manually tell nNewsbin to assemble the incompletes and then it either begins to unrar or I have to tell it to resume downloading more par (which are on pause). At this time all works perfectly but I'd like to have it assemble and continue without having to tell it to do so.

2. Another item I might have missed is how to have more than one download happen at the same time. I've missed the switch for that if the option exists.

Thanks and I really love this program, it is brilliant.

[Quade]

1 - What version are you using? In 6.81, if you enable "Aggressive Assembly". It'll tend to assemble incompletes early. It does it if there are PARs and autopar is enabled.

2 - All the downloads threads work on the top most file and work their way down as the file completes. Used to be you'd see files download in parallel with one connection working on each file. Now all the connections work on the top files and work their way down. The download speed is the same either way.

If you download small files, and have more connections than blocks in the file, you'll see something that looks like the old way but it's still working on the top files.

[megalomando]

Thanks for the reply.

1. I'm using V 6.81 #5080
I do have under Autopar options, Aggressive assembly mode checked. However, the tag line says (bypass retries and assemble when you have enough pars). This works beautifully when there are no incompletes to be filled and with this, Newsbin does not download un-needed pars when it gets enough RARs and it unrars as expected.

What it does not do is assemble incompletes in the RARs above the pars. Most RARs will say completed but those with pieces missing still say downloading and the pars say paused. This is what I am not sure that Newsbin deals with automatically & if so, how to engage it to do so automatically. - At this point, I see the held up file, either because it is not unraring and other files are downloading below or, it is a file that will not unrar. What I manually have to do is click on the + and expand the drop-down list, see which ones are incomplete, tell Newsbin to assemble those incompletes, and then Newsbin sees that more pars are needed and downloads what is necessary from the paused pars and then unrars.

Unless I tell it to assemble the incompletes, it stops and waits for these manual instructions. - What I would like it to do is upon completion of the RAR download, retry the download and if no pieces download to fulfill the incompletes, then assemble the incompletes automatically without needing my intervention to do so.


2. I have 300 Mbps download, usually, the speed is seen + - 200Mbps. Sometimes I find one file downloads at say. 20, 30 Mbps and when it is complete, the next file is downloaded at some much higher number. I would like to be able to engage multiple files to download simultaneously, not the individual RARs that make up a file.

Again, thanks for the reply.

[Quade]

How many retries do you have set? Retries gate when "assemble incompletes" kicks in. I recommend 2 retries normally. There's a balancing act between assembling too soon and not soon enough.

The only thing that should potentially stall the download is the unrar/repair process but that's optional. The performance options let you tell Newsbin whether to continue to download during unrar or not.

With the right server and PC, you should be getting 300 Mbps. It's not that uncommon to get twice that speed if your internet and server are up to it. 200 Mbps is just idling along to Newsbin. The question is why you can't sustain full speed. There are a number of possibilities. Your PC, security software, internet connection, server. It's the kind of thing you need to investigate on step at a time. What news server are you using?

[megalomando]

Thanks again.

1. I had it set to 50, I nave it now set to 2 as you suggested, hopefully this is the solution.


2. The PC I'm using is a new board, Asus Prime z370-A, i7-8700K @ 5GHz, 32 gig CAS14 G.Skill RAM. Malware Bytes and Defender are the antivirus. When I use DSL reports to check my speeds it is usually around 275 Mbps, this also when I'm seeing slow downloads. I am downloading from Usenet and Astraweb. I use wireless to connect and the router and antennas allow me to hit 300 Mbps. I can say that I have accessed DSLReports while seeing a slowdown of files and always the download is over 200, even when simultaneously downloading the file.

The impression I get is some files download slower than others. If I have say, 3 files in Queue, the first might download in the 230's, the 2nd in the double digits and the third back near 200. That may not be the source of the slowdown but that's the effect it seems to be. I doubt it is Newsbin, it is not repeatable, I only see it at times but it happens often.

[Quade]

Before I did any more trouble-shooting, I'd try a Ethernet cable to the router and see how fast she goes. Trying to sustain high speed downloads over wifi seems problematic.

[megalomando]

No can do. That would involve drilling a hole in the floor large enough to pull the plug through. I doubt it is the wireless router as it usually sustains the maximum download speed on files. If there's no option to download multiple files simultaneously, I suspect there is not much more to do, Is there any convenient way I can run two instances of Newsbin simultaneously?

1. Your suggestion of selecting retries @ 2 did the job. It is now assembling the incompletes and continuing with obtaining enough pars to unrar.

one out of two is good.

Thanks

[Quade]

If there's no option to download multiple files simultaneously, I suspect there is not much more to do, Is there any convenient way I can run two instances of Newsbin simultaneously?

This is how usenet works:

1 - A typical 50 meg RAR files is divided up into 67 blocks of data and these 67 blocks are then posted to the group.
2 - To make a 4 Gb file, you need to post 80 50 meg rar files divided into 67 blocks per file (not including the PAR Files). So a total if 5360 blocks of data are posted to the news server for a single 4 GB set of files.
3 - Newsbin downloads blocks of the files. If you have 10 connections enabled, Newsbin downloads 10 blocks of the file at a time.

The connections are fully utilized, the whole time pulling data. That's how people can get close to 1 Gbps download speed. This is more efficient than downloading files in parallel because all the connections remain active all the time.

Your speed problem is external to how Newsbin downloads. Many people download with Newsbin at easily 3 times your current download speed. Running 2 instances of Newsbin won't fix your problem. It'll just add more problems.

In your shoes, I'd experiment with different news servers. I'd make sure they are all the same priority so they were all used at the same time. I wouldn't use a crazy number of connections. Maybe 8-10 per server. I'd make sure you're not running out of connections (errors in the logs).

I'd remove any security software, reboot and try some downloads. This has often improved performance for many. The only security software that doesn't seem to impact performance is the built in MS stuff.

You may find you're simply ham-strung by your wireless connection but who knows?

[zaphodalive]

I have no idea if this has been covered since this thread in 2018, but I don't know what to search for to find something more recent.

I often find when downloading a TV series that the episodes will not be in order (and like a single download, so the contents aren't sortable). If I'm keen to watch the first episode I would like to prioritise downloading those blocks like some torrent applications can do, either by changing a priority value or ordering within that download.

Also, I have 2 Usenet providers and neither one of them maxes out my bandwidth. This means when the application downloads items from the top down, and a file is only available on 1 of my Usenet providers' servers, then I don't get full speed. My download list might have a file further down that's available on the other Usenet provider's server, which it could download at the same time and speed up my overall downloads. Maybe when the client receives a 430 "no such article" from a connection it just moves on through the files until it finds one that it has?

[Quade]

What version are you using?

1 - The download order should be the order in the download list.

2 - After download though, when the files are sent off to unrar, the order becomes fixed because unraring is "first come, first serve".

3 - Newsbin will search a certain distance down the download list looking for posts to download. It's dependent on the whether you're using server priority or not. A server at a lower priority won't even attempt a download till one of the other servers has gotten a chance to download and fails. If they're both the same priority then they should both download simultaneously. So a lower priority server will go idle until some chunks fail to download.

4 - 430's in general seem to be slow. I think it's because for good downloads, the article is coming from the fastest storage on the server while 430's have to be checked against the entire server.

I regularly max out my 1 Gbps connection to usenetserver with, I think 16 connections. Another user use had two different ISP's and gets 1.5+ GBps download. His Newsbin downloads faster than it can unrar.

If you have one server that's much better than the other you might want to make it the highest priority and make the worse server lower priority so it's only checked if something is missing from the first server. A slow server can gate the download speed from a fast server. Two servers isn't always better of one server is slow.

[zaphodalive]

1) The download order is the order in the download list, but a single download can have multiple files within it, sometimes .mkv files that I could play as soon as they're complete. Those files within a single download aren't sortable, and there's no option to prioritise files, so I have to wait for them to download in the order they are within that download. For an NZB that has, say, 150 TV episodes, and I want to get started on episode 1, that can be an awfully long wait.

2) Only talking about playable video files.

3 & 5) I've got 2 providers, one with 16 connections and one with 18. They're cheap providers, and I get roughly 180Mbps per provider. What seems to happen is that all connections on both servers just hammer away trying to download blocks, but when one can't find them they don't move ahead at all. I don't know if that's because they keep trying on blocks that aren't there, or they can't keep up with the speed that blocks are downloading on the other server to get ahead. Either way only one server ends up getting utilised.

[Quade]

3 & 5) I've got 2 providers, one with 16 connections and one with 18. They're cheap providers, and I get roughly 180Mbps per provider. What seems to happen is that all connections on both servers just hammer away trying to download blocks, but when one can't find them they don't move ahead at all. I don't know if that's because they keep trying on blocks that aren't there, or they can't keep up with the speed that blocks are downloading on the other server to get ahead. Either way only one server ends up getting utilised.

It's not retrying but 430's are very slow. Each failed block is tried by both servers before it's declared "failed" so, likely both servers are stalled because 430's are slow. When a server gets a 430, the server will move on to the next block but if it's 430'ing too the whole process is slow.

By default Newsbin 6.90+ will toss a download with > 10% failures. Depending on how large the file is, 10% might be quite a few blocks.

I'm not sure what version you're running.

There's currently no way to sort inside a grouping of files. I can think about maybe being able to pull a file out of the grouping so you can change the order.

[zaphodalive]

Ok just to user similar terminology, each download can be either a file or a grouping (of files).

By default Newsbin 6.90+ will toss a download with > 10% failures. Depending on how large the file is, 10% might be quite a few blocks.

How about also doing checks by server, so if a file within a grouping has > 10% failures then that server moves onto the next file within the grouping. If a download (either a file or grouping) has > 10% failures then that server moves onto the next download in the list. A download is still tossed with > 10% failures from all servers.

There's currently no way to sort inside a grouping of files. I can think about maybe being able to pull a file out of the grouping so you can change the order.

That would be great. Even better (I think) would be if you could keep a grouping together but be able to sort within it - that would keep the download list tidier.

Hope those ideas make sense; I can provide examples if needed.

[Quade]

How about also doing checks by server, so if a file within a grouping has > 10% failures then that server moves onto the next file within the grouping. If a download (either a file or grouping) has > 10% failures then that server moves onto the next download in the list. A download is still tossed with > 10% failures from all servers.

It's a good idea. I'll have to look into it.

I have to verify but, I'm not sure 6.91 will group media files anymore. Which would solve this problem without adding "in grouping" sorting. I put B10 up in the forum the other day, in the beta section. You could try it.