Compressed Headers

[stevef]

In case anyone has missed it, you can now download compressed headers from our servers using Newsbin.
You should see a substantial speed increase anywhere from 5x to 15x.

Compressed header downloading is available in Newsbin version 5.50 B7 and above. It might be available since B6.
The option can be activated in the server options screen. Make sure the "Header Compression" box is checked.


Enjoy!

[richy99]

works fantastically well too

[ozzii]

Works for me. Thanks

[hugzzz]

Just gone over to Astraweb from Powerusenet and WOW does the compressed headers make a hell of a difference.

On a 10Mb cable it's giving me a 95Mb header download

[Quade]

Yeah, it's pretty sweet.

[obvious]

Been trying to download all headers for a.b.multimedia using header compression. There were gaps in the data folders which newsbin 9246 seemed to be trying to fill but it just kept coming up with:-

"[12:24:36] Level:ERROR SERVER=eu.secure.news.astraweb.com, MSG=NNTPServer: Compressed Headers - No data after decompression"

..so I ended up deleting the a.b.m data folder and trying to 'download all'. It starts off with 49 million headers to download then after 15 million headers have downloaded it fails and restarts with 33million headers to go then promptly gets stuck repeating the same "No data after decompression" message.

If I cancel the failing/looping 'download all' with 33 million to go and do a 'download latest', it starts again from 49 million

[stevef]

Quade's aware of the "No data after decompression" loop. I believe it is fixed in the latest beta (might be IRC private beta).

[obvious]

It works fine (and still accelerated) if I uncheck the 'yEnc XZVER Support' box.

[Quade]

If you don't run the most current beta's,you should. I assume you're running B7 which was the first version with compression. Steve brought this to my attention and I fixed it in B8. Pretty sure B8 has the fix anyway. My todo list doesn't track versions, just days.

[richy99]

i can confirm its still has this issue in b9

[Quade]

The message is still there. Newsbin doesn't abandon the download now though.

Edit: Apparently this isn't true. I'm looking at it.

[richy99]

it stopped for me the same as it did for obvious above, it got to a certain point then just repeated the error message and the download latest just was flashing at 0 as the error repeated

its almost as tho when it hits a certain area it just chokes and wont go any further

[obvious]

If you don't run the most current beta's,you should. I assume you're running B7 which was the first version with compression. Steve brought this to my attention and I fixed it in B8. Pretty sure B8 has the fix anyway. My todo list doesn't track versions, just days.

Been running x64 B8 since a couple of hours after it was posted.

[Quade]

Yeah, none of the internal testers reported this. It's a little different than Steve's original issue. I'm looking at it.

[obvious]

[huanhuan19]

Works beautifully!

But I have a question to limit the size of the hearder downloads. It is too much to download all of them even if the speed is over 5 mbps.

[mojo-chan]

Even with compressed headers they are coming down really slowly for me. Newsbin 6.10RC2, ssl-eu in use. They seem to go in spurts, fast for a few seconds then stall for a few seconds.

[Quade]

Maybe try the US server and try it without SSL? If not using SSL works, you can actually set it up to download headers without SSL and use SSL just for the file parts.

[hemz]

I've had Astraweb over 2 years and never knew about this, how do i set it up on Newsbin.

Am a newbie to the site any help would be appreciated. using the 6.11 build.

Thanks

Craig

[Quade]

It's automatic these days. You don't have to do anything.

[Plankton]

stevef anychance you could fix header compressions across all your server farms and give it traffic priority seeing as it's only 1 connection so users can actually download all the headers of a large top group without it timing out or dropping to low speeds for days?

Thanks

[stevef]

There is no artificial priority difference between header and body downloads.

Any slow header download in the last 5 days can be attributed to a header server problem. This is no longer an issue.
If you're having slow header downloads at this very instant, and it is reproducible, then send helpdesk a ticket for testing.

[Quade]

How many connections are you running? Lots of people are running too many connections and causing issues. Just for grins, I'd run it back to 10 connections and try again. I get crazy fast speeds to AW when everything is working.

[stevef]

I wouldn't call 40mbps header download speeds "slow".

[qubin]

How many connections are you running? Lots of people are running too many connections and causing issues. Just for grins, I'd run it back to 10 connections and try again. I get crazy fast speeds to AW when everything is working.

he only has 1 connection for header fetch?

[qubin]

I wouldn't call 40mbps header download speeds "slow".

So as users we can download binaries at 120mbps but it's perfectly acceptable to download headers at a fake 40mbps compressed speed for hours and days and get to about 10% and then it times out without completing?

[stevef]

You are comparing article downloads using multiple connections, whereas header download is using a single connection per group.
Article download there is no server processing. Header download there is heavy parsing of the article to give you that single overview line.

[Quade]

I always update multiple groups at a time so, all my connections are used for header downloads. I've seen speeds close to 1 Gb/sec to AW. More typical is 500-600 Mbps. "Per connection" speed of 40 Mbps isn't really that bad. Giganews will barely do that, typically they're slower "per connection" than AW.

Newsbin has to work much harder downloading headers than downloading files. I wouldn't be surprised if Newsbin processing plays a role in that 40 Mbps per connection speed. I mean if nothing else it has to decompress a multi-mega-bit/sec data stream. When you use multiple connections, you get multiple cores doing the decompression.

If you just let Newsbin update your groups automatically once an hour, you'd always have mostly current headers too.

[qubin]

You can't know what is in a newsgroup and choose what to download unless you can download and view all the headers.

You can't download all the headers from boneless without it timing out out so we are paying for a service we can't use as we can't view the complete content.

Please don't says NZB groups as most of us have no interest in others choosing a small selection for us on our behalf or paying for a subscription search as we have already paid for access.

[Quade]

You can't download all the headers from boneless without it timing out out so we are paying for a service we can't use as we can't view the complete content.

Just restart the header download. Newsbin will continue where it left off. Just don't "download all headers" a second time because then you're starting over again and won't make any progress. Even if you completely exit Newsbin, restart and download headers again, it'll still download from where it left off till it gets all the headers.

I can only guess you "download all headers" again and throw away the headers you've already downloaded.

The problem you're having might just be you. Your PC or network. With Newsbin, it's easy to recover from a connections dropping out or disconnecting, just start the header download going again.

[Cocha]

I don't understand this crazy idea of repeatedly compressing such a large database over and over again per customer.

There should be a system that auto prunes any headers from the server database that is older than 30 days and compresses them once into monthly header archives that are then served to the customer as a multi part binary files so they can max out there connection like any other file.

[stevef]

Sounds like a good idea.
Now, how will you serve me the data when I request a range that is a fraction of the multipart, or a request that begins in the middle of one multipart and ends in the middle of another multipart? And remember, I don't want any extra data, because I've got a download cap.

[Cocha]

Couldn't there be a system coded that runs in parallel so if they need a fraction of a multiplart they get switched back over to the current way of serving the headers for that header range until they are back in sync with the monthly compressed snapshots?

Or even a system for those that request all headers from a group to send a new server command so they are treated differently and use a compressed archive for most part to take the load off the server and give them max speed?

Just a thought.

[stevef]

If you look at the pattern of requests, nearly ALL requests will be partial multiparts.
And introducing new commands would require the co-operation of all USENET providers and NNTP readers. Just introducing compressed headers without breaking the existing NNTP specification was a headache in itself.

[Cocha]

Ok my last attempt at an idea to make things easier for users. =)

Could you do a virgin install of newsbin at the Astraweb network and download all the headers of say the top 10 or 20 groups and when it has finished leave the SPOOL_V6 folder from newsbin in the members area for us to download and copy to our own newsbin folder as that is a compressed database of the headers and all we need to do is then update the headers from when it was done.?

Most of the other groups are not a problem for getting all the headers so it would only need to be the top ones like boneless.

Thanks

[stevef]

Could you do a virgin install of newsbin at the Astraweb network and download all the headers of say the top 10 or 20 groups and...

No, that's not going to be workable. Every time there's an update, you will all be downloading the entire SPOOL_V6 folder again. Plus the infrastructure at the moment has you downloading across a load balanced server farm. In your scenario, we would have to additionally make sure the web servers a load balanced just to serve out the SPOOL_V6 folder.