Newsbin 6.91 B8 Beta - build 5775

[Quade]

https://s3.us-east-1.amazonaws.com/down ... 8-5775.exe

The release notes are in the install folder. So I'm not going to repeat them here.

Highlights:

- Passwords from NZB's loaded to the post list now propagate to the download list when the files are added there.

- RAR library is now the primary for decoding RARS and 7Z for 7zip. It makes the progress bar work again. The difference in performance between the RAR and 7z library was negligable.

- Lots of optimization to loading groups. Group loading should be generally faster than before I haven't timed header import but it should be faster too.

- RAM optimization for download, should mean software never bottlenecks the download. Just the server and the network. I've had a reports of 1.6 Gbps download speeds.

- Improved compatibility with Sonarr. Sonarr added some checks that popped warnings about queue removal. That should have gone away. Sometimes Sonarr sends a naked MKV over the interface with no PARS or RARS. These files weren't getting reported up to Sonarr because they didn't decode. Now every set of files from an NZB gets added to the "History" which is what Sonarr checks to import downloads. Sonarr still only snatches the files from NZB's it supplies.

Lots of esoteric internal changes which should make Newsbin less prone to memory leaks (not that it leaked before). Some of the code in Newsbin is over 25 years old. I move it forwards to use improvements that the language has implemented over time.

[flat_beer]

Just did an upgrade from build 5731 to this. I found one issue with the upgrade.

Under the groups list I have about 10 folders to categorize the various groups. (like: Movies, TV, Music). Only 2 of the folders were expanded when I shutdown the Build 5731. When I started Build 5775 the first (top most) of the expanded folders was gone. It looks like all the groups in it were put into the "Unsorted" folder.

Ok. I did some testing before posting this and I think I found MY problem... It seems to repeat this at every Newsbin startup... My folder was "prn" (which is how I was abbreviating adult content). prn: was the old DOS lever filename that could be used for the printer. This is the first version of Newsbin that this seems to be a problem with. I'm guessing that other old DOS reserved names can't be used either. (I can only think of CON: at the moment.) Probably single letters can't be used either since they might map to drive letters.

Am I correct in assuming that old DOS reserved names can't be used for folder names?

[Quade]

There's an issue when the names are too short. Not what the names are.

I'll have to look into a fix

thanks for the report.

[WhatZit]

There's an issue when the names are too short. Not what the names are.

Very probably a related issue, but after upgrading to B8, one of my watch folders was being "automagically" deleted from the .NBI's [NZBWATCH] list on startup. BTW, it disappeared simultaneously from ALL the .NBI's at the same time after only one run of Newsbin, including the .BAK & .OLD!

The configuration entry (AddTop, DownloadPath, etc) for the watch folder always remained present & untouched.

I simply could not figure out what the {bleep} was going on. I tried deleting & recreating the folders on the drive in case there was some whacky corruption or junctioning or attributes or something. Nope. I manually created the entries in the .NBI, but they were "automagically" deleted again from EVERY .NBI whenever I loaded Newsbin.

Finally, I was about to roll back versions when I came here and read your comment above.

Well, well... my vanishing watch folder was called "TV" in the NZB Options tab, pointing to a folder called "!T!". Renaming the folder on the drive to "!TTTTTTTTT!" had no effect.

However, renaming the watch folder under NZB Options to "TeeVee" stopped it from being automatically deleted on startup! Pretty much the "too short" issue, I think.

I reckon you have multiple things to investigate:

1) Non-null strings being too short to be used

2) User settings being deleted from their configurations without notice

3) .NBI backups that actually backup previous states

[kenr]

Another strange one with new beta. I seem to be downloading ALL the files in a GOG rather than what is in the watch list.

[Quade]

1) Non-null strings being too short to be used

2) User settings being deleted from their configurations without notice

3) .NBI backups that actually backup previous states

The actual NBI backups are stored in "GuiItems.db3". The backups on the data folder are only there in case Newsbin closes or windows crashes 1/2 way through writing the NBI.

As for the rest, everything seems to be what I mentioned before. That field names that are two small are ignored.

The reason you see every NBI you load change is Newsbin loads the NBI. Then overwrites the old one when you save. If it's ignoring short entries, which it is, when the NBI is re-written, they don't be there.

[Quade]

Another strange one with new beta. I seem to be downloading ALL the files in a GOG rather than what is in the watch list.

I have about 20 watch list item. 5 of them are from a very busy GOG. They're only downloading matching files. I'd suspect that the filter is the problem. I'd verify the filtering is only feeding matching files to the watch list.

[kenr]

Found issue
Several of my search topics had the search for blanked. I think it was the ones with regex's in them but not sure. It was ones that contained a search like "a. b. cdef" the others are all "abc def" types.
EDIT:
yep found a backup and it was definitely ONLY the ones that had some regex in them like M[\.\s] Z[\.\.s] that were wiped out.
Just tried with a regex in the test and it added all the files not the ones it should have.
Will disable them just now.

[JayPea]

Upgrade to latest beta wiped my servers? They existed in the .nbi but were ignored resulting in a MOTD-like-message stating 'you have no servers'. Which was a lie. Resorted to manually setting them back up again, purging all historic stored data, repulling everying. Which was a fucking ball-ache.

[Quade]

"No servers" can also happen if no server is set to download headers but you ask Newsbin to download headers.

Where they listed? I wonder if this is another "Name too short" issue?

[kenr]

I can confirm that using a regex in the search (in Watch topic) causes it to blank and download everything.

[drez]

I updated and all this version seems to do is download the par files, is there something i need to click to get it to work right when i manually drag .nzb files to the download payne?
rolled back to b7 and it works fine

[Quade]

Are these encrypted NZB's?

[drez]

I dont know i'll just send you the two i tried before i went back to b7
you can tell me what im doing wrong with b8

[drez]

Ok, i sent two of them to you to look at

[Quade]

Thanks

Edit:

I figured it out. These have a special obscured encoding I apparently broke.

subject=""36f1908a24daa6a5c39fe9396de6ebfe.10" yEnc (1/82)"

B8 doesn't like the subject. I need to figure out a solution.

[drez]

okie dokie, take your time, b7 still works just fine fore my small needs
thanks for being a awesome dev and looking at stuff when people speak up, i mean it's been 20 years since i bought this and you *always* are on the mark

[Moondawgie]

At risk of publicly exposing my possible failing of an IQ test, what is the option "Extra Special IRC option" displayed as a part of the installation of Newsbin Pro version 6.91 Beta - Build 5775?

I'm hoping that enabling that option leads to my winning of a free 7 day trip to Hawaii, including airfare and lodging at the Four Seasons Resort Maui at Wailea, Hawaii.

[Quade]

It's a shortcut through the installer I believe. It just slaps the files into place and then exits the installer.

It's something Dex added to the installer. I believe not checking it just runs you through the regular installer.

I was just polishing up the B9 beta just now. I'll put it on IRC today and if nothing bad shows up post it here in a day or two.

[ccarlin]

Hi,

With B8 I'm getting files marked as "queued for download" and they never download. Data points are:

1. If I click on a group in the groups list it updates as expected. New items are in green.
2. If I click on a new green binary/item to download it - it gets stuck with "queued for download" and never downloads.
3. If I select a text file and select "read post body" nothing happens
4. #2 and #3 also happens with an item I've done a search for.
5. If I select one of the queued files that is not downloading and do "create NZB from posts" the file is created. If I then click on that .nzb file (in windows explorer for instance) it downloads successfully.

If I switch to B7 w/o any changes none of the above happens - everything works as expected. If I then switch back to B8 (just the newsbin.exe binary) the problems occur 100% of the time.

Thanks, Chris.

[Quade]

The header DB changed between B7 and B8 to support files with > 65,000 posts. It should be forward compatible but not backwards compatible. Meaning headers downloaded with B8 probably won't work with B7.

What you're reporting is the opposite of that but, I expect it's related. I had a B9 ready but someone in IRC noticed a problem so I delayed the release to fix it. It was a pretty minor issue.

I'll try to see if there's some way I can reproduce your issue. Ultimately going forward will be a one way trip because of the changes to the header DB

From the release notes:

6.91 B8 5739:
Display: Report that really large files have their sizes misrepresented because the chunk counter seems to top out at 65K in the display.
120,000 posts in the file. Extensive changes to make it work.

[stavros]

Hi Quade,

will the header DB change apply to both Storage.db3 and StorageData.db3?

I still have 1000's of GB of data spread accross both sets of files (total of more than 2.6TB spool).

regards
Stavros.

[Quade]

will the header DB change apply to both Storage.db3 and StorageData.db3?

1 - StorageData is really in legacy support. Meaning I still support it but new groups won't generate these files. I don't expect the new stuff will have any impact on StorageData. In fact I believe StorageData should have stopped growing some time ago and the data that used to go there is now in Storage.db3. Newsbin checks storage and if the records aren't there it looks for a "StorageData" to see if the data is there.

2 - The change doesn't break the old headers. You just can't go backwards. Meaning headers downloaded with B8 can't be used in B7 but all the existing header created before B8 still work.

I probably need to verify that StorageData still works with a Unit test.

[stavros]

Thanks Quade.

I seem to recall that some time back a header upgrade was done and that as you say, StorageData was moved to Storage,
but in my case it seems that a lot of data is still in StorageData (well over a TB). Most of my groups have both files with data in them,
although some have only Storage, so I assume that they are either newly added groups or ones that have had their data migrated by an earleir
NB upgrade process.

Is there some way to force migrate the data from StorageData to Storage, to future proof / make it easier for future upgrades?

regards
Stavros.

[Quade]

Using Sqlite tools you should be able to integrate the data from "StorageData" into storage. I believe the table names are the same. It's just a matter of opening both and doing a query that takes the data from StorageData and inserts it into the table in "Storage".

There are no plans for any breaking changes or removing read-support from StorageData.

I'd do a backup of each group before you attempt this.

[stavros]

ok, thanks Quade.

I'll give it a go and see what happens

[ccarlin]

Quade,

Maybe an important additional data point -- it appears (I'm not 100% sure) that with B8 running items/headers downloaded with B7 or earlier are the items that won't download. If an item/header was downloaded with B8 it appears to download as expected.

Thanks, Chris.

The header DB changed between B7 and B8 to support files with > 65,000 posts. It should be forward compatible but not backwards compatible. Meaning headers downloaded with B8 probably won't work with B7.

What you're reporting is the opposite of that but, I expect it's related. I had a B9 ready but someone in IRC noticed a problem so I delayed the release to fix it. It was a pretty minor issue.

I'll try to see if there's some way I can reproduce your issue. Ultimately going forward will be a one way trip because of the changes to the header DB

From the release notes:

6.91 B8 5739:
Display: Report that really large files have their sizes misrepresented because the chunk counter seems to top out at 65K in the display.
120,000 posts in the file. Extensive changes to make it work.