Newsbin Beta 6.90RC5 - build 5485

[Quade]

http://downloads.newsbin.com/downloads/ ... 5-5495.exe

Code: Select all

Base: If the data folder in the NBI is missing the trailing slash, the folder use is one above the desired folder.

Display: Files list - Download Descriptions and poster don’t sort.
Display: Changed watch lists to obey “display age”.

AutoPAR: Obscured rename - Files don’t have extensions now.  Scan files and add proper extensions based on file type.
AutoPAR: Look in subject for rename topic when downloading obscured.
AutoPAR: Certain NZB’s contain download that are 3 way encoded.  RAR/ZIP then RAR again.
AutoPAR: Fixed a password problem with QPMS 7zip file which need passwords.
AutoPAR: Default pause during decode to don’t pause during decode.

NZB: Make sure samples don’t get mixed in when doing forced compaction. Do I really need forced compaction?
NZB: Put filepath validator in Remote NZB Decode Path because there’s a chance it’s only a partial path.
NZB: Report that download from post list with NZB doesn’t use NZB Download folder.
NZB: If not decode folder is set, make sure the decode folder used is the NZB Download folder IF the NZB Download folder is used.  Apparently it uses the main decode folder instead.
NZB: Noted a case where a sample in the NZB but there’s no sample in the filename.

Download: Some PAR only sets linger in the download list even when they finish downloading.
When the final PAR downloads, send a notification to unpause more pars (in case they aren’t combined) but, always move the set to failed because the unrar didn’t complete.

Download: Confirmed using root path for download path won’t work.
Download: Report that apparent incompletes cause crashing.  Black lines in the progress bar anyway. No repo
Download: More new NZB Formats added which didn’t compact which resulted in some left over PAR files.
Download: Changed pathing again to make it make more sense.
- If no decode path is set, download and decode path are the same. Specific to headers and/or NZB Files.
- Autoload is the same with autoload NZB download/decode path overrides.

Sonarr: Remove “Cat tv;” from NZB filename stored in T download db. It interfered with obscured rename function
Sonarr: It’s possible if there’s a sample with identical name to the actual downloaded file for the names to collide. Now largest media files is stores in the succeeded list, sonarr uses this list to pick up downloaded files.
Sonarr: Default unrar path in sonarr options is bad
Sonarr: Sonarr uses category to ID which NZB's belong to it. The old mechanism for storing it wasn't reliable enough. New mechanism.

Posting: Report that posting doesn’t work to easynews but 6.82 works, single server issue? Report is that any text only posting fails. Fixed - Posted chunk was lost between adding the post, and posting the post.



[jcthorne]

Thanks for the updates. DL and running now. Will report if I see anything odd.

[Quade]

Thanks!

[Stan]

I have access to 5 different news services and they all have posting and all posts went through. Unfortunately, none are easynews.
Newshosting, usenetserver, usnetfarm, vipernews, and at least one other. These all work now.
Until this release, none would work unless it was version 6.82.

Good luck with the continued work and will look forward to a final version sometime in the future.

[DogSnake]

Posting is working for me also (Block News user)Thanks

[Quade]

It's pretty much done. Sonarr is working great and the NZBGeek NZB's compact correctly. I'm probably going to let it perk for a couple weeks then release if nothing bad shows up.

[pdf256]

Are you going to link it on the beta page?

pdf256

[stavros]

Hi Quade,

thanks for the new beta. I'll be trying it out in a day or two.

Will the final version run on 32 bit windows?

regards
Stavros.

[Quade]

Are you going to link it on the beta page?

Hopefully eventually. It depends on whether forum users find problems or not.

[mimauk]

With both the previous RC5 and this new one, whenever I use a nzb file from one of the indexer sites, the file will not download to the folder set up in the nzb path in options.

My download folder is on a seperate HDD D:\newsbin and the nzb download folder is D:\newsbin\nzb where all the nzb files from the indexers used to download to.

When the nzb file is downloading in real time, the download folder in the Download List tab shows D:\newsbin and at the same time the download folder in the File List tab shows D:\newsbin\nzb. The file downloads to D:\newsbin.

If I use D:\newsbin\$(GROUP)\ the file will make a folder whose name it takes from the actual nzb file - eg. a.b.misc or a.b.movie repost.

[Quade]

How are you feeding the NZB?

I specifically tested to make sure it downloads to the NZB folder. If you don't have an unrar folder set, it'll unrar to the same folder it downloads to.

[mimauk]

Before RC5 I would open an indexer site in my Browser (Firefox), pick a file I wanted to download and click on the nzb icon for that particular file. A popup would appear and ask me what I wanted to do with the nzb either open with -NewsbinPro newsgroup reader (default app for nzb files) - or to save the file. I would choose NBPro and the download would appear in the Download Tab and start to download to D:\newsbin\nzb. I also tried and saved a nzb file and dragged it to the download list with the same result.

My unrar folder is set to D:\newsbin\$(GROUP)\

You mentioned in your reply an nzb folder which I have never seen before. Checking in Options, I saw 'Nzb Options' with an nzb watch folder option - is this the folder you were referring to? I set up a folder here D:\newsbin\nzb\ and tried another indexer download using my usual method but it made no difference.

[Quade]

In the main options there are two download paths. One for NZB's and one for regular downloads. If you fill in the one for NZB's, all NZB files you drag and drop or open the way you do will download to that folder. Then they'll unrar to the unrar folder. If there's no NZB download path, it defaults to the main download path.

If you add $(NZBFILE) to the path, either the unrar path or the download path, it'll use the NZB FIlename as part of the path. If you want the unrar folder to include the NZB Filename, they you'd add $(NZBFILE) to it.

For example:

D:\newsbin\$(GROUP)\$(NZBFILE)\

[mimauk]

OK thanks Quade I'll make some adjustments and see what transpires.

Since the initial RC5 wiped my nbi file and I had to do a complete fresh install because of a missing backslash, I'm presuming that my old setup had something in it that allowed me to download indexer nzbs to the folders I had set up - maybe even something to do with that missing backslash. We'll never know. I think the last time i did a fresh install was a couple of years ago and it has been every beta upgrade since then, so something could have been left behind from one of the upgrades.

Thanks again.

[Quade]

Since the initial RC5 wiped my nbi file and I had to do a complete fresh install because of a missing backslash

It wasn't wiped. It was just sitting in the old data folder. If you don't run too Newsbin too many times, you can recover the NBI from a backup it keeps. In the main menu "Options/Load Backup Configuration". I think it keeps 10 runs worth of NBI's in there. Each time you run Newsbin one of the backup slots gets over-written.

You should probably keep a manual backup of the NBI too.

[alexon]

Hello Quade.
Something strange has happened. Maybe you can help me.
Until today, the latest RC5 version of NB worked flawlessly.
Today I received same Windows 10 updates:
Installation completed: The following update was successfully installed: 9WZDNCRD1HKW-Microsoft.xboxidentityprovider
Installation completed: The following update was successfully installed: 9NFFX4SZZ23L-Microsoft.549981C3F5F10
Installation completed: The following update was successfully installed: Cumulative update for Windows 10 Version 20H2 for systems based on x64 processors, 2021 11 (KB5007186)

Now when trying to load the headers of any groups, with the number of new messages about 4000, NB crashes at the end of the header list creation process.
At this point, it looks like synchronization with the database is taking place.
(The blue bar showing the loading of titles at this moment reaches the right border of the window)
The Data folder is located on a separate SSD drive. This folder is many years old and it is valuable to me.
The load on the CPU at this moment sharply increases for a couple of seconds, after which the program window collapses.
There are no error messages in the NB logs.
The following message about the program crash appears in the Windows logs.
----------------------
- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider Name="Application Error" />
<EventID Qualifiers="0">1000</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>100</Task>
<Opcode>0</Opcode>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2021-11-19T18:24:11.2828781Z" />
<EventRecordID>31970</EventRecordID>
<Correlation />
<Execution ProcessID="0" ThreadID="0" />
<Channel>Application</Channel>
<Computer>DESKTOP-*******</Computer>
<Security />
</System>
- <EventData>
<Data>NewsbinPro64.exe</Data>
<Data>6.8.0.0</Data>
<Data>61908942</Data>
<Data>NewsbinPro64.exe</Data>
<Data>6.8.0.0</Data>
<Data>61908942</Data>
<Data>c0000409</Data>
<Data>00000000007fbdd9</Data>
<Data>225c</Data>
<Data>01d7dd7260aabbbf</Data>
<Data>C:\Program Files\Newsbin\NewsbinPro64.exe</Data>
<Data>C:\Program Files\Newsbin\NewsbinPro64.exe</Data>
<Data>298f0155-ad61-4b48-b54d-c70a86b093ef</Data>
<Data />
<Data />
</EventData>
</Event>

----------------------
After returning to NB release 682, the program works again without problems.

My system config:
Windows 10 Pro
Version 20H2
Build 19042.1348
Windows Feature Experience Pack 120.2212.3920.0
Intel(R) Core(TM) i9-10920X CPU @ 3.50GHz 3.50 GHz
64 GB RAM

[Quade]

What's your "display age" set to?

I might try a right click "Show Posts Special" and try loading 10 days worth and see what happens

[alexon]

Thank You fo reply Quade.

the display age is set to 5000 days, but this parameter has been set this way for 8 years and there have been no problems, including with RC5.
The problems started after installing Windows 21H1, which appeared in my update center only a few days ago.
After that, I had problems with other programs, and I deleted this update.
Everything worked as it should.
But Windows continues to be updated despite the ban. Every day in the system stability monitor I find 3~4 hidden updates of Windows embedded components.
In addition, it looks like the antivirus has started a large-scale update.
The problem is clearly not NB. I'll wait until this crazy stream of updates of everything stops.
Perhaps the problem will solved itself, as it has already before.

[Quade]

Crashing in Newsbin is always a bug in Newsbin. It should never crash no matter what. So, I'm looking at it from that perspective. Even if something external messes with Newsbin's files, it shouldn't crash.

You know old records in your header DB's never change so, you might want to change the "Display Age" to something short like 30 days so, you quickly load the latest headers to view. You can always load all 5000 days if you need to find some old files. Using a short display age will vastly speed up loading and reduce RAM usage. Things like "finding" will be much faster too.

[alexon]

Thank you Quade!
I will definitely try to follow your recommendations.
Unfortunately, I cannot fully follow your recommendations on limiting "Display Age", since my specifics of work require periodic return to messages created several years ago.
I also have to say: I deeply respect You for such a responsible approach to your work. It's rare now.
Added:
I noticed the following: the set of groups to which I am subscribed has not changed for many years.
The size of the spool_v6 folder located on the NVMe SSD drive is approximately 11 GB.
The size of folders inside it varies from 1 MB to 900 MB.
In version 682, loading headers from the database was much slower than in RC5.
In both versions 6.82 and RC5, at the end of the process of creating a list of headers, the load on the CPU increases dramatically (the processor liquid cooling system howls).
But in the case of 6.82, the list of headers is formed successfully, but in the case of RC5, the program collapses without even leaving entries in the logs.
Maybe some software does not have time to process such a large data stream coming from NVMe SSD? It can be an antivirus or NB or a Windows indexer or something else.(maybe I'm wrong)
I think so, because even if there are no new messages in the binary group, but it has, for example, 600,000 messages, then RC5 crashes in the process of creating a list of headers,
but version 682, although slower, correctly creates a list of headers.
Another example: a binary group has 300,000 messages.
Process of creating the list of headers, in both RC5 and 6.82 works perfectly.

Added:
Your recommendations helped.
Set the NB display age to 3000 days. Everything works. Then I set the value to 3500 days - the NB crashes.
In version 6.82, I can read messages from 2008 and earlier.

[Quade]

Maybe some software does not have time to process such a large data stream coming from NVMe SSD? It can be an antivirus or NB or a Windows indexer or something else.(maybe I'm wrong)

There's a file called "DownloadMarker.db3" which maintains the "Old/New" status for every file downloaded. After you load the data it's compared to this database and the proper tags are applied to the files. The blast of CPU you're talking about is either referencing "DownloadMarker" or just the process of displaying 5000 days worth of files. The data coming off the SSD comes off in lock-step so, it's not possible for it to be too fast. I'm more suspecting some sort of corruption during the read process. Or maybe "DownloadMarker.db3" is corrupted in a way that 6.90 doesn't like but 6.82 shrugs off.

You could open the data folder from the options menu. Exit Newsbin, then move "DownloadMarkers.db3" to some other folder, restart Newsbin and see if that changes the symptoms.

[alexon]

Thank you so much Quade !!!
I did as you recommended. Moved the Download Master.db3 file to another folder, restarted NB, set the Display Age value back to 5000. And now everything is working fine again!
Thanks again for the real support!

[alexon]

I was overjoyed too soon.
NB worked without problems for 7 hours and suddenly, as before, collapsed just while downloading a video file.
As before, there are no errors in the logs.
In the Windows logs, as before, a report about the NB crash.

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider Name="Application Error" />
<EventID Qualifiers="0">1000</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>100</Task>
<Opcode>0</Opcode>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2021-11-21T03:55:14.1534129Z" />
<EventRecordID>32176</EventRecordID>
<Correlation />
<Execution ProcessID="0" ThreadID="0" />
<Channel>Application</Channel>
<Computer>DESKTOP-******</Computer>
<Security />
</System>
- <EventData>
<Data>NewsbinPro64.exe</Data>
<Data>6.8.0.0</Data>
<Data>61908942</Data>
<Data>NewsbinPro64.exe</Data>
<Data>6.8.0.0</Data>
<Data>61908942</Data>
<Data>c0000409</Data>
<Data>00000000007fbdd9</Data>
<Data>29e0</Data>
<Data>01d7de538ae19947</Data>
<Data>C:\Program Files\Newsbin\NewsbinPro64.exe</Data>
<Data>C:\Program Files\Newsbin\NewsbinPro64.exe</Data>
<Data>72a28847-3146-440c-9760-96ddd17e2b5c</Data>
<Data />
<Data />
</EventData>
</Event>

[JayPea]

Heya Quade.

I'm observing some application lock ups when sending large numbers (4-5k) of smaller files to the download window. I've monitored my system performance whilst this is happening and neither my cpu nor my disk activity get higher than 40% yet Newsbin starts show as 'Not responding' and then nothing can be operated, in-app, whilst it sorts itself out (approx 30s wait). Every other part of my system continues to function normally. Newsbin's Main download folder and Download folder for downloads from NZB files is set to it's own SSD, exclusive for NB access and my Unrar folder is set to another SSD. Both on the SATA3 Bus of a 32GB i7 8700K @ 4.8GHz.

Hopefully something that can be rectified.

Cheers.

[Quade]

What kind of drive is the data folder on?

What kind of files are they? Images, MP3's or large rar sets?

[JayPea]

What kind of drive is the data folder on?

What kind of files are they? Images, MP3's or large rar sets?

Data Folder is on a Kingston SH103S240G (240GB SSD) & Unrar folder is on a CT1000MX500SSD1 (1TB SSD)

images, nfo files, nzb files, they all get caught up. Generally block selecting when browsing a group.

[meenich]

After upgrading to RC5 Build 5495, I'm having problems with searching. I have a search subscription, and when I enter search terms, it goes off for a few moments with the status bar indicating it's working. But then the status bar stops updating shortly before completion, and no search results ever show up. I did not have any search problems with RC4.

Has anyone else had problems with searching?

Thanks.

[Quade]

Search is working here.

1 - From the options "Open data folder" then go into "Spool_V6" and delete "BASIC_SEARCH".

2 - After a search, the right most item in the top bar should show a count like "0/32". This is a new field that shows the visible and filtered counts. If your visible count is zero but the second shows a number, it means you have some filter set that's hiding the results.

Try this and let me know.

[meenich]

[quote="Quade"]Search is working here.
1 - From the options "Open data folder" then go into "Spool_V6" and delete "BASIC_SEARCH".
[/quote]

No filters, but deleting the BASIC_SEARCH folder as you suggested fixed it - thanks!

[meenich]

I spoke too early - search problems are still persisting. And I have no filters in place to the best of my knowledge. As I mentioned in my previous post - the search never actually completes, the progress bar gets stuck before it completes and then it just sits there.

I've also deleted the BASIC_SEARCH directory again, but this time no luck.

It's not failing for all searches - sometimes it completes, sometimes it doesn't.

As an experiment, I search for a term, found that it succeeded. The I search again for the same term with a few letters appended. Based on the previous successful search results, it should have returned a smaller non-empty set of results than the previous search, but instead got stuck and never completed. The two terms I used in this "All Groups" search are "stand" followed by a search for "standoff". The first search returns 1976 results. Filtering those search results for "standoff" finds 16 results within the 1976. Directly searching for "standoff" gets stuck and never completes (progress bar at bottom of search window/pane gets stuck at somewhere around 95% or so (estimating).

[Quade]

I got about the same number of results when I did either "stand" or "standoff" 1016/1018.

How about opening data folder again, exiting Newsbin, then from that folder move "DownloadMarker..." files to another folder then restart Newsbin and try again. There have been a couple of odd reports with "downloadmarker".

How about that field I suggested you look at? What does it show?

[meenich]

Thanks, Quade!

Moving the DownloadMarker files to another location seems to take care of things. A few random searches seem to all be working now.

I'll try to beat up on it a bit more and let you know if any further issues come up.

The filter fields says 1061/1067.

[alexon]

Hi Quade.
Since we are talking about "DownloadMarker" again, I will still ask the question that I deleted, deciding that the problem is gone.
A few posts above, I described the problem of NB crashes that occurred at the header loading stage.
I was also asked to delete the "DownloadMarkers.db3". I deleted the file "DownloadMarker.db3"
This solved my problem for a short time, after which the problem returned,
but in a slightly different form: the program began to fall not in the process of downloading headers, but at any arbitrary time (for example, in the middle of downloading a media file).
Having not received advice on solving a new problem, I chose the easy way and returned to version 6.82.
However, now I began to doubt whether I had followed the Quade's instructions correctly.
So I will repeat my question: Which "DownloadMarker.db3" file should I have deleted?

In the "Data Folder" I have three files "DownloadMarker.db3"

DownloadMarker.db3
DownloadMarker.DB3-shm
DownloadMarker.db3-wal

Only the first or all?

[Quade]

I'd kill all three.

I'm not having any issues with marker so, I'm sort of working blind. I am adding some more tests to it. It sounds like it's been corrupted but that shouldn't cause any crashing. It should just stop working.

So far the only issues seem related to marker so, I'm looking at it.

[alexon]

Thank you Quade.
It seems to me that the problem is still in my data folder. It's very old. The folder has survived the change of several computers and several operating systems, not to mention the change of dozens of beta versions of the NB.
In addition, the servers were changed several times.

Nevertheless, all this time your wonderful program has been working without problems. The only serious database-related failure was observed after the update from version 6.82 to the first beta of 6.90.
The failure occurred during the conversion of databases to a new format. The conversion was not completed.

The Newsbin.nbi file, and the entire data folder then looked monstrous. Duplicates of folders appeared, duplicates of entries in the Newsbin.nbi file appeared.

And although after returning to 6.82, the program's performance was restored, it is possible that somewhere in the databases there are records that RC5 does not like, as You say.
I need to try to do a clean install of the NB. Maybe I can live normally without a 13-year database.
Moreover, yusenet has changed a lot during this time.
In the meantime, I will follow your advice and will definitely write if I see something unusual.

[kenr]

Weird one
NB is running under wine on a separate disk from the download folder
When repairing a set it was complaining that it was out of disk space. The wine disk had 1Gb free and the d/l folder has 1.67Tb all the files were in the download folder so why was it trying to use the wine space?

[Quade]

It shows you what it thinks is the free space down on the status bar.

What's the actual error message?

[kenr]

It was showing 1.04Gb in red and pausing the repair.
No error message

[Bismark]

It appears the find command for the file list tab does not search the filename column.

Also, I the header on the program says it is build 5495, not 5485. It does have the correct Beta, i.e. 6.90RC5

[Quade]

I'll check it out. Thanks. It seems to be working here but, maybe it's searching on descriptions.

[Bismark]

search finds the search terms in the download path or the descriptors, but not in the file name column. is there a way I can change where it looks?

[Stan]

I seem to recall reading this before but can't find it. In the watch list marking a post as old with hide old posts on it does hide the post but when I shut down and re-open 6.90RC5 the post I set as mark old shows again. I did it 3 times, the same thing. Went back to 6.82 and did the mark old and then again put back 6.90RC5 and the post now stays as hidden.
So with that seems to still be some issue or maybe some new setting not right in 6.90RC5.
again not sure where I read it before or what RC it was or if there is something different now.

[Quade]

I'll give it a try.

[mednzb]

Before you officially release 6.90 can you please fix the percentage on repair. Every time 6.90 does a repair it just shows 100% I have no idea how long it's really going to take.

[JayPea]

Right-click/Reply to Group (without attachment) is shafted again on my client.

[Quade]

Every time 6.90 does a repair it just shows 100% I have no idea how long it's really going to take.

Sure I added it to the list. Thanks.

[Quade]

Right-click/Reply to Group (without attachment) is shafted again on my client.

I'll check thanks. I think it'll need another RC. Maybe tomorrow.

[meenich]

I'm seeing a problem with file renaming on some downloads (it seems to be limited to ones ending in -4P, for some reason). The directory/folder is created appropriately and with the correct name of the NZB file, but the unrared content file is left inside the folder with a garbage name - usually 4 random letters with an mkv extension.

These releases seemed to be getting treated correctly in previous RCs.

Thanks!

[Quade]

If you email me some of the NZB's I can try them out. Are you downloading and unraring to the same folder?

ts @ Newsbin dot com

[meenich]

[quote="Quade"]If you email me some of the NZB's I can try them out. Are you downloading and unraring to the same folder?

ts @ Newsbin dot com[/quote]

My settings for
Main Download folder: B:\Usenet\Chunks\
Download folder for downloads from NZB files: B:\Usenet\$(NZBFILE)\
Unrar folder: B:\Usenet\$(NZBFILE)\

Sent. Thanks!