Downloaded Files Show as New

[ppan]

I recently started to use Newsbin v6.41 again, and just now upgraded to 6.71RC3.
I tried to download a few files:
1. Even though the files were downloaded, the Status shows as "Queued for Download"!? Shouldn't it say "Downloaded"?
2. If close and reopen Newsbin, the status for the downloaded files shows as "Idle/New" (and with green background). Shouldn't it say "Downloaded", or maybe "Idle/Old"?

[Quade]

How are you downloading? Search, Headers or NZB Files?

[ppan]

How are you downloading? Search, Headers or NZB Files?

I downloaded the headers and then selected the headers to download.
Note that I didn't have this issue with v6.41

[Quade]

The whole mechanism for tracking downloads is different in 6.7X so the fact it worked in older versions is sort of a moot point. 6.7x+ track downloads from search, headers and NZB's. That's why the mechanism had to change.

In the options select "Open Data Folder". Then exit Newsbin. Delete the file "DownloadMarker.db3". Then start Newsbin and try some more downloads. The download marker file is what tracks downloads.

[ppan]

I deleted DownloadMarker.DB3 (and a new one was created after download)

However, it didn't help much:
Right after download, the status showed as "Queued for Download" (and the downloaded files were shown in the File List)
After exiting/reopening Newsbin the status for the downloaded files showed as "Idle/New" (and the downloaded file(s) were still shown in the File List)


p.s. With a few files it worked properly (not sure what is so special about those):
Right after after download, the status showed as downloaded (and the downloaded file(s) showed in the File List)
After exiting/reopening Newsbin it still showed as downloaded.

[ppan]

I tried to downgrade to v6.70, but it didn't help. Same issue.

I then downgraded to v6.62 and it worked a little better:
Right after download, the status showed as "Queued for Download" (and the downloaded files were shown in the File List)
After exiting/reopening Newsbin it showed as "Idle/Old"

Several questions:
1. Shouldn't it show as "Downloaded" instead of "Queued for Download" after the download is completed?
2. Should it show as "Idle/Old" or "Downloaded" after exiting/reopening Newsbin? Maybe it depends on the Newsbin version?

3. Even though I had deleted the DownloadMarker.DB3 file, files downloaded with v6.41 still showed as "Idle/Old". I thought that the download status was stored in the DownloadMarker.DB3 file?

[Quade]

1 - Yes.

2 - Downloaded.

3 - As I said, the mechanism changed between them so, what you do in 6.70 may have no connection to what you do in 6.4.

I'm honestly puzzled. You seem to say it sometimes works but I'm not sure how it could work some of the time. It implies that download marker can't be written to.

There's a fix in the latest version for "marking downloaded" in search based downloads which is why I was wondering if you did headers or search. The issue is that what search displays isn't always the same subject as what's downloaded. In search, posts that show a "<comment><subject>" format weren't getting marked downloaded. You're telling me you use headers so this shouldn't apply to you.

I sit here and select 20 image files, add them to the download list, 20 seconds later they're all marked downloaded in the post list.

If you want to PM me an example of something that won't get tagged downloaded, I can try it. Don't post it here.

[ppan]

TEST:
1. I upgraded back to v6.71RC3
2. I created an empty data folder "TEST"
3. I copied my current configuration file, named it TEST.nbi and put it in the TEST data folder
4. I changed the data path in TEST.nbi to the TEST data folder
5. I downloaded all the headers for a group myGroup
6. I displayed old headers using "show Special"
7. I downloaded a few files
8. It worked fine (files marked as downloaded; also when closing/opening Newsbin)

Thus, there is something that needs to be fixed in my original data directory to get this to work (I do not want to start fresh).
Note that when I upgraded from v6.41, I simply let the installer overwrite my v6.41 installation. I didn't bother clearing the Download list etc.

p.s. Before upgrading, I first made copies of the installation and data folders, so those are still available if needed.

[Quade]

I simply let the installer overwrite my v6.41 installation. I didn't bother clearing the Download list etc.

I guess is could be related to this. Maybe the subjects of the files already in the download list don't match what's shown in the group anymore.

You could start Newsbin, Ctrl-A in the download list, then hit the pause button to pause all the existing downloads. Then try downloading new files only and see if all the new files get tagged.

[ppan]

The download list is empty so that will not help.
Maybe I could move and overwrite data from the original data directory to the new working TEST one, one file or folder at a time and see when the problem starts?
Do you recommend any particular order of the files/folders to copy over?

Could you please explain the purpose of each of these files (in v6.71RC3):
Downloaded.db3
DownloadedFiles.txt
DownloadMarker.DB3
DownloadsV2.db3

[ppan]

I copied over the spool files (first deleted current) from my original data directory (used with v6.41) to the new TEST data directory (used with v6.71RC3).
Nothing else was copied.

Now the problem started again; after downloading some files, they still show as "Queued for Download" and Idle/New after restart.
Apparently, the spool directories are not compatible between v6.41 and v6.71RC3 (in contrast to this post: viewtopic.php?f=43&t=40347)?

Thus:
1. I guess that I would need to download the headers again, or can I fix the spool directory somehow?
2. Can I copy over (first deleted current) all the other files from my old to my new data directory, including all the DownloadX.X files questioned above?

[Quade]

Downloaded.db3 - files list.
DownloadedFiles.txt - probably unused at this point.
DownloadMarker.DB3 - Download tracking.
DownloadsV2.db3 - Download list.

If you really think it's a spool file issue, purge and re-download headers from a single not particularly large group and test it. I'm not aware of any spool incompatibilities nor are there other reports of this from other users. Still, all it would take would be as little of a change as an additional space for it to not match up because the downloaded state is determined by the hash of the subject.

I honestly don't know what's happened. Even if the spools are incompatible, new header downloads should be writing in the new way when using 6.71 so, you ought to be be able to determine this by downloading files from both old and new headers.

As for fixing it, you'd have to determine what's different between 6.4 and 6.71 spool files and then write something to re-write all the existing subjects to match the 6.71 way. It's doable. I'm not clear how easy it would be.

[ppan]

It is definitely a spool issue as explained above; To summarize (now using v6.71RC3):
I created a new TEST data folder and downloaded headers for a group, "MyGroup". I didn't have a problem in this case; newly downloaded files showed as Downloaded.

I then replaced the spool file for "MyGroup" with the one used in v6.41, and I had the problem (newly downloaded files showed as Queued for Download). Furthermore, when I used this old spool file, it showed which files had already been downloaded with v6.41 (shown as Idle/Old). Therefore, the problem is with the old spool file when using it with v6.71RC3

I next tried to use the new spool file and copied over all the other files from v6.41 data folder (including all the DownloadX.X files). In this case, newly downloaded files were shown as Idle/Old. I next deleted DownloadMarker.db3, and then the newly downloaded files showed as Downloaded.
However, old files downloaded with v6.41, showed as Idle/New !


Conclusion: If I download new headers there is no way to carry over the download status from v6.41 data folder because the download status was stored in the v6.41 spool file and it is not compatible with 6.71RC3.
Am I missing something or is there some way to copy over the download state from my old v6.41 old data folder while using new spool files?
Maybe there was some version in-between 6.41 and 6.71RC3 that converted the spool file?

p.s. I am subscribed to this thread, but I am no longer receiving notifications (and they are not in SPAM).

[Quade]

Conclusion: If I download new headers there is no way to carry over the download status from v6.41 data folder because the download status was stored in the v6.41 spool file and it is not compatible with 6.71RC3.

This is kinda of a given because of the change in mechanism. Reading your original post, I thought you were reporting that "downloaded" state wasn't working for some new posts in 6.71. That's all I've been addressing here.


While it is possible to find and move forward the downloaded state, it would require writing a program to do it. To scan though all the spools and at old marked downloaded posts to the new database. If could tell you how it would need to be done if you want to try.

1 - Look at the flags field
2 - If a post is marked downloaded, note the subject (I'd have to see if the subject is doctored or not).
3 - MD5 hash the subject
4 - Feed it to the "DownloadMarker.db3"

These are the basic steps.

[ppan]

Conclusion: If I download new headers there is no way to carry over the download status from v6.41 data folder because the download status was stored in the v6.41 spool file and it is not compatible with 6.71RC3.

This is kinda of a given because of the change in mechanism. Reading your original post, I thought you were reporting that "downloaded" state wasn't working for some new posts in 6.71. That's all I've been addressing here.

Sorry if I wasn't clear. Thre were two issues:
1. The download state mechanism is not working for new downloads when using the old spool file from v6.41 becuase they are marked as "Queued for Download" after downloading, and as Idle/New after restart.
2. The "workaround" is to download the headers again. However in this case, the old download state (for v6.41 downloads) is lost.
Do you agree?

That is why I had previously asked if it would be possible to use the old spools when upgrading form v6.41 to v6.71RC3 (see here: viewtopic.php?f=43&t=40347 ). As I have discovered now, it is not possible because new downloads are then not marked properly .

[Quade]

1 - I agree that downloaded state of existing posts in the spools will be lost if you re-download. That's another weakness in the old way that doesn't exist in the new way.

2 - I'll take your word for the 6.41 issue. Could be 6.41 is just too old for me to remember.

That is why I had previously asked if it would be possible to use the old spools when upgrading form v6.41 to v6.71RC3 (see here: viewtopic.php?f=43&t=40347 ). As I have discovered now, it is not possible because new downloads are then not marked properly

You and I just have different ideas about "working". You can download files without re-downloading the headers. That's what I consider to be "working". The duplicate checker will still work even if the subject is different so, there's another level of "downloaded" checking. For me, the fact it doesn't mark old format spools downloaded, just isn't that important.

I can tell it's important to you.

I don't remember a change in spool format but then again, I consider re-downloading to be a problem, so I avoid changes that require re-downloading headers but, I consider the different states to be cosmetic so, they have lower consideration.

[ppan]

1 - I agree that downloaded state of existing posts in the spools will be lost if you re-download. That's another weakness in the old way that doesn't exist in the new way.

For me, the fact it doesn't mark old format spools downloaded, just isn't that important.

I am not sure if I am expressing myself clearly.
The issue is not to mark old format spools as downloaded. The issue is that new downloads, when using the old spool file from v6.41, are marked as "Queued for Download" after downloading, and as Idle/New after restart. Thus, I have no idea of knowing if newly downloaded files have been downloaded or not.

[Quade]

Seems to me you have a couple choices:

1 - Stick with 6.71, ignore the download marking of old spools generated with older newsbin versions and move forward with only new headers downloaded with 6.71 getting marked old when downloaded.

2 - Re-download all headers so, old files get tagged downloaded but you lose the history of already downloaded files (but you don't lose the duplicate detector which is a different mechanism). Likely this only applies to files you haven't added to the download list yet too. Files already in the download list, keep whatever format they had when they were sent to download.

3 - Revert to 6.41 and keep using it from now on.

6.41 is 4 years old. I think you've done a good job of zeroing in on the issue. There's probably just nothing I can do about it.

[ppan]

I think that there is a 4th alternative, and I will try to use this:
Run v6.41 with the old spools simultaneously with v6.71RC3 with newly downloaded headers.
I will use v6.41 to see which files have been downloaded, and then switch to v6.71RC to only download files that had not been downloaded with v6.41. Not ideal, but no other solution if I want to move forward.

Since no one else seem to have this issue, I wonder if there was maybe an automatic conversion of the old spool file format to the new format with some Newsbin version in-between v6.41 and v6.71RC3?

[Quade]

Since no one else seem to have this issue, I wonder if there was maybe an automatic conversion of the old spool file format to the new format with some Newsbin version in-between v6.41 and v6.71RC3?

I think it's more likely people encountered it, shrugged and move on with their downloading, knowing that the problem will solve itself over time. As I pointed out, the difference of a space in the subject is all it would take for the subjects to not match up so, I suspect it simply changed over the 4 years between 6.41 and 6.71.