Duplicates not working v6.73

[RangerXus]

I'm a long time user of NewsBin. I have been running v6.62 for quite a while and updated to v6.73 to get the filter by poster capability and block downloads in groups.
But I have noticed a problem with duplicates.

I have always downloaded duplicates. I have the "Use Duplicate Detector" unchecked, "Folder Dup Bypass" checked, "Auto Rename" checked, "Copy Style" unchecked.
In the past if I downloaded the same jpg (for example) that was in multiple groups (each group has it's own folder) then I would get a jpg in each folder. Also if I downloaded (bypass filters) the file again in the same group/folder it would be downloaded with suffix 001, 002, etc. and the jpg would show in the thumbnail viewer. If different posts in a group had the same filename/sig then it would be downloaded with the suffix 001, 002, etc as well.

Under v6.73:
if I download the jpg a second time using bypass filters it does not show up in the folder as a 001 and does not display in thumbnail viewer.
It does download one time for each group/folder but again, if I download bypass the file again it does not show up.
I know signature checking is not being stored because my signature db is just a few KB and the date of the file is very old.

As a test I deleted the jpg from the group folder and downloaded it and it did appear in the folder.
I then deleted the jpg, renamed a different jpg to that file name (the file had a different size) and downloaded the file. It did show up in the folder as a 001 file and displayed in the thumbnail viewer. I did this multiple times and every time it was downloaded with the next suffix.

So NewsBin seems to be checking if the file exists already in that folder and if it is the same size (which is not a reliable test for duplicates) or maybe is doing a sig check of the actual file on disk and comparing it to the new downloaded file and if it is the same does not do the duplicate rename... sort of as if the "Use Duplicate Detector" was active. Again, I know my sig file is not being used as it is not growing and has an old date.

For my tests I used the same post over and over. I don't know what would happen if different posts in the same group/folder had the same filename/sig... would the download occur with 001 or not. In the past it would have been downloaded with the suffix 001, 002, etc.

This has to be a bug.

Thanks

-------------

Quade, I just saw this post...

Disable duplicate file detection?
Is it possible to disable the duplicate file detection?
I'm constantly getting false positives.
VERSION : 6.72: 00 A8 C7 15 FE 9C
bevo n00b
Unread postby Quade » Wed Aug 23, 2017 10:11 pm
Yes, It's in the advanced options.
Keep in mind if Newsbin detects the same filename on disk and either it's identical or the first whole chunk is identical, it won't make a new file with the same name.
END POST

Quade, Concerning your last sentence I don't remember it working that way if Duplicate Detection is turn off... especially if you use Download Bypass Filters. So is that the problem... Bypass Filters is not working or has been changed in how it works? Actually I don't have a problem if that is the case as long as the thumbnail would be displayed of the existing file. I am wanting to see the thumbnail... I don't care if it downloads again or not. Of course we both know this only works for the first file saved with filename. If the filename is used by another file with a different sig it would have been saved with a suffix so NewsBin would not know that and in those cases the file would always download with a new suffix. Right?

[Quade]

Quade, Concerning your last sentence I don't remember it working that way if Duplicate Detection is turn off...

True, it's different. If the same file and same data already exist, it doesn't land it again. Bypass filters or no bypass filters.

I don't care if it downloads again or not. Of course we both know this only works for the first file saved with filename. If the filename is used by another file with a different sig it would have been saved with a suffix so NewsBin would not know that and in those cases the file would always download with a new suffix. Right?

If the filename is different, but same content, it saves.
If the filename is the same but the contents are different, it saves.
If the filename and content are the same, it doesn't save.

I'll think about your "show the thumbnail either way" request.

[RangerXus]

Quade,
Thanks for replying.
Yes, I think the change in Download Bypass Filters must have occurred sometime between version 5 and 6. I went back to v6.62 and it did not download duplicates doing Download Bypass Filters. Actually thinking about it, I did notice in v6.62 that I could not get duplicates with Bypass Filters but I always thought I was doing something wrong and never got around to researching it further. It was and is frustrating not having that ability like prior versions.

I don't know the reason why you changed Download Bypass Filters so it no longer creates duplicates, but to me that was always a logical way to force duplicate downloads. If I don't have the Duplicated Detector turned on then it means I want the ability to get duplicates. The new behavior means you will get duplicates in some cases and in others you won't and don't have the ability to get them.

In the previous behavior it was okay that doing a normal Download of a post/file would not created a duplicate of the file in the destination folder if it already existed. But then you could do a Download Bypass Filters on the post/file and it would override the default behavior.

The way I have used NewsBinPro for years is impacted by this changed behavior and makes it a hassle when I can't force a duplicate download for whatever reason. Especially considering I have told NewsbinPro that I do not want the Duplicate Detector active.

And if you think about it, the new behavior is inconsistent. If I download a post containing File-Test size 1mb it will be saved as File-Test. If I download another post containing File-Test size 2mb, the 2mb file will be saved as File-Test-001. If I Download Bypass Filters File-Test 1mb again I won't get it because the name matches. But if I download File-Test 2mb again then I will get it as File-Test-002 because it was originally saved as File-Test-001 so the name does not match. So sometime I can get a duplicate and sometimes I can't... it just depends on if the file was the first one downloaded with that name or the second, third, etc.

I hope you will restore the original functionality of Download Bypass Filters to always download duplicates (if Duplicate Detector is turned off). If there is a reason some don't want that behavior them could you make it a selectable option? I don't care if there is a GUI option or just a hidden option in the .nbi file... I don't mind editing the .nbi and adding a line to turn on the old behavior.

If you can't give us the ability to have the old behavior of Download Bypass Filters to download duplicates then at least displaying the existing file in the thumbnail viewer would be a compromise... but not as good as having the old behavior of Download Bypass Filters.

I hope you will consider my request. I have been a long time user and supporter of your product. It is the best out there and full functioned. But it is these less obvious type issues that can make using NewsBinPro complicated by having to do manual work-arounds to do some basic things.

Thanks

[BZee]

I would like the option to download duplicates even if the file already exist. When downloading multiple part pictures near the edge of retention sometimes one or more pictures in a set will be missing or be missing a part. For those missing a part I use "Assemble Incompletes" to save it. If another poster has posted the same set, I'll download it to fill in missing pictures but Newsbin will skip replacing corrupt pictures with good ones if the first parts of both pictures match - which is usually the case as I skip using "Assemble Incompletes" if the picture's 1st part is missing (often no pars were posted).

[RangerXus]

Hi Quade,
Have you addressed this request to download duplicates or at least display the thumbnail if already exists. I don't see anything listed in the v7.80 betas about this.
Thanks

[BZee]

This was added and has really helped me when downloading photos on the edge of retention.

[RangerXus]

Quade, want to thank you for addressing and implementing my requests

In version 6.80 RC5 the following now works:

- By default if a jpg has already been downloaded to a folder and I tell Newsbin to download it again, the download does not occur (no duplicate) but the jpg appears in the thumbnail viewer.
- Using the new BypassFileCheck=1 option results in files being downloaded again with the duplicate naming format.

(I will post this on the other related thread as well for completeness when others browse these threads.)

Great! Thanks very much!