Filtering out header downloads.

[Quade]

I will investigate the search option - I had no idea that that would let you avoid downloading headers.

You don't need a subscription to test it. You can go into the search tab and if it's set for "Internet" do some searches. I recommend setting the minimum file size to a reasonable number to filter out junk posts. You can also tell search to only return results from specific groups too.

You only need a subscription to download from it. It's useful for seeing what's in the groups even if you do download headers.

[StrangeLove]

I changed my filter name from Spam to HeaderSpam, since I noticed Newsbin had a Spam filter already in the nbi file. I thought maybe there was some possibility of conflict, in naming. Deleted teevee again and tried to download new headers. Set for 1 day, double click caused it to start downloading 8+ million headers for the last 24 hours. So failed again, stopped it and retried with Special 100k, right click command. It still downloaded the spam I'm trying to filter. I even added a second condition;
Reject if Subject \[Private\]
Reject if Poster [email protected]

Still no luck. It also seems to leave the spam in the Storage.db , since it gets too big too quick, afai can see. Also I can look in the top panels and see Unsorted (Filter Profile) HeaderSpam and a.b.teevee (Filter Profile) HeaderSpam. I actually tried adding it to both, in case one had an issue the other might register.

Thanks for any tips or help.

[dexter]

Send us a screenshot of your filter options screen with your "HeaderSpam" filter profile selected and send your .nbi file to ts at newsbin dot com and I'll take a look.

[StrangeLove]

Sent, thanks for your help and work on this great program.

[dexter]

FYI, Version 6.73RC2 now allows you to configure a header download filter from the GUI so you don't have to mess with the .nbi file.

viewtopic.php?f=44&t=41375

New instructions (including pictures!) are in the Newsbin Help Wiki.

[StrangeLove]

Firstly, I did find an older version of Newsbin, pre v6 where the nbi is in the /data folder. I've double checked the install path and there doesn't seem to be anyway for it to be reading off my I: drive. It's certainly not pathed in the .nbi or even this install of windows. So no idea how it could read it, it would be looking in a different unregistered drive (to the newsbin install and windows), it's just a storage drive.

Installed, found the drop down and added just fine. Deleted teevee, closed and reopened, updated all my groups with the new tab. Closed and reopened, added teevee, added settings, closed and reopened. Double clicked update, it did cut the 8+ million headers in half, now it's only trying to download 4.5 million. It still seems a lot, just for 24 hours. Maybe it has download them before it can remove them from the group? I was tempted not to let it continue at 4.5 million and just do 100k again but I'll let it run. It's up to 2 million downloaded, will check back in about 20 min.

Thanks

[dexter]

It is still going to download all the headers in the range, it will then throw away anything matching your filter before it inserts it into the database. I downloaded 2 days of a.b.teevee yesterday and it was 9 million headers. Filtered down represented like 28,000 real posts.

Let it run, wait for Newsbin to process the headers (including filtering out stuff you don't want) and then load the group and see what you have. Remember to watch the Cache display in the Status bar to tell when the headers are done processing. The number in the parenthesis, i.e. "Cache 90/100 (xx)", will go to 0.

[StrangeLove]

YES! It worked, thanks again for all your hard work and help!

[Valsidur]

A critical instruction that might be missing is once newsbin is running, to right click on the group or group of groups, and check the "Use Profile" and select the filter name you created for the group of groups you are applying it too.

[dexter]

No, that is not critical, that turns on the filter profile to be applied when you load headers from the database. It has nothing to do with setting the filter to apply to headers *before* they are in the database.

[ack9]

Bravo guys on adding the additional setting in the new RC2.

[jacob733]

A little tip for how to identify the spammers if newsbin locks up on importing headers in the DB if you happen to run Linux:

Navigate to the import folder in a terminal and zcat the files. This will output the raw headers and it should be relatively easy to spot the big spammers by starting and stopping a couple of times, the spam headers will show up most of the time then.

If there are multiple spammers then you can use grep -v to filter out known spammers one by one so you can spot the new ones.

[Brad#4]

Now that I picked up RC2 that makes it a tad bit easier to filter out the bulk spamers.

I have to prune/sweep up the db3 that has all of that spam in it.

I downloaded the current version of sqlite3

Tried this script under cmd shell with no luck (I did remember to exit the newsbin program first)

set SPOOLDIR=C:\Users\brad#4\AppData\Local\Newsbin\spool_v6

for /F %%x in ('dir /b/d/ad %SPOOLDIR%\alt*') do (
echo %%x
C:\Users\brad#4\AppData\Local\Newsbin\sqlite3.exe %SPOOLDIR%\%%x\Storage.db3 "select count(*) from Compact_V4 where NN_From='nEwZ[NZB] <[email protected]>';"
)

but the offending poster is still in the database files. - right click purge all posts from user looks like it is doing something for 5+ mins - then hangs on a wait.

highlight one message and sift del fails to remove one message.

I rather not remove the full database because the download times suck due to how much spam has been stuffed in the impacted groups.

[dexter]

Right-click and delete all posts from poster is the way to go. Do it before you go to bed and let it run overnight. It could take hours depending on how long you've been downloading this spam.

[alxnet]

No, that is not critical, that turns on the filter profile to be applied when you load headers from the database. It has nothing to do with setting the filter to apply to headers *before* they are in the database.

Good to know. Thanks.

[zorro6204]

How do you get 6.73? I updated and it only gave me 6.72. I tried the instructions anyway, exactly as they pictured it, but no go, it's currently trying to download 2.5 million headers . . .

[ack9]

Now that I picked up RC2 that makes it a tad bit easier to filter out the bulk spamers.

...
C:\Users\brad#4\AppData\Local\Newsbin\sqlite3.exe %SPOOLDIR%\%%x\Storage.db3 "select count(*) from Compact_V4 where NN_From='nEwZ[NZB] <[email protected]>';"
)

You're using the wrong part of my message: "here's a quick batch script to -->automate finding<--- the affected groups:"

The DELETE command is up above that section.

[ack9]

A little tip for how to identify the spammers if newsbin locks up on importing headers in the DB if you happen to run Linux:

Navigate to the import folder in a terminal and zcat the files. This will output the raw headers and it should be relatively easy to spot the big spammers by starting and stopping a couple of times, the spam headers will show up most of the time then.

If there are multiple spammers then you can use grep -v to filter out known spammers one by one so you can spot the new ones.

Or use SQLite for this. The following will find posters, in a given Storage.db3 file, with more than 500 posts (change to whatever value you want) and print their Poster value (NN_From) and total records count, sorted by smallest number of records to largest:

From SQLite's command prompt after you have opened a .db3 file:
# select NN_From, count(*) AS CountOf FROM Compact_V4 GROUP BY NN_From HAVING count(*)>500 ORDER BY count(*);

[jacob733]

Or use SQLite for this.

But if Newsbin locks up while importing headers then the database might not have the headers yet. That's why it is worth checking the import folder.

[dexter]

How do you get 6.73? I updated and it only gave me 6.72. I tried the instructions anyway, exactly as they pictured it, but no go, it's currently trying to download 2.5 million headers . . .

It's on the Beta page, not yet released as a production version.

Even with the filters in place, you will still see the complete header count in the header download task in the download list. Fewer headers will be inserted into the header database on your hard drive though. And 2.5 million is nothing during this spam flood. a.b.teevee gets that much every day now.

[zorro6204]

Thanks, I'll check it out. I had the max days set to one, so 2.5M is about right, there was a lull that day. Still, it crashed Newsbin and froze my PC.

I'm getting more comfortable using a combination of Binsearch and Easynews thumbnails for searching and just using Newsbin to handle the nzb's during this mess (which, if the past is any clue, will end). It's not like I don't know what I'm looking for, I don't need to see everything posted. Directly accessing the troubked group through Newsbin was faster than those two, no delay, but one can learn to be little patient.

[lamekuma]

I had a quick question, is there a tutorial anywhere so I can add additional posters?

For example, in your tutorial, you have the following example:
nEwZ[NZB] Links not allowed for unregistered users = nEwZ\[NZB\].*[email protected]

However, I had added it into my lockout poster list and it turned out to be: nEwZ\[NZB\] \<pr3d@NET\.world\>

Thanks for all your assistance so far!

[wiggins09]

I am a bit mystified by todays behavior.
Yesterday the filter was set & the alt.binaries.teevee loaded up OK.
Today I see my import folder is up to >11gb the alt.binaries.teevee spool is >10gb & my ram use is 91%.
I look on binsearch at alt.binaries.teevee & there are the 00s of 'normal' posts but NB is showing me just 22 entries for the last 24 hrs.
Has the spammer changed the post naming format? Or am I just lucky?

I have the filter setup & active via 'properties'
Hopefully I'm right in thinking a 'filter in' is exclusive - anything but the value is rejected. That is what I will try next.

Using w7x64 + nb 6.73rc2 b4820

[JesseKnows]

Still on 6.72.

Question about the filters:
In the Filter Profiles dialog I see that the filter named "KillThisPoster" contains "reject if Poster contains pr3d\@NET\.world".
I put "ccc" in the box next to "Enter Test Text" and after clicking Test I get the Test Results "Test text was accepted - Will be included." I put "[email protected]" in the box and I get the same Test Results. Same results with "pr3d\@NET\.world". Is the test supposed to reflect the same matching as the actual filtering?

[dexter]

I had a quick question, is there a tutorial anywhere so I can add additional posters?

For example, in your tutorial, you have the following example:
nEwZ[NZB] [email protected] = nEwZ\[NZB\].*[email protected]

However, I had added it into my lockout poster list and it turned out to be: nEwZ\[NZB\] \<pr3d@NET\.world\>

Thanks for all your assistance so far!

Either will work. There are several ways to write a regular expression that will work on that string. The first one is more broad, the 2nd one was generated by Newsbin to match the exact term. If it is working for you I wouldn't touch it.

Keep in mind that each additional item you add to your filter profile will start to degrade performance so choose wisely.
If the regular expressions are targeting different matching items, you can just add an additional rule to the same filter profile.

[dexter]

I am a bit mystified by todays behavior.
Yesterday the filter was set & the alt.binaries.teevee loaded up OK.
Today I see my import folder is up to >11gb the alt.binaries.teevee spool is >10gb & my ram use is 91%.
I look on binsearch at alt.binaries.teevee & there are the 00s of 'normal' posts but NB is showing me just 22 entries for the last 24 hrs.
Has the spammer changed the post naming format? Or am I just lucky?

I have the filter setup & active via 'properties'
Hopefully I'm right in thinking a 'filter in' is exclusive - anything but the value is rejected. That is what I will try next.

Using w7x64 + nb 6.73rc2 b4820

Are you sure nothing changed with your filters? I just ran 1 day of a.b.teevee and there were almost 5 million posts but the header filter was still effective.

You aren't going to see the latest headers until the backlog clears out. You can monitor it by watching the Cache display at the bottom of the Newsbin window. It looks something like "Cache 200/200 (xx)". As long as there are numbers in the parenthesis, it means Newsbin is still processing headers into the database.

[wiggins09]

Thanks Dexter

AFAIK my filters/settings were unchanged & I did go thru a check to ensure everything was as it should be.
Ended up deleting the <22gb & starting again with a new batch of <42m headers
It is still processing the cache but not using so much RAM since adding the 'accept if' statement

[dward51]

Thanks to all who have contributed to this thread!!!!

The filter poster during header download is working perfectly for me (although I see there is now a patch in the program update to make the process easier). Even with 32gb of ram and an octa-core processor on SSD drives, all that spam basically was putting the affected groups in an endless sort for me.

[zorro6204]

I tried again, downloaded the new version, did exactly and precisely what the "wiki" said, clicked on the teevee group, and about an hour later Newsbin crashed with an "out of memory" error. That left my PC running all messed up and slow, so I cleared all the junk out with Cyber Scrub and rebooted, tried again. The download said it was only trying to load 60K headers, but the blue bar has been crawling across the bottom of the screen for 15 minutes, and I'm going to abort.

[jacob733]

I tried again, downloaded the new version, did exactly and precisely what the "wiki" said, clicked on the teevee group...

You should probably wait to actually open the group until the headers finish downloading and importing. Newsbin is much faster once the download and import is complete.

[robe36]

Is it possible to add message-id to filter gui? Today they are posting the generated names, thousands of different names, in the a.b.teevee and a.b.moovee.

I do notice one thing they share in common is the message id showing @PRIVATE at the end of each message ID.

I would copy an example here, but when rt clicking a header and selecting properties, I have no ability to copy that info to the clipboard.

See this same trend on another poster in a.b.misc and a.b.etc with message ID ending with @reader.easyusenet.nl

Thanks

[BillFrugge]

I am a bit mystified by todays behavior.
Yesterday the filter was set & the alt.binaries.teevee loaded up OK.
Today I see my import folder is up to >11gb the alt.binaries.teevee spool is >10gb & my ram use is 91%.
I look on binsearch at alt.binaries.teevee & there are the 00s of 'normal' posts but NB is showing me just 22 entries for the last 24 hrs.
Has the spammer changed the post naming format? Or am I just lucky?

I have the filter setup & active via 'properties'
Hopefully I'm right in thinking a 'filter in' is exclusive - anything but the value is rejected. That is what I will try next.

Using w7x64 + nb 6.73rc2 b4820

I'm seeing the same thing. Suddenly I'm stalling again during header download. Nothing on the search engines.

I killed the download and looked in on the headers. I don't know if it's the same clown, but someone figured out a new angle. Every post has a unique email address, and there is nothing common in any of the headers that can be filtered.

Best bet is to reverse your filter as was suggested earlier. Change it to accept only posts containing '\[FULL\]\-\[\#a.b.teevee\@EFNet\]' in the subject line. You'll want to do this in a unique folder because this will prevent anything else that does not include '[FULL]-[#a.b.teevee@EFNet]' in the subject from coming through. I tried it on my setup, and I was finally able to load headers. 2 Million in the last 3 hours!

[stephane]

I guess that
Links not allowed for unregistered users (RIPPER)
should be filtered out too.

Groups:
a.b.comp
a.b.erotica
a.b.movies
a.b.multimedia
a.b.warez

Update sent to https://t.me/NewsbinUsers

[robe36]

Hey BillFrugge,

I'm confused on what you mentioned by adding this as an accept filter.. \[FULL\]\-\[\#a.b.teevee\@EFNet\] .. I usually search several groups I'm subscribed to, unknowing what I will get from them. Are you saying to add exceptions for each poster that I usually get the downloads from? otherwise that filter to add may cause me to miss alot of posts.

Would be nice to get the message ID filter added so we can reject from that most likely spammer PRIVATE

I sent 2 emails to their usenet provider abuse email address, but they don't respond to emails, at least from me. [email protected]

[BillFrugge]

Hey BillFrugge,

I'm confused on what you mentioned by adding this as an accept filter.. \[FULL\]\-\[\#a.b.teevee\@EFNet\] .. I usually search several groups I'm subscribed to, unknowing what I will get from them. Are you saying to add exceptions for each poster that I usually get the downloads from? otherwise that filter to add may cause me to miss alot of posts.

Would be nice to get the message ID filter added so we can reject from that most likely spammer PRIVATE

I sent 2 emails to their usenet provider abuse email address, but they don't respond to emails, at least from me. [email protected]

Whereas you have a filter now that says something to the effect of "reject if poster contains," or "reject if subject contains," my DownloadFilter now says "Accept if subject contains" and used the above string for the criteria.

I added to my message that you'll want to move teevee to it's own folder in your group list (and add the DownloadFilter line) so that this filter is only applied to the teevee group. It will filter out everyone else so that you only see the posts by the bots. I hate doing that as well, but it's the only way I see of getting only the posts that I want.

Somehow, I don't think their usenet provider is going to pay attention. They must be aware of all of the traffic that they've suddenly received and they haven't addressed it yet.

[toryin40]

I ended using a new Reject filter if subject contains \[..........\] because via regex I can kill any post with 10 characters inside of two brackets... and that seems to be their pattern at the moment. With a future proof of \\..........\\ since some of the older posts used that for the hash mark

[jacob733]

I took a closer look at the data. It is not spam as such. It looks like it is on-topic data. It is just obfuscated and possibly encrypted. I guess someone grew tired of the stuff getting DMCAed.

Bad news is that this would mean that the posts are not going away again. And they will probably just increase in volume.

They have a special header named "Verify-Signature-Newznzb", so if it was possible to filter on existence of a header, we could target this.

In the mean time, I will try to make a regex that matches the subject. The characters look hexadecimal, so I will work on that. How advanced is the regex support in Newsbin?

[Taylor]

Now flooding with random header and posting fields.

As he is using an unusual usenet provider can't that be blocked or an ipaddress from his posts?

[zorro6204]

"Now flooding with random header and posting fields."

Maybe that explains why the procedure didn't work?


"You should probably wait to actually open the group until the headers finish downloading and importing. Newsbin is much faster once the download and import is complete."

But, isn't the purpose of the filter to kill the bad posts? On a normal day Newsbin would load the teevee headers in seconds.


Also, I doubt the poster is well intentioned, you don't fill multiple newsgroups with old posts to the point where the software blows up to be doing a service. He knows he's hated and is revels in it, it's the only explanation.

[MichaelW_]

well, the asshole in teevee has switched to random characters now, with random poster id's. Every post seems to include in the subject a ::/ and/or a \:: so i tried trapping a filter for subject contains one of those, but it doesn't seem to be working. Here is a sample subject line:

[ee8cea4558] \775a118142\::3660a589b05a25.e7c07822a7ff1bdf548456da991610.b901d0e7::/2970e1e8293c/ [a441a920a9] [1/2] - yEnc 26342209536

is there any reason this shouldn't work?


Update - I then tried trapping for subject contains ] \ and that doesnt work either

[lamekuma]

I ended using a new Reject filter if subject contains \[..........\] because via regex I can kill any post with 10 characters inside of two brackets... and that seems to be their pattern at the moment. With a future proof of \\..........\\ since some of the older posts used that for the hash mark

Okay, I'll try to use the reject subject header if it contains \[..........\]

I just hope it doesn't block unnecessary stuff

[zorro6204]

Maybe the solution is to abandon teevee for . . . alt.binaries.cute.cat.pictures or something and cross post to teevee so it takes asshat awhile to catch on.

[jacob733]

This subject reject filter seems to work for me:
\[[0-9a-f]{10}\] \\[0-9a-f]{10}\\::[0-9a-f]{14}\.[0-9a-f]{30}\.[0-9a-f]{8}::/[0-9a-f]{12}/ \[([0-9a-f]{10}|newzNZB)\]

Also, I doubt the poster is well intentioned, you don't fill multiple newsgroups with old posts to the point where the software blows up to be doing a service. He knows he's hated and is revels in it, it's the only explanation.

Actually, he probably doesn't care. He just charges people $10 a month to give them a DMCA free warez service, and this is his data store. At least that is what I get from https://usenetreviewz.com/newznzb-review/ and https://newznzb.info/

I guess we can hope that he switches groups soon, otherwise this will never end.

[fcorreia666]

I wish I knew how to kill the spamm. But I do not know, unfortunately.
But the idea to abandon teevee, moovee and other groups may not function as the spammer may be reading the posts in this forum.
So, everyone goes to alt.binaries.something and the follows. The only thing he needs to do is add another NG.
I am not tech savvy but I suppose he is using automated tools to do the spamming.
Maybe there are other tools to kill the tools he uses to do his spamming.
I leave that to the guys who know about software.
Just for the record. I sent a complaint to Giganews about this person and never received an answer.

[jacob733]

But the idea to abandon teevee, moovee and other groups may not function as the spammer may be reading the posts in this forum.

I pretty much guarantee that he doesn't care. He just needs somewhere to store his obfuscated files. Or I should probably say they, since it looks like a group of some sorts. Perhaps the Russian mafia or another organisation dealing with cyber crime.

Anyway, I looked at some of the files and my guess is that if you have some special newzNZB software and an account to someplace then you can get real videos from those posts. Unfortunately that probably means that the poster will need to mutate the format regularly to stay ahead of DMCA requests, and at the same time cause regular annoyance for us when we need to update our filters.

[StrangeLove]

Of course they are reading the posts, hell there probably reading here. It's obviously an attack on the server. There's tons of places they could dump files if they are some secret russian mafia files. There's no need to pick the most popular newsgroup.


Thanks again for all your work on this great program!
How would you guys feel about adding some functionality that would allow, bypass of Delete Poster and Approval popup. I.e. what if we could say hold down CTL, then right click on a post, automatically deleting all posts from that spammer? Then there's only 2 clicks or 1 (if you don't count holding CTL), instead of 4.

[ytaxq889]

When I saw the irritating posts I noticed they were often of a uniform size.

Perhaps a download filter could be created along the lines of "if post greater than XX bytes and less than YY bytes, then don't download". That might not get them all, but enough to make the Group usable. A Message of the Day could be sent to suggest the range if it becomes a variable. Obviously if he makes the post sizes large, then the quantity of posts will decline.

Also this person must be being paid to do this. E.g., if Giganews et al stop this poster, then that proves they are able to exclude posters. In turn that may result in "requests" to stop the good posters. If this person is selling some service, I would avoid giving any personal information to him.

Also it would be nice to have a finer gradation in Min and Max sizes.

[shadowmn]

FYI, Version 6.73RC2 now allows you to configure a header download filter from the GUI so you don't have to mess with the .nbi file.

viewtopic.php?f=44&t=41375

New instructions (including pictures!) are in the Newsbin Help Wiki.

I think the instructions need to include that you need to restart Newsbin after you add the Header Filter before it will begin taking effect.

shadowmn

[bnf]

Eight million new headers in twelve hours. Even with my filter working, the initial download takes an eternity now. Sigh.

[shawn]

I don't think there's any choice now but to set up filters based on what you will accept and not what you will reject.
That way you can filter any new iteration that the spammer uses (at least for now.) His new headers are just too different
to try adding them all in to any reject filter.