Filtering out header downloads.

[zorro6204]

"\[[0-9a-f]{10}\] \\[0-9a-f]{10}\\::[0-9a-f]{14}\.[0-9a-f]{30}\.[0-9a-f]{8}::/[0-9a-f]{12}/ \[([0-9a-f]{10}|newzNZB)\]"

I tried that as a subject filter, didn't work for me. When I restarted Newsbin, it's still trying to download 8M headers, and it can't, even if I left it all night it would crash.

"Even with my filter working, the initial download takes an eternity now. Sigh."

Yeah, well, that's the problem now, I can't even get to the point where I can see if the filter is working. Why is the software ignoring the filter and trying to download all those headers?

It seems to me the solution would be to include as an option in the setup, a choice whether to ignore all posters who have put up X number of files in the last 24 hours (say). Put in a pull-down option bar with numbers like 10,000, 50,000, 100,000, etc. That would include all the good guys and exclude the bad, even if they change their name every few hours.

But for something like that to work, it has to work immediately, as soon as you add the group, or you can never load the group to use it at all.

[StrangeLove]

Shouldn't the Storage.db3 file be getting smaller as I delete all these spam files or is there a separate command to resize it ?

Thanks

[OMG2011]

I which we could filter by msgID...

[JayPea]

This current onslaught of spam in the groups is making Newsbin quite unusable. The headers are posted in such volume that even on a [email protected], 32GB DDR4, NVME SSD the application is just sitting as not responding when trying to delete these spam posts from the group. I can't imagine what it's like on a mechanical drive connected to a low-spec machine. I know there's a tendency to think of pulling headers as old fashioned but some of us still do it. Something's gotta change

[PCZ]

The latest batch of spam in ab.teevee can be filtered out quite easily.
Removing it if you do get the headers in your database is anything but.
You have to delete a few at a time otherwise NB will just hang up.

To match the latest spam add a subject filter for ::
Just two colons all the filenames have the double colon.

[alxnet]

To match the latest spam add a subject filter for ::
Just two colons all the filenames have the double colon.

Thumbs up for this.

[JayPea]

Well, the fix for me was to add a header filter to reject if poster contains [0-9a-f]{9}@[0-9a-f]{9}.[0-9a-f]{9} then purge the message stores and then pull the headers down again. Now NB seems to process the group without having a meltdown.

God damn these monstrous pricks ruining usenet. A pox on them all.

PS. will also add the :: filter now. Cheers.

[zorro6204]

I tried that, but I can't even get started, no matter what I do the software crawls, displays a memory error and crashes. For example, when I reinstate teevee I tried just downloading 50K headers, and that's all that shows on the download tab. But the blue bar still crawls and it crashes, so apparently the software is still having to process all those millions of headers.

[MarvinGardens]

It seems to me the solution would be to include as an option in the setup, a choice whether to ignore all posters who have put up X number of files in the last 24 hours (say). Put in a pull-down option bar with numbers like 10,000, 50,000, 100,000, etc. That would include all the good guys and exclude the bad, even if they change their name every few hours.

If we can determine that a single poster has put up multiple files, even though he is using different names, can't we filter on that?

I have to wonder what this spammer hopes to accomplish. Is this some sort of DOS?

[jacob733]

I have to wonder what this spammer hopes to accomplish. Is this some sort of DOS?

No. There is actual data in there. He is selling access to nzb files and possibly some app that unscrambles that data. At least that is what I get from googling newznzb

[shawn]

Has anyone else noticed something odd with the Test function for the filter? It seems that once it rejects a filter it gets stuck and
won't test new filters. I thought it might be due the amount of memory Newsbin Pro was using at the time so I restarted NBP and did
a quick test with a filter that it would reject and then tested a simple filter of just the letter a. It kept showing it as rejected even as I
tried testing it multiple times. Only a restart would allow NBP to accept the filter. This is with the RC2 candidate.

[shawn]

He may be selling a service but essentially it is a DOS for everyone else trying to use the newsgroups he is flooding. Any responsible
news server owner should be rejecting him and if they don't then the upstream provider should reject that news server until such
time as he stops flooding the news groups as it makes them very hard to use for everyone else.

[StrangeLove]

I tried that, but I can't even get started, no matter what I do the software crawls, displays a memory error and crashes. For example, when I reinstate teevee I tried just downloading 50K headers, and that's all that shows on the download tab. But the blue bar still crawls and it crashes, so apparently the software is still having to process all those millions of headers.

You've been following this thread? You need to update to the newest version to have the new drop down. Follow the instructions you will find on page 3 of this thread.

Once you delete teevee, you need to close Newsbin. Go to your install folder, then /SPOOL_V6/ and make sure you don't see a /alt.binaries.teevee folder, if so delete it. Reopen Newsbin, add new groups alt.binaries.teevee and I close here (may not be necessary). Reopen, make sure you to add your new filter to the group. Once you get it all done right click on your new alt.binaries.teevee download special 100k. You should be good to go.

[chimmyjim]

FWIW, you can get pretty specific with the following:

Code: Select all

\[[0-f][0-f][0-f][0-f][0-f][0-f][0-f][0-f][0-f][0-f]\]

This will only target the spam files and leave most everything else untouched.

Jim

[MarvinGardens]

I have to wonder what this spammer hopes to accomplish. Is this some sort of DOS?

No. There is actual data in there. He is selling access to nzb files and possibly some app that unscrambles that data. At least that is what I get from googling newznzb

Interesting marketting strategy "Let's take some action that will really, really annoy all our potential customers". I guess they went to the same school as "Rachel from Account Services" that calls me 5 times a day.

[unyquy]

Does what PCZ wrote, namely

"To match the latest spam add a subject filter for ::
Just two colons all the filenames have the double colon."

work for version 6.62. Can someone please let me know. Thanks. unyquy

[sunishun]

What a mess. But, adding this reject filter seems to be working for me so far. Thanks for all the tips and help filtering out these bastard's posts.

I ended using a new Reject filter if subject contains \[..........\] because via regex I can kill any post with 10 characters inside of two brackets... and that seems to be their pattern at the moment. With a future proof of \\..........\\ since some of the older posts used that for the hash mark

[Rilly]

After adding these new filters, I have many headers that say Subject Reject under status - but I'm still seeing those headers. What am I doing wrong?

[shawn]

Maybe those headers were already in your database? As others have said the filter feature won't get rid of the rejected from your database if they are already there. It just serves to block new headers from being added to the database if they match the filters.

[dr_dx]

To match the latest spam add a subject filter for ::
Just two colons all the filenames have the double colon.

Currently, this is working for me with 6.73RC2. Out of 10mil headers DL in teevee, 1692 legit headers came through.

[salparadise]

Setting up this filter in 6.56 was working perfectly. But whatever this spammer has changed over the last day or two has messed it up again.

From the affected groups it downloads the headers but chokes and never makes progress importing them.

I understand this is a problem for the new 6.73 RC2 as well, correct?

[Rilly]

Maybe those headers were already in your database? As others have said the filter feature won't get rid of the rejected from your database if they are already there. It just serves to block new headers from being added to the database if they match the filters.

I started with deleting the group using the "purge storage" option. Shouldn't that start it from scratch or does that not delete the headers in the database?

[lincsat]

I have to wonder what this spammer hopes to accomplish. Is this some sort of DOS?

No. There is actual data in there. He is selling access to nzb files and possibly some app that unscrambles that data. At least that is what I get from googling newznzb

Interesting marketting strategy "Let's take some action that will really, really annoy all our potential customers". I guess they went to the same school as "Rachel from Account Services" that calls me 5 times a day.

I've always suspected that this sort of spam flooding is something to do with the Media industry. DMCA orders don't work, so make the usenet unusable

[jlindquist]

Could also be Mike Pence cleaning up America's morals.

[stephane]

My config allowed Newsbin to "update" the groups on startup. When your connection to the Internet is fast, that leads to thousands and thousands unwanted headers downloaded.
My only option was to delete the group and start fresh, with the latest filters.

Works well, but - tell me if I'm wrong - I cannot load more than 10 000 000 headers. Today that's roughly the last 2 days of a spammed group.

The other choice is to reload all the headers.

I suggest there is a few new steps to the Download Special menu, maybe 50 000 000 or 100 000 000 posts ?

[alxnet]

I suggest there is a few new steps to the Download Special menu, maybe 50 000 000 or 100 000 000 posts ?

Great idea. I'd like these options too.

[johnisme]

I can't be bothered to use filters to block spammers,takes too much time,cant even load all the headers from teevee since it eats all my ram(13gb+ out of 16).All i use now is the search engine.

[Zen_Rebel]

I was focused on rejecting headers with more or less acceptable results using a filter.

I now created a filter with just: accept if....subject contains...#a.b.teevee@EFNet and the processing of todays 9 million headers took 20 seconds and i retrieved all the legit headers from the spider. Of course you won't have the personal posts and/or commentaries but i never opened them anyways. Don't forget to retrace your steps if you edited the .nbi file manually. Good luck to all!

[technogeek]

I have to wonder what this spammer hopes to accomplish. Is this some sort of DOS?

No. There is actual data in there. He is selling access to nzb files and possibly some app that unscrambles that data. At least that is what I get from googling newznzb

Interesting marketting strategy "Let's take some action that will really, really annoy all our potential customers". I guess they went to the same school as "Rachel from Account Services" that calls me 5 times a day.

I've always suspected that this sort of spam flooding is something to do with the Media industry. DMCA orders don't work, so make the usenet unusable
==========================================

Nah, it's that their customers don't use NG readers or deal with headers at all. The spammers who are selling access to the data they are parking there just need a NG that has good volume and activity which in turn cause the Providers to up the retention and disk size for it. The chaos they cause for traditional users likely never even reaches their notice. Very scammy.

[zorro6204]

"so make the usenet unusable"

But all they've done is to make one feature of usenet software unusable for one or more groups, that is, display all the headers posted over a given time. That was useful, as it hipped me to new shows I was unaware of, but there are other ways to gather that information, websites on new shows in the US, UK and elsewhere. In the meantime, posting has gone on as usual, everything I wanted I found, either using Binsearch, if the show was posted using its name, or by scanning thumbnails with Easynews, for shows posted using gibberish file names (like a particularly icy one).

So if this is a grandiose effort to shut everything down, it failed. But it doesn't feel like that, it feels like the rantings of a child.

[StrangeLove]

If it was legit, he wouldn't be using 400k files and flooding the 2 of the most popular groups. It's clearly a bullshit dos attempt at flooding.

Btw is it over for the moment, only 100k headers in the last 4 hours? That's download not, after filter header count.

[Stuart264]

Quade

Just an idea but would it be possible to add an option to use the "killthisspam" (or whatever you have called it) filter in compact newsgroups, that way you can add a filter retrospectively and clean up the fertilizer without having to delete all posts and download the headers again with an updated filter in place?

Stuart.

[Usual Suspect]

If it was legit, he wouldn't be using 400k files and flooding the 2 of the most popular groups. It's clearly a bullshit dos attempt at flooding.

If he was "just" posting for his customers he wouldn't have felt the need to change it up when other people learned how to block him out. And the volume he is posting is beyond any reasonable size, even for warez.

[Quade]

Just an idea but would it be possible to add an option to use the "killthisspam" (or whatever you have called it) filter in compact newsgroups, that way you can add a filter retrospectively and clean up the fertilizer without having to delete all posts and download the headers again with an updated filter in place?

I was wondering about this the other day. Whether there needs to be a method to apply a filter to stored headers. I'm still thinking about it. It won't be trivial.

But all they've done is to make one feature of usenet software unusable for one or more groups, that is, display all the headers posted over a given time. That was useful, as it hipped me to new shows I was unaware of, but there are other ways to gather that information, websites on new shows in the US, UK and elsewhere. In the meantime, posting has gone on as usual, everything I wanted I found, either using Binse

There's no need for them to post to popular groups. If they posted one time to some obscure groups, it would still work for them but not annoy other people.

I now created a filter with just: accept if....subject contains...#a.b.teevee@EFNet and the processing of todays 9 million headers took 20 seconds and i retrieved all the legit headers from the spider. Of course you won't have the personal posts and/or commentaries but i never opened them anyways. Don't forget to retrace your steps if you edited the .nbi file manually. Good luck to all!

To me this is the best way to filter. Only filter in the posts you want. It saves the most disk space and should make import the fastest.

[StrangeLove]

If it was legit, he wouldn't be using 400k files and flooding the 2 of the most popular groups. It's clearly a bullshit dos attempt at flooding.

If he was "just" posting for his customers he wouldn't have felt the need to change it up when other people learned how to block him out. And the volume he is posting is beyond any reasonable size, even for warez.

Had no idea he changed it up but yes, another good reason he's a fraud. I actually thought it was a different guy, since I couldn't still see the original spam once I got my filters working.

[robe36]

Any thought of adding the ability to filter message-id?

[ItsMe]

Shouldn't the Storage.db3 file be getting smaller as I delete all these spam files or is there a separate command to resize it ?

Thanks

To shrink the storage.db3 file you need to right click the group go to post storage and then click compact group, then sit back and prepare to wait, best to do just before going to bed....

[rh]

So I guess there's no way to test the filter without downloading headers? For instance, doing some sort of search through posts I've already downloaded to make sure my regex is correct.

[dexter]

So I guess there's no way to test the filter without downloading headers? For instance, doing some sort of search through posts I've already downloaded to make sure my regex is correct.

Sure there is, just create the filter profile then load a group that has some of the posts you want filtered and apply the filter profile then see if they disappear. You just have to turn the filters on in the filter bar first by clicking the icon that looks like a power button. Then select the filter profile in the last field to the right.

[dexter]

Any thought of adding the ability to filter message-id?

It exists already. Quade documented it here.

[johnisme]

There no reason to use header filtering now since the poster has stopped flooding the newsgroups,been working good with no issues for the last week.

[dexter]

There no reason to use header filtering now since the poster has stopped flooding the newsgroups,been working good with no issues for the last week.

Unless you are downloading headers from January 2017. Then you'll definitely need it. For example, the default download age is 30 days, which will definitely (from mid Feb 2017 at the time of this writing) dip in to the spam flood period.

This is only for people who add the group or reset the group. If you have been downloading headers all along and are just doing updates then yeah, probably don't need filters at the moment.

[Mark Allie]

I am going to try filtering using accept instead of reject. Do you know how to setup filtering to reject all accept what is in the accept filters. I tried accept filters and they don't actually reject things which are not accepted. Is there a reject all accept type statement or not? What do you use?

Mark

[dexter]

Do you know how to setup filtering to reject all accept what is in the accept filters.Mark

That's the definition of "Accept If". It will only bring in posts that match the regular expression of posts you wish to accept. If it isn't working then it must not be configured correctly.

You can test your filter profile by loading a few days of a group you want to use this on then apply the filter profile in that post list. If you can't figure it out, tell us what regular expression you are using and maybe we can see what's wrong with it. If you are not familiar with regular expressions, it's easy to enter something that will match everything. Also if there is a syntax error in the expression you wrote, it will appear to match everything.

[dexter]

Do you know how to setup filtering to reject all accept what is in the accept filters.Mark

That's the definition of "Accept If". It will only bring in posts that match the regular expression you want to apply to posts you wish to accept. If it isn't working then it must not be configured correctly.

You can test your filter profile by loading a few days of a group you want to use this on then apply the filter profile in that post list. If you can't figure it out, tell us what regular expression you are using and maybe we can see what's wrong with it. If you are not familiar with regular expressions, it's easy to enter something that will match everything. Also if there is a syntax error in the expression you wrote, it will appear to match everything.

[StrangeLove]

Ooc, is there a way to update your current database files to the new filters, or is Deleting the group and starting over the only way, atm?

The reason I ask, it would be nice to keep all the data I collected on another group. I lost all my past data from the teevee situation. Att, I really didn't know how to back them up properly or use filters. This other group is getting a much more sophisticated attack, where the title and poster name is randomly generated for each post. There is no common factor to create a 'Reject If' parameter. So my only real choice is to 'Accept If' parameter.


Thanks, again for all your work on this great program!

[dexter]

There currently is no way within Newsbin to apply filters to delete items already in the database except to load items in a post list, apply the filter there, then ctrl-a to highlight them and shift-delete to remove them from the database. If it is a spammed group, the system resources required to do this may be prohibitive but you could try to load the group a week at a time and attack it that way.

[pjcamp]

Would someone mind saying what, specifically, your filters are that will make Newsbin usable again? I'm not having a great deal of luck.

[StrangeLove]

Thanks for the reply!

Pjcamp, it really depends on the group your trying to filter and who/which your trying to filter. You could list the group and the poster/technique or read the thread and/or the wiki. All the answers are here.

[Quade]

"\[[0-9a-f]{10}\] \\[0-9a-f]{10}\\::[0-9a-f]{14}\.[0-9a-f]{30}\.[0-9a-f]{8}::/[0-9a-f]{12}/ \[([0-9a-f]{10}|newzNZB)\]"

Seems to be the best one with a "subject reject". If you group is already contaminated, you might need to set this filter and then purge the group and re-download headers.