Newsbin seems to hang deleting large numbers

Technical support and discussion of Newsbin Version 6 series.

Newsbin seems to hang deleting large numbers

Postby deebee » Sat Jan 21, 2017 5:22 pm

:?: :?:
There are a couple of groups being spammed with huge numbers of small posts. Looks like over a million in a 24 hour period. I have them filtered. If I try and select them all so I can delete them Newsbin says not responding but keeps on using CPU and Memory if I look in task manager. It will sit like that for hours. My only option is to kill Newsbin, restart it and now only select small groups of posts to delete.Suggestions??
deebee
Occasional Contributor
Occasional Contributor
 
Posts: 19
Joined: Mon Feb 09, 2009 3:20 pm

Registered Newsbin User since: 10/02/08

Re: Newsbin seems to hang deleting large numbers

Postby Quade » Sat Jan 21, 2017 6:19 pm

Try right click "Delete all Posts by Poster" on one of them.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 45009
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Newsbin seems to hang deleting large numbers

Postby Bexley » Sun Jan 22, 2017 11:44 am

Using delete all posts from poster hangs as well. It was coming back, but the recent increased amount of spam seems too much for something in Newsbin. Is there an external sqlite delete that would clean this up from outside Newsbin? Regards.
Bexley
Occasional Contributor
Occasional Contributor
 
Posts: 35
Joined: Tue Nov 22, 2011 8:30 pm

Registered Newsbin User since: 05/01/09

Re: Newsbin seems to hang deleting large numbers

Postby bw909 » Sun Jan 22, 2017 2:41 pm

Newsbin is really choking on all the posts on a.b.teevee

If I try right clicking "Delete all Posts by Poster" the program completely freezes with "not responding" and I have to use task manager to quit Newsbin.

It had been doing reasonably well with all the spam until last night when it started taking up to 20 minutes to display new headers after downloading them.
bw909
Occasional Contributor
Occasional Contributor
 
Posts: 15
Joined: Thu Dec 18, 2014 7:22 pm

Registered Newsbin User since: 12/09/14

Re: Newsbin seems to hang deleting large numbers

Postby dexter » Sun Jan 22, 2017 6:26 pm

You should install a filter to prevent these headers from getting in to the database during the header download. Quade posted about it here:

viewtopic.php?f=8&t=41268
User avatar
dexter
Site Admin
Site Admin
 
Posts: 9514
Joined: Fri May 18, 2001 3:50 pm
Location: Northern Virginia, US

Registered Newsbin User since: 10/24/97

Re: Newsbin seems to hang deleting large numbers

Postby don_colchon » Sat Jan 28, 2017 3:22 am

I guess it is a group display issue:
After "Show Posts" (not "Show all Posts") on a group which is hit by the spammer and then clicking "Delete all Posts by Poster" I see a lot CPU & a lot I/O with process explorer. Storage.db3-journal grows and stop growing, StorageData.db3-journal grows and stop growing. Then I/O ends. But CPU stays at maximum without any I/O. Newsbin "freezes".

Then I kill newsbin, restart and use "Compact Group...", successfully.

Happens with 6.72 & 6.73RC2 (Build 4820, with or without the header-load filter applied)
..come un gatto vorrei vivere..
User avatar
don_colchon
Occasional Contributor
Occasional Contributor
 
Posts: 29
Joined: Sat Jun 07, 2003 11:04 pm

Registered Newsbin User since: 10/17/02

Re: Newsbin seems to hang deleting large numbers

Postby dexter » Sat Jan 28, 2017 9:56 am

Yeah, you are asking Newsbin to delete millions and millions of records due to the spam. It is resource intensive.

If you implement the header filtering as described here, you can avoid having to do this as long as the poster of the spam is consistent.
User avatar
dexter
Site Admin
Site Admin
 
Posts: 9514
Joined: Fri May 18, 2001 3:50 pm
Location: Northern Virginia, US

Registered Newsbin User since: 10/24/97

Re: Newsbin seems to hang deleting large numbers

Postby Chuck D. Bones » Sun Jan 29, 2017 8:34 pm

There has got to be a better way of cleaning the SPAM out of the a.b.teevee database. I had the exclude pr3d@NET filter in place, then the %$^& spammer changed his poster name and I got one day's worth of trash in my database. Delete all posts from poster is taking forever because the poster name keeps changing. Need a more powerful filter that can clean the database after the fact. I appreciate all of the work you are doing, but this %$&^ manages circumvent the countermeasures we put in place. Right now I have a download filter that only lets certain posters in, but I still have a database full of trash to contend with. Right now my other option is to delete the database and start over. That sucks because I have 3,000 days worth of headers and like being my own indexing service.

Thanks,
Chuck

It is hard enough to remember my opinions, without also remembering my reasons for them!
- Friedrich Nietzsche
User avatar
Chuck D. Bones
Active Participant
Active Participant
 
Posts: 52
Joined: Sat Jul 05, 2003 1:46 pm
Location: Somewhere in L.A.

Registered Newsbin User since: 04/03/03

Re: Newsbin seems to hang deleting large numbers

Postby Quade » Mon Jan 30, 2017 4:20 am

You could try

"\[[0-9a-f]{10}\] \\[0-9a-f]{10}\\::[0-9a-f]{14}\.[0-9a-f]{30}\.[0-9a-f]{8}::/[0-9a-f]{12}/ \[([0-9a-f]{10}|newzNZB)\]"

As a subject reject filter in the header filter. Don't add the quotes.

Not going to help the existing DB. Might be better off just deleting it and downloading the last 20 days worth of headers.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 45009
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Newsbin seems to hang deleting large numbers

Postby don_colchon » Mon Jan 30, 2017 1:52 pm

I found a way to delete up to 400000 headers from the DB, and so, step by step, I was able to remove the unwanted headers without using "Delete all posts from Poster" because I really, really need the collections of 10 year old headers.. :D But it needs a powerful PC.

1. In Options set "Display Age:" to a few days.
2. Invent & use some search REGEX to restrict the number of posts displayed somehow. I used 16 search patterns:
search in Poster Field: "(?<first>0[A-Za-z0-9]{8})\s<\k<first>@\k<first>\.\k<first>>"
...
search in Poster Field: "(?<first>f[A-Za-z0-9]{8})\s<\k<first>@\k<first>\.\k<first>>"
3. Open a search window and use one pattern (this step needs a few Gigabytes memory)
4. Hit CTRL-A ... wait .... hit SHIFT-DEL .... wait some minutes (in this step I used process explorer to find out when the delete's really finished (=no CPU+no I/O) (this steps need some more memory + 100% single core usage and a lot of I/O for several minutes)
5. close the search window (<- it appears that not all memory is freed in this step) and repeat with the next search pattern until all groups all clean.
6. restart newsbin (to free memory)
7. Use "Post Storage" -> "Compact Group" to reclaim the space the deletes have freed.
..come un gatto vorrei vivere..
User avatar
don_colchon
Occasional Contributor
Occasional Contributor
 
Posts: 29
Joined: Sat Jun 07, 2003 11:04 pm

Registered Newsbin User since: 10/17/02

Re: Newsbin seems to hang deleting large numbers

Postby nethead » Mon Jan 30, 2017 11:00 pm

would this work to block all the new spam if placed in the reject/poster contains?

([0-9a-f]){9}@([0-9a-f]){9}\.([0-9a-f]){9}
nethead
Occasional Contributor
Occasional Contributor
 
Posts: 44
Joined: Sat Feb 12, 2011 3:03 pm

Registered Newsbin User since: 02/25/06

Re: Newsbin seems to hang deleting large numbers

Postby Quade » Tue Jan 31, 2017 12:45 am

It depends on whether the poster is using the same looking from field or not. The from field can be totally random so, I'd avoid using it if possible. The subject field seems more stable.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 45009
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97


Return to V6 Technical Support

Who is online

Users browsing this forum: No registered users and 27 guests