DBWorker: Gz to DB Failed (sometimes) = memory leak?

This is the place to help test and discuss Version 6 Beta releases.

DBWorker: Gz to DB Failed (sometimes) = memory leak?

Postby ack9 » Sun Aug 20, 2017 3:13 am

This week I noticed that my VM running Newsbin 6.80B6 (and B7) were eating memory to the max and overly using CPU. Newsbin didn't show up eating such on the usual Task Manager stats, but it definitely was (a shutdown of NBP returned things to normal) and was easily reproducible. I also noticed a 1000+ backlog in the import folder, and I instantly knew what was at failed, the dreaded Gz to DB failed...

So I stopped Newbin, moved the set of import files I thought were at fault and start Newsbin back up. The import queue started clearing again. And then it hit another import failed. Once it did this, you could watch the Physical Memory Usage chart start progressing increasing until it hit the 6GB physical memory limit.

The stranger more annoying thing is that *some* of the new headers would give the error but seemingly move on, however for some it would just get stuck there and create the aforementioned condition.

So I cleared the whole import folder out, rebooted, and wouldn't you know the next header download had at least one group with the same problem again. Now I'm stuck in this really vicious cycle of trying to stay on top of Newsbin lest my VMs go out of control again.

I don't care if the import process decides it can't deal with a file, I just wish it would move on consistently. Help!

-Ack
ack9
Active Participant
Active Participant
 
Posts: 69
Joined: Thu Nov 11, 2004 3:15 pm

Registered Newsbin User since: 11/11/04

Re: DBWorker: Gz to DB Failed (sometimes) = memory leak?

Postby Quade » Sun Aug 20, 2017 2:57 pm

Just looking at the code, the only way to get that error is if no headers are available to feed to the DB. Meaning it's not really a fatal error. I'll change it so you don't get an error in this situation.

Just looking at things, error or no error there's no memory leak. The error though does cause an "Integrity Check" of the database which might cause memory use to bounce.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 45006
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: DBWorker: Gz to DB Failed (sometimes) = memory leak?

Postby ack9 » Sun Aug 20, 2017 8:10 pm

Quade wrote:Just looking at the code, the only way to get that error is if no headers are available to feed to the DB. Meaning it's not really a fatal error. I'll change it so you don't get an error in this situation.


Any investigating on my side that might help? There's enough files, of varying size, that are triggering this.

Also, any way to turn off the integrity check?
ack9
Active Participant
Active Participant
 
Posts: 69
Joined: Thu Nov 11, 2004 3:15 pm

Registered Newsbin User since: 11/11/04

Re: DBWorker: Gz to DB Failed (sometimes) = memory leak?

Postby Quade » Mon Aug 21, 2017 10:29 am

If you're using header download filtering, this probably isn't that unusual. You could filter down the GZ files till there's nothing left to feed into the DB.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 45006
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97


Return to Newsbin Version 6 Beta Support

Who is online

Users browsing this forum: No registered users and 31 guests