DBWorker: Gz to DB Failed (sometimes) = memory leak?

[ack9]

This week I noticed that my VM running Newsbin 6.80B6 (and B7) were eating memory to the max and overly using CPU. Newsbin didn't show up eating such on the usual Task Manager stats, but it definitely was (a shutdown of NBP returned things to normal) and was easily reproducible. I also noticed a 1000+ backlog in the import folder, and I instantly knew what was at failed, the dreaded Gz to DB failed...

So I stopped Newbin, moved the set of import files I thought were at fault and start Newsbin back up. The import queue started clearing again. And then it hit another import failed. Once it did this, you could watch the Physical Memory Usage chart start progressing increasing until it hit the 6GB physical memory limit.

The stranger more annoying thing is that *some* of the new headers would give the error but seemingly move on, however for some it would just get stuck there and create the aforementioned condition.

So I cleared the whole import folder out, rebooted, and wouldn't you know the next header download had at least one group with the same problem again. Now I'm stuck in this really vicious cycle of trying to stay on top of Newsbin lest my VMs go out of control again.

I don't care if the import process decides it can't deal with a file, I just wish it would move on consistently. Help!

-Ack

[Quade]

Just looking at the code, the only way to get that error is if no headers are available to feed to the DB. Meaning it's not really a fatal error. I'll change it so you don't get an error in this situation.

Just looking at things, error or no error there's no memory leak. The error though does cause an "Integrity Check" of the database which might cause memory use to bounce.

[ack9]

Just looking at the code, the only way to get that error is if no headers are available to feed to the DB. Meaning it's not really a fatal error. I'll change it so you don't get an error in this situation.

Any investigating on my side that might help? There's enough files, of varying size, that are triggering this.

Also, any way to turn off the integrity check?

[Quade]

If you're using header download filtering, this probably isn't that unusual. You could filter down the GZ files till there's nothing left to feed into the DB.