Any way to speed up Header importing?

[Lurker]

I try to download headers from even these largely spammed groups and the download of the headers seems fast, but it sits and imports the downloaded headers into the newsbin header database incredibly slow. I am talking 6-8 hours long. Is there any way to speed this up? I am on a 16core processor and while I watch with the system basically idle, task manager says newsbin only occasionally uses 0.1% of my CPU power and negligible amount of system ram. So I am thinking newsbin is not really using the available resources effectively.

Worst part of it is it interferes with the usage of newsbin. So for example if I want to pull the headers from a large group like a.b.ath I right click on the group name select download latest headers and newsbin downloads it relatively quickly. Then it takes a while to actually bring the downloaded header into newsbin's database (it can take hours). Now while newsbin is still processing the header files say I want to check another group immediately like a.b.multimedia.<something> so I double click on the group name to see the headers. Newsbin will just sits there after displaying the non updated headers and does nothing even after it downloaded the headers because a.b.m.<something> is alphabetically after a.b.ath. So till a.b.ath finishes processing it's headers, a.b.m.<something> won't import the headers so the display won't update. The only way to kind of get around it is to actually go into the newbinpro_data\import folder and delete the files ahead of a.b.m.<something>.txt temporarily (ie to the recycle bin) except the very first file which newsbin has open and is processing then wait for it to finish that one file then it imports the a.b.m.<something> headers and updates the display. I can then restore the files in the recycle bin and newsbin will process the remaining files. It seems a very sequential process, so is there a way to multi-thread this so it can process multiple groups at once or maybe have newsbin give priority to processing the group whole posts are being displayed currently?

[ddaniel51]

I use an NVME SSD for my OS and additional NVME SSD's for newsbin data and downloads. Doing even a large header update only takes a minute or so (I Luv Newsbin). Intel 7900x, 64GB, Win 10, no page files, 9 TB Samsung 960 Pro SSD's, no mechanical HDD's allowed. Sounds like a badly fragmented HDD churning page files and data.

[Quade]

Sorry I forgot to reply.

Large header imports are a bit problematic. If you load and display a group. The headers for the displayed groups get displayed before other headers. Basically there's an import priority system based on loaded groups.

Once 6.80 is out, I can consider making it multi-threaded. The reason I didn't before was because it was already relatively intensive but with 32 bit systems essentially going away, multi-threading it makes more sense.

[syshog]

Sorry I forgot to reply.

Large header imports are a bit problematic. If you load and display a group. The headers for the displayed groups get displayed before other headers. Basically there's an import priority system based on loaded groups.

Once 6.80 is out, I can consider making it multi-threaded. The reason I didn't before was because it was already relatively intensive but with 32 bit systems essentially going away, multi-threading it makes more sense.

If you are going to enable multi-threaded support for header imports could you at least give us the option of using GZ compression again? I was unimpressed with compressing the Import folder. The txt files get out of control when there are a lot of updates.

[Quade]

The txt files get out of control when there are a lot of updates.

What's "out of control" mean? People were complaining about stuck imports and that's mostly gone away with the switch to text only header imports. Virus scanners don't typically touch text files but they do mess with GZ Files. Import is faster too without the compression.