Weird performance loading headers

[BrownBullhead]

Hello. Recently factory restored my PC. Install NB Pro and downloaded 10 years of headers from a group. Yes, I want to browse 1000s of binaries to decide what to download.

It took about 2 hours to download the headers. I closed and re-opened NB today.

I've now been waiting about 4 hours for headers to load. Seeing 1000's of lines like this in the log.

What am I looking at here?

Code: Select all


[04/23 21:55:03]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:04]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:05]  ALWAYS CNNTPDbProcessingThread - Importing Compacts Finished
[04/23 21:55:05]  ALWAYS CNNTPDbProcessingThread - Importing Compacts
[04/23 21:55:06]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:07]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:08]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:09]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:10]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:11]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:12]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:14]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:15]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:16]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:17]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:18]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:19]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:20]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:21]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:22]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:23]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:24]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:25]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:26]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:27]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:28]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:29]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:30]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:31]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:32]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:33]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:34]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:35]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:36]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:37]  ALWAYS CNNTPDbProcessingThread - Importing Compacts Finished
[04/23 21:55:37]  ALWAYS CNNTPDbProcessingThread - Importing Compacts
[04/23 21:55:38]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:39]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:40]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:41]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:42]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:43]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:44]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:45]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:46]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:47]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:48]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:49]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:50]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:51]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:52]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:53]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:54]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:55]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:56]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:57]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:58]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:55:59]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:00]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:01]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:02]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:03]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:04]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:05]  ALWAYS CNNTPDbProcessingThread - Importing Compacts Finished
[04/23 21:56:05]  ALWAYS CNNTPDbProcessingThread - Importing Compacts
[04/23 21:56:06]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:07]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:08]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:09]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:10]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:11]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:12]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:13]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:14]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:15]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:16]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:17]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:18]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:19]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:20]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:21]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:22]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:23]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:24]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:25]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:26]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:27]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:28]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:29]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:30]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:31]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:32]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:33]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:34]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:35]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:36]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:43]  ALWAYS CNNTPDbProcessingThread - Importing Compacts Finished
[04/23 21:56:43]  ALWAYS CNNTPDbProcessingThread - Importing Compacts
[04/23 21:56:44]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:45]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:46]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:47]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:48]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:49]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:50]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:51]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:52]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:53]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:54]  ALWAYS Feeder: Waiting for last block to finish....
[04/23 21:56:55]  ALWAYS Feeder: Waiting for last block to finish....


[BrownBullhead]

So as I further dig on this end.

My 250,000,000 header download yesterday dumped hundreds of 50-mb or so files .TXT and .COM[random number] into D:\Newsbin\Import . Those are all time stamped yesterday.

Now tonight while waiting 4 hours to load headers, in a folder D:\Newsbin\spool_v6\[xxx group name redacted]\ I can see a file now over 3GB that is modified continuously and the time stamp reflects the current time as i type this.

So is what I am witnessing, the creation of a massive header database STORAGE.DB3, from those 650 or so files in the first line of this post?

Yes I think I am onto someting. The file count in D:\Newsbin\Import\group name\ is decreasing by about 2 files per minute, while that DB3 file in the other folder is continuining to grow into the 3.2gb range.

[Quade]

You downloaded TB's worth of headers. Call it 250,000,000 * 400 bytes = alot.

Newsbin downloads them as fast as it can then imports then into a database. What you're seeing is the import process. It's worked this way for years 6.90+ multi-theads the import to improve performance and the message you're seeing is indicating that the reader thread is much faster than the writer so the reader is stalling the import until the writer finishes the current block and moves to the next.

Only the .txt files will be imported, the other files indicate an aborted header download that left partial files behind. Newsbin will automatically clean those files up over time.

[BrownBullhead]

@QUADE
Brilliant explanation. I appreciate you taking the time to explain.

Now, if i could only find an NNTP provide that can saturday my 1000 mbps ISP connection.

Newshosting never gets me more than about 450 mbps even downloading and extracting to SSD drives.

[Quade]

One of my IRC guys in the EU gets greater than 1 Gbps to Eweka. My speeds to usenetserver seem to depend on what I'm downloading. I can max out my 1 Gbps Ethernet to my router sometimes using 10 connections. That's 900-950 Mbps. Unless you have a faster than 1 Gbps connection between you and the router you're probably not going to get much more than 900-950 or so. Ethernet has overhead which means getting full 1 Gbps is unlikely.

[BrownBullhead]

One of my IRC guys in the EU gets greater than 1 Gbps to Eweka. My speeds to usenetserver seem to depend on what I'm downloading. I can max out my 1 Gbps Ethernet to my router sometimes using 10 connections. That's 900-950 Mbps. Unless you have a faster than 1 Gbps connection between you and the router you're probably not going to get much more than 900-950 or so. Ethernet has overhead which means getting full 1 Gbps is unlikely.

Right now I would like to get more than 400 mbps.

I can get 940mbps DN and 105mbps UP on Speedtest-OOkla Windows App.