Watch topic missing some items

[rh]

Search for: *
Look in groups: <blank>

In the past few months this has been very persistent. My Watch Topic using the search above is not returning somethings that ARE out there and match my chosen Filter Profile.

So I have to remember to do a manual search for them. When I do that, I ALWAYS find what I'm looking for. Don't know why Watch Topic didn't place it in the list. I can see from the manual search that it was a recent post so no reason for Watch Topic not to have found it. Having to to more manual searches just reduces the value of Watch Topic.

Watch Topic runs every 4 hours.

I'll go ahead now and install the current beta. I've been following it but seen no relevant changes that might fix this.

[Quade]

I did a wildcard search on all groups and it went back in time about zero minutes. Since that's the case your 4 hour wildcard will probably not return meaningful results because there are simply too many results from trying to wild card without group filtering. If I use group filtering, it seem to go back in time about 18 hours

I'll ask Dex what he thinks. I think you'd be better off making a group of groups topic with the groups you normally download from and use them in your watch list so, you only look in meaningful groups.

Look in groups: <blank>

This part basically. Dex did spend the past 3 days fighting a spam flood so, yesterdays results might be iffy.

[rh]

I only left it blank in the beginning just being lazy and thinking what the heck, surprise me with groups I never knew existed. But, in the past year, it's really been the usual groups that showed up in the matches that did appear. I do have two GOG's I've used forever. I might just merge them into one and use that.

Thanks for the tip!

[rh]

...and using my current GOG in the Watch List "Look in groups" has enabled it to pick up more posts more frequently. So searching all was just too much for it. Understandable given the scope of what's out there. Thanks again for the tip!

[rh]

Well, I spoke too soon. Also I should clarify that I've had a Filter Profile active for all these searches. The one I've used for years. So my assumption is that Search for: * with <blank> for look in groups would look everywhere (all indexed groups) for anything matching my Filter Profile. That Filter Profile doesn't return a heap of results so quantity is certainly not an issue.

So, I've since created two Watch Lists. One looking in a GOG that has just 4 groups, the other looking in a GOG that has maybe a dozen that don't get as many posts. Theory being limiting what to search through helps reduce the chance of missing posts. Still some posts are being missed.

For example: Both Watch Lists now show a post from about 1h45m ago. Missing is an expected related post of the same content. Regular internet search finds that missing post and indicates it was posted about 3h ago. My interval is 4h.

Therefore, while not searching through ALL groups out there does seem to allow Watch List to pull more content, some things still get missed.

Also, speaking of TWO Watch Lists, each ones "Look in Groups" set to a different GOG. Odd thing is, after running a bit, I've noticed results from one group appearing in both Watch Lists, despite one of them being configured to look in a GOG that does not include that group.

FYI I'm still on 6.73 - other than the above running solid - and didn't recall mentions of Watch List changes during the betas so I've just sat on the sideline for the next official release.

[dexter]

Can you PM me some exact examples so I can look into it? A few points:

1) Wildcard returns a max of 10,000 matches. If there is some kind of flood going on during your poll interval, you may miss some stuff.
2) Curious why you have polling set to every 4 hours? If you have an unlimited search tier, there is no reason not to run at the default of 30 minutes which will help insure you don't miss anything due to hitting the max results allowed per poll.
3) Seeing a group appear that is not in your list of groups for the watch typically means it was cross-posted and Newsbin just chose to display that specific group out of the list of groups to which it was posted.

For me to research this, I will need all the details of your setup. Unless you can come up with a minimal cofiguration that recreates the situation.

[rh]

1) Thinking maybe I didn't understand the scenario fully. So I guess there's no reveal of the actual # of matches? Using wildcard means everything so even though I've got a filter profile active - given enough time lapsed - I could (unknowingly) hit >10,000 matches and my filter profile would only run against those 10,000 matches? If so, then that could very well be what I'm seeing.

2) I'm on Standard, so the most frequent without running out in a month is 90 minutes. I might try that since over the past year I seldom run low on searches. Was just being conservative without realizing what was going on under-the-hood. If the 10,000 limit is part of my problem, then this 'might' help some. If I'm understanding #1 above, it would be handy if the 10,000 limit could be logged in some say so you know you're pulling less than what's out there for any specific search. Not sure if it's still the same because I seldom hit the number, but regular internet search at one point was limited to displaying 2000 matches. So the few times I ever saw 2000, I knew to narrow things down.

3) ah, makes sense. the ones I'm seeing are often cross posted.

I'll adjust my interval to 90 minutes and keep a watch this week. If I hit a clear example of a post that's not showing up, I'll PM you that info.

[Quade]

1) Thinking maybe I didn't understand the scenario fully. So I guess there's no reveal of the actual # of matches? Using wildcard means everything so even though I've got a filter profile active - given enough time lapsed - I could (unknowingly) hit >10,000 matches and my filter profile would only run against those 10,000 matches? If so, then that could very well be what I'm seeing.

I had this discussion on our IRC channel the other day.

1 - You can do a wild card, allow ALL of the hits to accumulate in the watch list then load the watch list, apply the filter to the display and only see what passes the filter.

- Benefit is you always keep all the data so when the posting format or poster changes, which seem to happen frequently these days, you just need to adjust the profile but the data is all there.

2 - You do a wild card with a filter profile. Anything that don't match gets tossed in the trash before you might see it.

- If you're trying to do auto-mode, this might be the only choice but you have to ensure that you filter profile is always correct or you'll lose meaningful results.

10,000 hits isn't that many when you compare it to usenet activity. Usenet might get 10,000 an hour which means if you only poll every 4 hours, you only get 1/4th of the total hits between the two poll intervals. If you use group filtering, 4 hours might be fine but you have to check how many hits happen in 4 hours. If it's > 10,000 you might lose results.

[rh]

Interesting - so I hadn't considered #1 - applying the filter afterwards. Mostly because I presumed with a storage age setting of 2000, performance would eventually become unbearable. However, given my experience with occasional 1000+ day old posts seldom downloading, perhaps 365 is a better setting.

I'd never explored what volume of posts might occur in any given hour. But in that example you gave, that could easily explain the missing items I'm encountering.

As I mentioned earlier, since restricting to a GOG to search through has yielded a small but noticeable change in hits. So this week I'll set the interval to 30 minutes. In theory, next week at this time I should not have missed a thing.

I also just might try #1, letting everything come down - THEN - filtering. Because I've mostly ignored new stuff having switched to Watch Lists.

[Quade]

I also just might try #1, letting everything come down - THEN - filtering. Because I've mostly ignored new stuff having switched to Watch Lists.

I tend to err on the "keep more data" side. Nothing prevents you from Ctrl-A - delete to purge the watch list once you've gotten all you want from it. It'll then fill up with new data. You could wait till performance goes to hell then prune the watch list. I use "delete all posts by poster" a decent amount too.

[rh]

Yeah, I keep forgetting with Internet search I don't need to keep so much history on disk since it's easily found again later. So just let the watch list fill up - toggle on the Filter Profile when needed and clean up as needed.

So I set Filter Profile to "No Profile" and changed the interval to 90 minutes. After about 24 hours I see about 1700 posts. But wow, even though I left Look in Groups set to a GOG, I'm seeing what seems like posts from every group out there. Is that the cross posting thing again - that post is probably also in one of my GOG but the Watch List just displays the first one it found it in? Presumably if I turned off Look in Groups entirely, I'd get gobs more results.

[Quade]

But wow, even though I left Look in Groups set to a GOG,

That's what Dex and I believe. Some groups are purposely spreading files over multiple groups so, it might take 10 groups worth of posts to complete a single file.

[rh]

and confirmed. One was odd enough to make an easy search on. Expanding the search result showed it posted to 5 groups, one of which was in my GOG.

[rh]

Reviewing some random posts that came through last week, I do find some that ARE in multiple groups but NONE of those groups are in my GOG I've chosen in the Look in Groups field.

That explains why I'm seeing allot of things I used to never see when downloading headers the old way using the same GOG. So somehow search is pulling down SOME things that are not posted in the GOG's I've specified. At least according to the Groups column.