It might be worth your time to separate the temperature effect from the fouling effect.
For what it's worth, I try to tune my loads to shoot a cold bore group. It takes forever to shoot and gets boring as heck at the range, but I set a timer to make sure I have at least three minutes between shots. Obviously the first shot is the one that counts while hunting. The bore will always be cold. A load that tightens as a barrel warms is of no interest to me.
In regards to fouling, I usually scrub the crap out of the bore before load development. I use a nylon brush and a cordless screwdriver to get the carbon ring out of the end of the chamber just ahead of where the case neck ends. JB's bore paste to eliminate fouling from the barrel, especially the "tight spot" that forms from the throat into the first few inches of rifling. (I can share more on how to detect this if you don't already know) I always want a clean barrel as a baseline starting point.
After the major scrubbing, the first clean and cold bore shot is invariably at a different POI from subsequent shots. If the cold bore shots do not group well after that, I move on to a different powder, bullet or primer until I find a match up that works.
If fouling creates a sweet spot, you probably ought to run with it as long as it is consistent. If I can track a consistent fouling pattern that promotes accuracy I will scrub the bore before hunting season, then shoot the predetermined amount of foulers, then dry patch the bore a few times and hunt with it.
Hope that helps. I also hope I did not insult your intelligence by stating things you already know. Good luck with it. I'd be curious to hear your progress.
BTW, what bullet and powder combos are you trying?