He really screwed by adding people to a hunt with you without your permission and you need to be honest with him about how you feel about it.
That really makes a guy wonder about his integrity. Guys that have issues with integrity are not guys you probably want to hunt with. You might find he wants to party hunt or that he always takes the first shot, shoots at distances he can't handle, won't follow up wounded game...
As far as his buddies go, it's pretty hard to get a real good handle just from meeting them. People can change out in the woods.