"dillweeds" with horses are just as bad as "dillweeds" on atv's or walking in from the nearest trailhead or road. The thing is in most places where there is enough room to get that far back on a horse someone (most likely an outfitter) is already going back there. Will you see less competition than the folks by the road, yes; but will you rid yourself of every other hunter, no. The best you can do is go farther than than the atv crowd, and into nastier terrain than a pack animal can handle. Places where only elk, deer, sheep, and a few crazy humans go. Even then, you might find another hunter out there with the recent push of young guns wanting to be the next cameron hanes.
To directly answer your question, the real steep and nasty of the sangres deters most folks. There are many a drainage that would be next to impossible to get a horse into and down right dangerous to pack an elk out of. Make sure you are in the absolute best shape of your life and tough as nails if that is your plan though.