I regularly use a bike on logging roads and other places where allowed, and don't see why an e-bike is that different. But maybe it's because I only use it while scouting and exploring? While hunting it always seemed like a liability to me. They aren't completely silent, and let's face it, you're not packing out a 90lb quarter while riding a bike. I always figured it would just mean one extra trip "back in" to retrieve it. But maybe that's just my hunting style.
I hunt a lot of wilderness areas where even non-e-bikes aren't allowed, so that cuts the usefulness in half. Maybe that's part of it.