My general take is that you will be giving up a lot of ballistic performance. The 180 gr hits real hard and you get flatter shooting. My baseline for load performance has always been to try and get as close as possible to 3000 fps at the muzzle. Just my opinion.
Remember one thing...dead is dead. 180 or 220, so why penalize yourself.