I've used a bivy a number of times and love them. Most of the time its during backcountry camping and a few times during the winter. Ill throw my ground pad inside it and sleep right on the snow if needed. They add a lot of warmth because they allow the sleeping bag to further contain the body heat. Make sure it's gortex or some other breathable fabric, otherwise you will get condensation even if the head area is wide open. The bivy I use is REI's basic one. Use one of their 20% off coupons and it's something like $80. I wouldn't sleep in the rain in it on purpose, but I've done it. Most of the time i find it too warm, but its great in cold weather. For the basic one I have, there's no head covering. It's really just a sleeping bag shell. So if running into bad weather I tuck my head under a tree or rock overhang. The rest is waterproof. Sorry, getting off topic. Just saying I've used them a lot in all types of conditions.
Fleece vs bivy? This is the way I look at it. If you were cold outside would you be warmer by throwing on a fleece or a gortex shell? In my experience a shell is much much warmer. It's all about blocking the outside air and containing your own.
Clothes worn in the bag? I go back and forth on this. Not sure why but sometimes it's warmer with longjohns and other times its not. Generally bulky clothes don't work. Air in a bag is good, you just dont want it to circulate out of the bag. I will mention that blood circulation is just as important as other factors. Wearing socks, LJs or other clothes that are a little tight can make you colder than not wearing them at all. It's important to not be too constricted in a bag or any one part of your body. This doesn't happen often, but it does under the right circumstances...usually wool socks that shrunk on me in the dryer and I'm realizing it too late.
Also....it's counter intuitive, but a decent ground pad can make a difference. Thermarests are comfortable, but not great for insulating in cold weather. a foam pad is much warmer. In the winter when its in the teens or lower at night ill use both.
I've noticed the past couple hunting seasons that sleeping on a cot in a tent is colder than the ground. Having that air circulating below you on a cot has an impact when the temps drop. Just thought I would mention it since you didn't state what you sleep on.