Only time down is an issue is day after day in 100% humidity and climbing through wet grass alders all day in the rain...your rain gear WILL soak through eventually...then down can be an issue for sure. But I rarely hike in my puffy stuff.
Down can be an issue, even if you aren't soaking through your raingear. As you pointed out, most people don't hike in puffy gear... but that just means you are taking it on and off when you stop.
In 2011, I was sheep hunting with a guy that took a down jacket as his insulation layer. He packed it in a dry bag, and had decent raingear, so thought he was set. It was a cold, wet trip for much of the time so we got to test that theory. The weak link was actually putting it on/taking it off. Because it was cold, we needed to layer up at glassing stops, but every time he did it would be briefly exposed to the weather and soak up more water. That combined with the usual issues of water wicking in around the cuffs of sleeves, dripping down the neckline, and putting it on over sweat soaked clothing, had reduced it to a soggy mass of uselessness by day two.
Looks cold doesn't he?
The newer DWR treated fabrics and down may very well reduce this issue enough to make it workable (and is something I'll be testing), but it's something to consider if you hunt or play in wet country. Wet = cold, and that's when you want your insulation layers to work.
Very good points YK...we'll see how it works out for me this year, hopefully I don't regret it...I know I won't regret the puffy pants as that saves 14 oz over the other ones and I rarely wear them except in the tent or bivying out, but the jacket we'll see, if nothing else it'll make for a good lightweight option for an extra insulation layer in the winter. I can see applications to have both...so now I do