I always wear my merino boxers when hunting. I may or may not wear a base layer.
Earlier this year I was hunting Coues in MX. Some of us were helping glass for another hunter who had not yet tagged out. One day, late morning, I looked to my right. The hunter was bent over and removing clothes. He was not wearing underwear under his base layer. This hunter was also up there in years and gravity was unkind. That was a most unpleasant site.
Do not be that guy.