Smaller guys generally always want to bigger.....I was one, 6'00 160....worked put hard and am now 190. I wish I was 160 again. I would concentrate on being as fit as you can. Big doesn't mean you can handle heavy loads better. So many other factors that come into play. I know many small guys who can pack with the best of them. Take a look at most of the Alaskan guides, lean dudes that are mountain tough. Those guys don't even work out, they just work hard.
Interesting. I'm similar at 6', but trying to go the other way. This spring I was 176 and am now down to about 157, and would like to drop a few more. The mountains aren't about beach muscle, it's all lower body. It's about being lean and you can't get that way easily. Lots and lots of work.