I'm in a similar situation, I set a goal for myself to not harvest anything less than a 6 point bull. I've been practicing that for 7 years and in that time I've killed four 6 point bulls and I lost one 6 point bull that I never found.
For me deciding to only harvest mature bulls was easy, I've been hunting elk a long time, finding and getting into range has never been an issue. I'm not saying it's easy but if I shot the first elk I had a chance at every year I wouldn't get to do much elk hunting. I don't have a sliding scale either so even on the last day I'm not shooting a dink, I just made that decision and have stuck to it. I have also passed up several small 6 point bulls over the years as well, I am just looking for something that makes me happy to shoot. I run a lot of cameras and know there are some big bulls in the areas I hunt which has helped me. The bull in my profile pic is still alive as far as I know in one of the areas I run cams on public land OTC here in Oregon.
Stopping hunting small bulls has made me a much better elk hunter, I have learned a lot in the last 7 years, one of the biggest things is that I am a lot more carefree and stress-free when calling or stalking an elk. I do things that I probably wouldn't do if I only wanted to harvest one elk. One thing that I do is hunt super aggressively, I do this because 70% of the bulls I encounter are not shooter bulls to me, this means it's not a big deal to bump them or blow the opportunity. More often than not I get myself a shot opportunity and pass on it though.
My wife gets a cow tag and both of my boys get elk and deer tags. It's a rare occasion when we don't have a freezer full.
My oldest wants to archery hunting elk instead of rifle, he did it for the first time this fall and got to draw back his bow and a big 6 point bull which he thought was awesome. For me, I just really love elk hunting, I love the interaction and love to call and get close to elk. This year my two boys 11 & 14 called the bull in that I shot and me and my youngest called in several bulls while my oldest tried to get a shot at them.