I base mine off the ogive as mentioned and that is what I keep in my notes. I set the dies up for that to start my reloading session and then I'm good to go (I check occasionally but it holds fine), I don't worry the tips vary slightly so COAL drifts slightly.
Even measuring off the Ogive you'll see variances of a thousandth here and there. From what I've seen and documented, proper annealing goes a long ways in having consistent neck tension and maintaining the elasticity of the brass. If you have a mix of cases where the necks have work hardened to varying degrees, they won't spring back like they should and you'll sometimes see it reflected in inconsistent seating depths (.002-.005" is what I've seen). Have had this happen on 300 and 338 Normas as well as a few 28 Noslers. Using proper bushing dies as well as annealing has all but eliminated the problem.
One other thing to mention as well is the bearing surfaces on some bullets will vary. I know of a 300 win mag shooter than has as much as .0015 difference in bearing surface, and he sorts them in .0005" increments, adjusts his powder charge accordingly and has significantly reduced his velocity ES.