How would I sand down a spindle that is on my boat trailer?

There are a couple gouges on the spindle from a recently spun bearing. The spindle looks good, but due to the small grooves in it...I can't put the replacement bearing on the spindle. It catches on the grooves. Should I emery paper sand it? Or is there some sort of mechanism I can put on my drill to sort of mill down it evenly? I don't want to just take the orbital sander to it, I would think that would create flat spots. Thanks for your help doods