Great to see you Andy. You finally managed to claw your way out from under a pile of wriggly grandchildren. Are any of them old enough to sweep out your basement yet?
If you are willing to spend that much effort then it would be worth drilling oversize and plugging. You wouldn't then have the metal to "worry" the finishing process.