Yup, very true. I know that Fast Fret is a shortcut, in a way, and using lemon oil is the done thing, but I find that FF is all right, and quick, and slightly less hassle, for a fingerboard that's simply a bit dry as opposed to needing an overhaul. Incidentally, the directions on the Fast Fret container state the following: "Rub stick firmly over strings and fingerboard, before and after playing", which seems to imply more than just getting a bit of oil on the wood while cleaning the strings.
I knew about the fret polishing cloths. I need to buy some, in fact