Yes, if a volume control was linear, although the sound level is, as you say, dropping at a consistent rate, the human ear hears it as an accelerated amount in the first (or last depending which way we're going) section of the sweep. So a logarithmic control counter balances that quirk of human hearing to make it sound equal throughout the sweep.