If you're pushing the cone in and out nice and gently, I don't quite get what the 6 o'clock position is, but if you hear any scraping, you almost certainly have a duff driver. The magnet could have moved, as Spondon mentioned earliler, or more likely the speaker has been overdriven and the coating on the copper wire has expanded and is scraping.