Well, Britain's Got Talent occasionally hosts musicians, but it's a variety show... I believe that X-Factor needs to have huge viewing and participant application numbers in order to compensate for the high salaries and big production costs while maintaining global appeal. The world's Joe Public is quite familiar with "songs", can relate to them much easier through life experiences and therefore much more attracted to these sort of competitions. Also, there's a proportionally higher number of people who can passably sing, than play an instrument. Well, they think they can.
A show based on musical performance - which would basically entail instrumentals - would mostly appeal only to musicians, their friends and families, and maybe some "guitar god" wannabes who feel they have lost their opportunity of becoming some sort of a musician on their turn... It would simply not have the mass appeal that a pop song performance could attract, whether terrible, mediocre, or great. You must keep in mind that this, and other similar shows, are Not really about the existence and nurturing of talent, but about the money. Simon says: "Show me da money!"