Having done our VAT return for quite some time, I can categorically say that company's costs are rarely VAT-free!
The only time it's actually VAT exempt, as opposed to zero-rated, is for services such as insurance, subscriptions etc, or from businesses that aren't VAT registered due to being exempt (i.e. below the threshold or if they only deal in exempt goods) - or at least that's my understanding!
To make things entertaining, we also sell zero-rated stuff which can also be a bit of a minefield. Books and kid's clothes are 0% (i.e. tax is levied, but at 0% 🤪), same with food plants. However, if it's classed as ornamental it's VATed at 20% - so a bay bush is 0, but a standard bay (i.e. a single stalked bay) is 20%.
I'm sure they will make things more complicated at some point, cos who would want a nice easy tax system that everyone understands?