Hmmm, I've had the opposite experience - I've found the B really did just get in the way, and I never, ever needed it for anything I was playing - I have a Detuner to drop the D, and that's all I need.
If you love the B and, like Grangur, you'd miss it, then yeah, ditch the 4s and go for it.
EDIT I meant to say, in a same-but-opposite-way, I shifted the 5 on because I wasn't playing it, and it had no sentimental value to me.