Yep, walk away. Anyone that far below the level of basic requirements will, unfortunately, almost always need a long time in order to catch up. Plus, the experience, as unpleasant as it may be, may give the drummer the impulse to address the issue i.e. you are actually doing them a favour.