It seems like quite a problem for you. It's a horrible feeling trying to influence others to adjust their tempo during a performance, and once it's playing out wrong, it's harder to gauge exactly where it should be... then as you say, with thinking about that, it's easy to forget where you are in the song.
Is communication generally good within the band? I wonder if you're all on the same page about what the tempos of your songs should be... and whether it's something for the band to work on.
How does the tempo thing compare at rehearsals vs playing live? I'm sure it's common for bands to tend to rush through the set with the adrenaline boost of being on stage. Then perhaps being aware of that on the first few songs, a player may then start one off too slowly in an attempt to redress the balance?