No too far from Philly
...but this one thing I do, forgetting those things which are behind,
...and reaching forth unto those things which are before,...
...but am I slowing other players down by doing that?
You are not playing a FPS, so I would doubt it. Well, what have you seen in your games so far? Have you noticed lag or delay? If not, I would say not to worry about it.
The best thing you can do to speed up online games time is to be quick and decisive with you moves and turns because that's where the seconds really add up. On two separate occasions I have played a round with a VGGer from Belgium where it was just us plus one AI and we got done both games in just 35 minutes each.