Here's a set of exercises which will force you to really get to know the &s of beats 2 and 4, and, by their frequent absence, beats 1 and 3:
Swing the 8th notes. Listening is important, here, so I suggest playing the exercises along with a recording, or with a metronome clicking on all four beats, or just on beats 1 and 3. Do you have the tom moves memorized yet?
Get the pdf