When they say to hold the remote firmly in your hand, they mean it.
If you keep your hand in the same position throughout the whole song as you had it in the beginning, making sure that the sensor bar is centered on or below the tv, you should be good to go, and will probably get a pretty good score. As far as the tiny little details in movements go, that doesn't really seem to be that big of an issue. But if you're getting all bad scores and are pretty sure that you've been hitting the movements right on time, check the remote signal, change the batteries, and keep the remote facing up in your right hand. Beyond that, it's a pretty easy, fun game to play by yourself or at a party.