There are a few ways to get Professor Pester and his Ruffians out of your garden, or just get them to stop messing with your pinatas:
1. Have a Limeoceros in your garden when he comes. The pinata rams into Professor Pester and he flies out of your garden. You can't direct the pinata to Professor Pester, so the Limeoceros does it itself.
2. You can buy Decoys at Costalot's shop, but they are sold at a very high price. Professor Pester always tries to hit the Decoy, and not your pinatas or your valuable items. I suggest having at least four Decoys in the garden at a time, just to be safe.
3. Pay Professor Pester and his Ruffians to leave. Press X when you're hovering over any of them, and you can pay them to leave. Paying Professor Pester at least 700 Chocolate Coins should get him to leave, and the Ruffians usually settle for just 400 Chocolate Coins.