To rob an establishment, you need to do the same thing, just aim your weapon at the proprietor. You won’t attract any attention unless one of you shoots. However, like strangers, there are shop owners that don’t take well against robbers. Some may even ignore your demand and threats, and even shoot you, causing a unwanted commotion.
Backroom Rackets
There are certain shops that have suspicious backroom or basement operations. To be access the unique robbery option during the hold up, you should at least observe the said operation in advance by peeking through a window or talking to a stranger. You may even get a tip of these shady dealings by helping escaped prisoners you’ll randomly encounter on the road. Once done, you’ll have the option to order the shop owner to take you to the back room or basement. More details of these missions
here.