Gems are the 'real' currency of Bakery Story. By that, it means that you have to purchase them with your own money. It is the way that TeamLava makes money from the game, and is the only reason they are able to offer it for free to begin with. To really do what you like to your bakery, you have to splash the cash.
The gems icon is second from the left at the top of the screen.
How to Get Free Gems
It is possible to get gems for free, without being scammed. TeamLava gives free gems away via competitions on their forums, and in the past have done so when you get and play any of their other games, like Restaurant Story or Fashion Story.
The best way to get gems is to cook as much as possible. You are rewarded with gems at level 2 and level 4 mastery of each recipe.
You can also get gems by achieving a gold medal. The easiest way to get gems, however, is to buy them. You can do this (with real money!) by tapping the blue gem icon.
It should be said that you will need to buy Gems at some point if you want to experience everything that there is to see in Bakery Story.