Can you delete items purchased for the bakery?

Yes; you simply click on the "DESIGN" box in the lower right-hand corner (with the hammer). Then you'll click on whichever item it is that you want to get rid of. Two small icons will come up, "STORAGE" and "SELL". Clicking on storage will simply remove said item from your view, but you can always retrieve it later. Clicking sell will allow you to sell the item back for a (very small) fraction of what you initially paid for it.

To retrieve items that you've previously stored, click again on the "DESIGN" box in the lower right-hand corner. A list of tabs will come up. Say you stored a few white chairs a while back that you didn't have room for, and now you'd like to put them back into your Bakery. Click on the second tab, the one with the chair on it. A stream of all of the chairs that Bakery Story has to offer will come up. Scroll through it until you get to the white chairs. You will notice that there is a pink triangle with a number on it. This is the amount of white chairs that you've stored. From there, just click on the picture of the white chair, and place it back into your Bakery. You won't get charged for it because you already paid for it once!