When you enter a town, the party may enter private action mode if you have made sufficient progress in the story. This will occur when you come into contact with specific points denoted by whistle icons, which causes all the characters to scatter throughout the town. If you wish to see these side stories with your allies unfold, you will first have to find your party members. Empty diamonds depicting the number of private actions you can view will display in the bottom right-hand corner of the screen, and will fill in every time you view one. This way, the likelihood you will accidentally skip over a private action will be drastically reduced.
Depending on the content of the private action, or sometimes the choices you make as Fidel during it, you can improve your standing with the party members involved. You can always exit private action mode in much the same manner as you entered it - by walking into gold circle. After completing the available private actions and after regrouping with your allies, you can sleep at the town’s inn to “reset” the available private action(s). Keep doing this until you exhaust all the available PAs in the location at that given point in the game.
There are also random Private Actions that will trigger automatically when you pass through specific fixed points in a field or town. This doesn't count towards the PAs you can trigger in towns but does give good interaction between the current members of the party.