How do you get Simon and Mallory to join you again?

It depends on what part you are on. You usually have to accomplish something first. Go to the book in the bottom right corner and click map. On your top screen there should be a message saying what you need to do. After you do this, they are most likely to rejoin you.

Well I thought that it is you kill the huge troll that's killing the tree

You have to defeat the mole troll and then the tree folk will warp you to Arthur spiderwick speak to Arthur spiderwick and he will give you a bloomer sprite to collect flowers and then go to the swamp and place a flower down to collect the skimmer sprite and the one in the tunnels then the 2nd 3rd 4th parts of the river have junevill river trolls but the second part 1st part of water has a river troll the bigger one use your confuse sprites on the small troll as it will not work on the bigger one.that way you will have all of the kids to attack the bigger river troll.