Does quick magic charge work for physical attacks?

That is not really how the charge meter works mate - it is not the meter that is adding the power to the attack, the meter just displays for you convenience the current state of your stored power (charge) so that you know what the potential result of an attack might be.

The meter is actually showing you the results of charging that was accomplished by other actions that your character did, as the farther right the meter reads the more potential energy has been stored and is available for an attack.

For magic attacks the charge -- or rather the higher the charge -- indicates that higher damage will result from an attack or that more than one target will take damage during an attack.

If you look at the meter you will notice that there is a narrow red area marked on it - that is the target charge and if you get it to that level and then utilize the stored energy you will obtain maximum effect while spending minimum points. Just saying.