I know of one guy that uses the CBR master (not sure if it's the same one you have or not) with the SV and 320mm rotor; he likes it. Personally, I find the stock master to work well with the SV caliper, EBC 320mm rotor, russel ss lines and the dime mod.
Basically, the master can change the feel, the fluid ratio and leverage (effort) required but it does absolutely nothing for braking power like the larger rotor (increased leverage due to the larger diameter rotor) and the SV caliper (larger friction surface). Feel is a bit of a personal preference but I certainly wouldn't worry about the brakes being too powerful - I don't believe that's an issue.
I say go ahead and if you don't like the feel then swap back in the stock MC. I strongly recommend getting the SV caliper adaptor and EBC rotor from Eaglemike as there are more than one EBC 320 listed for the KLR and the wrong one has had fitment issues.