The MODULATOR does introduce some difference to each UNISON voice. It can be to avoid very high clickings in some cases. I can't tell you, if this is normal, if one can say talk about normal in this.
You can try to set the PHASE MOD Sync/ Retrig to Mod Osc.
Also you can keep the Key setting for Sync/ Retrig in the PHASE MOD, and try to drop the MODULATOR modulation in your sound, and use the PHASE DISTORTION Amount and Target settings instead. Try to start with the Amount at twelve o'clock position, and perhaps Psudo PW as Target. And try Self as Sync/ Retrig for PHASE MOD. But in generel exsperiment as it also depend on which of the very different sounds you get now, that you like.
Keep in mind, that what you don't like now actually also contribute to the sound as it is. Try to set the PHASE MOD Sync/ Retrig to Mod Osc in this example, then i guess, you'll know, what i mean.