Hi, I don't have this problem - yet
Clipping a noise signal can make it sound grainy. I once thought my eurorack Doepfer A118 noise generator was broken, but I found out the signal was clipping somewhere in my setup, a simple twist of a knob solved the problem.
What happens if you lower the noise fader?
Perhaps try to simplify everything, if you hadn't tried already, switch off brute factor, flatten the filter, zero LFO to Amp modulation etc..
Which CV input did you use; pitch / filter / Amp (VCA)?