So, after a bit more reading, I came across a post where someone decresed the "latency" in preferences from 99 to 50. I never really tried the in-between settings. It seems to work much better now.
With the SB128, I can get away with a latency setting of about 35ms and it is much more playable, below 35ms the buffer is emptied faster than it is filled, which leads to a choppy sound. Oddly, going all the way down to 4ms seems to produce the same results as the full 99ms setting.. kind of misleading.