I have partly figured out what was going on here.
Basically my model was a rack and pinion. I was trying to change all settings in the UI to try to get it to work but my mistake was not looking at the collision wireframe in the simulation. It seems that the Compound From CD collision shape in the MSPhysics UI is not getting either the rack or the pinion gear correct when the simulation is done.
Perhaps this will be resolved if I try changing some other simulation parameters like world scale maybe?