Which of the materials are displaced? From your sample render I don't see any materials where it's obvious. Maybe bumpmap would do? Displacement is very memory intensive. And is you use it on large surfaces you force V-Ray to keep it in memory for longer - so you want to subdivide large displaced surfaces. But - you have a ~1million face model - it might very well be all the geometry that eat your memory.