Which of the materials are displaced?
From your sample render I don't see any materials where it's obvious. Maybe bumpmap would do?
Displacement is very memory intensive. And is you use it on large surfaces you force V-Ray to keep it in memory for longer - so you want to subdivide large displaced surfaces.

But - you have a ~1million face model - it might very well be all the geometry that eat your memory.