I'll give your tips with subdividing the geometry a try.
I messed around with the displacement subdivs and the edge lengths etc already and took it all the way down to 32subdivs. Seemed to provide the best level of detail vs speed. 16 just looked liked a mess although it rendered fast.
Instead of doing the whole render with displacement I took to the old trick of rendering the LC and IR passes without it on, saved them, ran them from file for the final render with the displacement on. Still took a while but much faster than calculating the LC and IR with the displacement on.
I hope that it is just a kink and they are planning on ironing out how vfsu handles CPU management with displacements once Chaos Group decides vfsu is worth their time again.