The problem is that V-Ray only recognizes the geometry as it is in the very first scene. Hidden geometry, layer visibility, and Face-Me component's direction in the following scenes are ignored.
The animation is basically limited to camera position.
Check out the ASGVis forum for more info on V-Ray animation. I've never really given it a go. You might find some useful info there.