You can turn off the RGB and Alpha channel and that saves you a little bit of time. But V-Ray still has to calculate the light passes in order to get those channels, so you wont save yourself a lot of time. In my tests its one or two seconds per minute. On larger renders over the course of hours this might be helpful, but I doubt it.