That was my experience, I was checking parsing time, and it ended up being about identical with layer off versus geometry deleted. There may be issues if there are extra materials that are in that off layer, it may be that processing materials could slow things down too.
Found the thread. Man, I wish they would fix the old links!
http://www.chaosgroup.com/forums/vbulletin/showthread.php?62797-Lights-and-Render-Time
Maybe I'll bump this for Fernando, I am very curious to get an answer from someone who knows...