Hello SketchUcators!
I have a lot of interest in this topic. While modeling in Sketchup can be bliss, creating impressive visual displays without a full blown renderer is very difficult. Being a game developer hobbyist, this limitation keeps me in Blender for most of my game related art.
Anyways, I was curious about raytest and what it's quirks were. As was stated earlier in this thread, limiting the raytest's scanning distance could be the solution to faster raycasting. Unfortunately, the ruby API doesn't offer this option; I suppose all rays are infinite (or my theory is the raytests are to the bounds of the scene.)
I decided to do a test: I made a ruby method > raytest_repeating(ray, repeatCount) and cast a ray along the y axis 10,000 times. With no obstructions, it completed it's process on average around 0.15 seconds. Introducing two cubes in separate groups to intersect with, the same method took about 0.68 seconds on average. Here's the interesting part. I placed 2000 groups along the y axis and ran the same method again. This process took ~938 seconds to complete! Keeping the same cubes in the scene and moving all but two of them out of the ray path dropped the calculation back down to the 0.68 average!
In conclusion, raytest must be spending time on all objects the ray interesects, even though it only returns the closest object. It's no wonder your occlusion process drops to a crawl in dense scenes, since I assume you're casting hemispherical rays into 'infinite space', colliding with EVERYTHING along the way! I believe a possible solution to your problem is to find the faces that are in the range of a raytest (IE, nearby faces that actually effect the shading) and to dynamically hide the rest of your scene as needed. Since the raytest can ignore hidden objects, it should speed up the overall process considerably. I ran one final test with the 2000 cubes, hiding all but 2 of them. While the process was slower, 10,000 rays came back in 5.2 seconds, which is far better than 938!
I hope my findings can assist you on this wonderful creation of yours!
-Thomas Pendergrass