So here my latest thought. Create a tool to draw a 2d rectangle on screen. Depending upon the direction of the rectangle it becomes a window or crossing box (or I have separate tools for each so direction it is drawn doesn't play a role).
Then I use view.PickRay on the corners of the 2d rectangle.
If I am using my rectangle as a window, I test vector from camera to each point in the model to see if it lies within the rectangle (x and y ray range test against my rectangle). If all the points of a face are pass the test that face and attached edges are in the selection (in my case I only care about faces, not edges). If any point of a face fails the test then that face is not in the selection.
If I am using my rectangle as a crossing box, then I want to test each face in my model to see if it intersects with any of the four rays from my 2d triangle. If any of the four rays intersect a face, then that face is included in the selection. This doesn't address edges, which is OK for my use.
If I wanted to get edges in crossing box, it would be great to convert the edge to 2d line on screen, and then see if the edge points either pass the ray range test, or interest the lines that make up the rectangle. Haven't dug into the API to see if that is even possible.)
An entirely different approach (again just for faces, and in this case just window, not crossing box) would be to have the script first copy every point in the model to a loose top level point. Then I would activate the built in SketchUp selection tool. The user would make a section like they normally do. This would create an array of 3d points. I could then test the nested faces. If all of the points of a given face are in the 3d point array, then that face would be selected. I would loop through all faces, and then throw away the temporary top level points used to make the array in the first place. This solution doesn't seem nearly as clean as the first solution.