As I see, over the help forums, they also tried - just the file size did not turn out.
First of all, purge everything not used in the file. Go to window > model info > statistics and at the bottom...
If this is not enough, best is to start with image textures. Make sure that if you used photos as textures, right click on the face and make texture unique. Here is a plugin that can help doing on the whole file.
http://forums.sketchucation.com/viewtopic.php?t=41441
If you already exported to kmz, rename the file extension to zip and open it. Inside, there will be some folder with the images. If there are some really big ones, edit them and make them smaller and also compress. Here is a plugin that can do it in your model, too.
http://forums.sketchucation.com/viewtopic.php?t=40720
A "normal" model for GE (to stream efficiently on the end user's computer) should not be more than 1 Mb. Maybe 2 Mb if otherwise really complex. Especially if there are many 3D buildings in the neighbourhood already.