It is tricky, for sure, to get multi-image PhotoMatch projects to work.
One thing that may cause problems is that no lens is perfect.
Most wide angle lenses get a barrel, and most teles get a pillow radial distortion. This is usually more pronounced in zoom lenses than in primes.
This should really be corrected to get good results, and that is what real photogrammetry apps do.
It may be done with PS, AcDSee or other tools like PTgui and other panorama stitchers.
Another thing that may cause problems is that house corners aren't always 90 degrees, particularly in city streets. Very often they follow the street instead of being perfectly squared corners. As SU relies on such corners being 90 degrees it fails when trying to force the photo into a model with 90deg corners.
Using cropped photos is another source for problems - always use the full image.
Always try to avoid ground lines for aligning perspective. The ground is rarely completely level (at least here in Norway 
.
PhotoMatch is great fun to use, and very fast, but it does have its limitations.
There is a real photogrammetry solution for SU now, PhotoScan from tgi3D. I'm testing it right now (30 days full version trial), and it is very promising. It uses triangulation to place points in 3D space, and then allows you to build the models quite easily inside SU. Like this partly built boat made from 5 photos (too closely for capturing the full boat).
