Ruby efficiency for large number of operations

drewmorgan

I have a text file that contains some geometry description in a non-SKP format.. I wrote a Ruby script to read from the file, convert as needed, and then add faces to the model based on the conversion.. This works great for when there's a small number of faces to add (say, < 100).. But the file I really want to load has more like 20000 faces. Observed behavior in the console is that the first 80 or so scream by, but then very quickly you can see it slowing down and it continue to slow down to the point where it it doing maybe 2 a second, and eventually slows to where it just hangs so I have to kill the instance of SketchUp that is running.

I wonder if there's something I can do with batching operations, or approaching it some different way to get the 20000 case to load in a reasonably fast manner (I'm not looking for 2 seconds or something, but rather a solution that would run to completion.. Even if it took 10 minutes, that'd be fine..)

I'm not completely new to Ruby and SKP, but I'm certainly no expert.. My general structure is like this:


def myFunc
  model = Sketchup.active_model
  #get filename
  myFile = File.open(filename, "r")

  for i in (0..numItems-1)
    #read in data for this item, split it into chunks, grab what I want...
    print "Working item; ", i, "\n"
    verts = []
    vert0 = Geom;;Point3d.new(computedX0, computedY0, computedZ0)
    vert1 = Geom;;Point3d.new(computedX1, computedY1, computedZ1)
    vert2 = Geom;;Point3d.new(computedX2, computedY2, computedZ2)
    vert3 = Geom;;Point3d.new(computedX3, computedY3, computedZ3)
    
    verts << vert0
    verts << vert1
    verts << vert2
    verts << vert3

    #status = model.start_operation('Add dheds', true)
    newGroup = model.entities.add_group
    newGroup.name = "somethingUseful" + i.to_s
    groupEnts = newGroup.entities

    plane = Geom;;fit_plane_to_points verts
    numVerts = verts.size
    for vertInd in 0..numVerts-1 do
      verts[vertInd] = verts[vertInd].project_to_plane plane
    end
    groupEnts.add_face verts

    #model.commit_operation
  end
end

I've experimented with the "start_operation" and "commit_operation" a bit inside and outside the loop - doesn't seem to make a significant difference..

I hope its just something I'm doing wrong (or poorly) and a little restructuring would make this work for me.. Any thoughts or suggestions would be greatly appreciated..

tt_su

You will find that generating a Geom::PolygonMesh and using Entities.fill_from_mesh to be many times faster. Entities.add_face do a lot of geometry merging which isn't needed for the type of operation you are doing here.

Note: it require you to have full control over the Entities collection you add to -* and it needs to be empty.

I have also found that pre-populating the PolygonMesh with the points first and caching the index you get in a hash also gives a further improvement. But you probably want to try without first and see if it's sufficient.

drewmorgan

Hmm, ok, I will look into that and give it a shot..

I wonder why it slows down though - generally that implies some data structure is getting larger and operations on that data structure are taking more time (to search for things, etc).. Is it just that the model is growing, and when I do add_face it uses the existing faces in the model somehow during that operation? I'm guessing that's what you meant by geometry merging.. Interesting..

I'll give the PolygonMesh a shot, and report back.. Thanks for the quick response!

jiminy-billy-bob

add_face compare the added face to the existing geometry in the active context. So the more geometry in that context, the slower it gets.

tt_su

@drewmorgan said:

Is it just that the model is growing, and when I do add_face it uses the existing faces in the model somehow during that operation? I'm guessing that's what you meant by geometry merging..

Yes. Entity.add_face (or line or anything) isn't just a simple method that blindly adds raw geometry, it tries to merge vertices and split edges etc - applying much of the SketchUp "stickyness" magic. It's good for tools where you add a face or edge at a time based on user input in a container with existing geometry, but for what you are going generating a large lump of geometry it quickly gets slow.

PolygonMesh does less of this - though it will merge vertices.

drewmorgan

Ok, that makes sense.. Unforuntately, though, I am still seeing essentially the same behavior.. It does seem to be better - it doesn't hang until like 1200 now, but still slows down to a crawl and eventually a hang..

I tried making a group for each item I want to add (which would be ideal) and also moved the group and fill_from_mesh call out of the loop, which didn't seem to make a difference..

Just saw tt_su's post indicating that PolygonMesh still merges vertices.. That must be what's causing the problem, but is there any way around it? 20000 meshes doesn't seem like such a big number to me, so I'm surprised its so problematic.. Is this just something I can't do with SKP? I'm actually not trying to generate a "good SketchUp model", but rather just trying to get a visualization.. Maybe Rhino or something would be better at this type of thing?

I could do some conversion outside of SKP I suppose, but it doesn't seem like a format conversion issue, it seems like a number of points issue, so I don't think that would provide any benefit.. Hmmm..

Aerilius

You mentioned you have the console open. At least in older versions, the console is slow. Sometimes I get the idea it rewrites the whole content with every line added. In SU14 it is supposed to be slightly better, and there is a secret clear command, but still it is best when you output only relevant data to the console and only during development (later only errors labeled with Our plugin's name).

drewmorgan

Well, reducing the console output helped.. I printed out only every 500th item, and over about 30 or 45 minutes it made it to 5500 (of 20000), which is the farthest I've seen it get yet.. But it still ended up not responding and having to be killed.. Maybe it just simply too many points for it to handle? I have to believe SKP supports large facet-count models reasonably well.. Does SKP performance degrade that significantly when you want to add a face to a large facet/point count model, or is it something specific with the Ruby API? As I mentioned before, I'm talking a total of ~20,000 rectangular facets, so I wouldn't normally even consider that a high facet count or high point count..

I might play with it a little more, but I'll probably resort to Rhino.. If I make progress on the SKP front, I will update the thread, and if there's any other suggestions or thoughts you all have, I'd still be happy to hear them, because the SketchUp solution would be far preferred.. Thanks!

tt_su

Can we see some sample code?

As Andreas mentioned, Ruby Console output significantly slow down code execution. Having the outliner open might also affect performance.

I'd suggest you test performance by not outputting to the console.

The absolutely fastest was I have found is:
Have a rough estimate of how many points and polygons you will generate and use this when you initialize the new PolygonMesh. Internally this reserves space for all the data so we avoid too many memory allocations.
Then generate all the unique points to be used, add them to the mesh and take the index returned by PolygonMesh.add_point and store it in a Hash for quick retrieval later on. Then add your polygons using the indices you built in the previous step.

You should be able to generate 20.000 triangles in less than a second or two.

fredo6

You should post your data file, so that we can try.
Personally, I think you can probably generate the 20,000 faces in less than a minute in SU14

Fredo

tt_su

@fredo6 said:

Personally, I think you can probably generate the 20,000 faces in less than a minute in SU14

For sure! But forget minutes - it should be around a second.

drewmorgan

I made up a small test program that uses the same general flow as my real case, and it produces the same result, so I can post that.. I also generated a simple test file.. This is not the actual file I'm trying to use, but just a test case generated with a script, so its obviously very regular and not very interesting..

Given what you guys are saying, I'm more confident its something wrong with my code, so looking forward to your responses.. Thanks a lot!

The Ruby script from my plugins dir

A test input file

fredo6

Thomthom was right, time is in seconds and the longest is to read the file and generate the Polygon Mesh (4 sec) whereas the face generation takes 0.5 seconds.

This is the output with your 20000 quad faces

[highlight=#ffff00:2915pieu]Reading File and creating Polygon Mesh....
Nfaces = 20000 - Creating Mesh time = 4.291246
Generation of Faces [20000] - Time = 0.517029[/highlight:2915pieu]

Here is the script for that. Note that it inserts a menu item in the Plugins menu
drewmorgan - test_skp_efficiency.rb

Fredo

tt_su

It would probably be possible to squeeze even more performance out of it, but it'd get increasingly more challenging.

If you where able to pre-process all the 3d points first and index them it would save more time. As PolygonMesh is doing a linear search for each point you add.
However, as the data is structured in your sample file this would probably be too slow to do in Ruby. But if you wrote a small Ruby C Extension to read the file and compute the unique set of points needed in C before converting it to Ruby it should take a couple of seconds of the time to create the PolygonMesh.

fredo6

I thought PolygonMesh would do that, based on C code. So doing it in Ruby would not help for performance (which is already quite good).

Fredo

tt_su

Yes, PolygonMesh will merge the points using C, but it's still doing it very inefficiently. It does a linear search each time I received a Point3d object. So when you grow to a large set of points this slows down noticeably.
Which is why you will get best performance out of it if you use add_point first before using add_polygon (with indices).

fredo6

@tt_su said:

Which is why you will get best performance out of it if you use add_point first before using add_polygon (with indices).

Do you mean that add_polygon will accept indexes instead of points? This is not obvious from the doc!
If so, then yes, we can probably improve performance.

Thanks for the tip

Fredo

tt_su

@fredo6 said:

Do you mean that add_polygon will accept indexes instead of points? This is not obvious from the doc!

Correct. I just checked the docs again - I was working from memory - and you're right. It's not obvious, it's only mentioned in add_point.

Using indices you can also control the soft/smooth or hidden properties of the edges bounding the polygons you add. See Entities.fill_from_mesh for more info on how that is controlled.

drewmorgan

Sorry I've not gotten back to you lately - I got pulled off this for a while..

The program you provided is definitely a lot faster and seems to work well, except for in the 20000 face test case, it causes SkecthUp to crash.. It seems to be during the fill_from_mesh call, because it prints out a message to console right before that happens.. But then I get a window that pops up saying "SketchUp Application has stopped working", and it crashes.. It seems to work on some test files I have with a lower number of faces, so maybe its just too many faces for a single polymesh? Does that seem reasonable, or do you think the crash is due to something else?

A couple of observations / things I've learned:

If I make each face be its own group (i.e. make a polymesh INSIDE the loop and fill_from_mesh inside the loop as well) it goes slooooooow. Ideally for my plugin, each one of these would be their own group (of course, what I originally was trying to do is a bit more complicated so each one would be more than a single plate).. I wonder if there's a way to get each one to be a separate group but still be reasonably fast? Based on our previous discussion, I'm guessing once you "fill_from_mesh", it then checks the points that are read in later will all the previous ones, whereas if you just put them all in the polymesh it doesn't do those checks?
If there's more than one implementation of the ruby function in the same directory, SKP will not provide an error message, but will pick one and use that.. This made me think I was losing my mind as I was modifying stuff left and right and not seeing any updates. Turns out when I created the file to post here, I just wrote a subset of my main plugins file to the same dir. All my mods were to my main file, but apparently, SKP was using the (unmodified) implementation in the subset file.. Argh. That took me far longer to figure out than it should have.

tt_su

@drewmorgan said:

The program you provided is definitely a lot faster and seems to work well, except for in the 20000 face test case, it causes SkecthUp to crash.. It seems to be during the fill_from_mesh call, because it prints out a message to console right before that happens.. But then I get a window that pops up saying "SketchUp Application has stopped working", and it crashes.. It seems to work on some test files I have with a lower number of faces, so maybe its just too many faces for a single polymesh? Does that seem reasonable, or do you think the crash is due to something else?

20000 faces shouldn't crash SU, no.
Did you submit the BugSplat? If you did, did you enter some details that I can use to look it up?