• Login
sketchucation logo sketchucation
  • Login
⚠️ Libfredo 15.4b | Minor release with bugfixes and improvements Update

Ruby efficiency for large number of operations

Scheduled Pinned Locked Moved SketchUp Discussions
sketchup
34 Posts 5 Posters 1.2k Views 5 Watching
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • D Offline
    drewmorgan
    last edited by 8 Apr 2014, 12:25

    I have a text file that contains some geometry description in a non-SKP format.. I wrote a Ruby script to read from the file, convert as needed, and then add faces to the model based on the conversion.. This works great for when there's a small number of faces to add (say, < 100).. But the file I really want to load has more like 20000 faces. Observed behavior in the console is that the first 80 or so scream by, but then very quickly you can see it slowing down and it continue to slow down to the point where it it doing maybe 2 a second, and eventually slows to where it just hangs so I have to kill the instance of SketchUp that is running.

    I wonder if there's something I can do with batching operations, or approaching it some different way to get the 20000 case to load in a reasonably fast manner (I'm not looking for 2 seconds or something, but rather a solution that would run to completion.. Even if it took 10 minutes, that'd be fine..)

    I'm not completely new to Ruby and SKP, but I'm certainly no expert.. My general structure is like this:

    
    def myFunc
      model = Sketchup.active_model
      #get filename
      myFile = File.open(filename, "r")
    
      for i in (0..numItems-1)
        #read in data for this item, split it into chunks, grab what I want...
        print "Working item; ", i, "\n"
        verts = []
        vert0 = Geom;;Point3d.new(computedX0, computedY0, computedZ0)
        vert1 = Geom;;Point3d.new(computedX1, computedY1, computedZ1)
        vert2 = Geom;;Point3d.new(computedX2, computedY2, computedZ2)
        vert3 = Geom;;Point3d.new(computedX3, computedY3, computedZ3)
        
        verts << vert0
        verts << vert1
        verts << vert2
        verts << vert3
    
        #status = model.start_operation('Add dheds', true)
        newGroup = model.entities.add_group
        newGroup.name = "somethingUseful" + i.to_s
        groupEnts = newGroup.entities
    
        plane = Geom;;fit_plane_to_points verts
        numVerts = verts.size
        for vertInd in 0..numVerts-1 do
          verts[vertInd] = verts[vertInd].project_to_plane plane
        end
        groupEnts.add_face verts
    
        #model.commit_operation
      end
    end
    
    

    I've experimented with the "start_operation" and "commit_operation" a bit inside and outside the loop - doesn't seem to make a significant difference..

    I hope its just something I'm doing wrong (or poorly) and a little restructuring would make this work for me.. Any thoughts or suggestions would be greatly appreciated..

    1 Reply Last reply Reply Quote 0
    • T Offline
      tt_su
      last edited by 8 Apr 2014, 13:05

      You will find that generating a Geom::PolygonMesh and using Entities.fill_from_mesh to be many times faster. Entities.add_face do a lot of geometry merging which isn't needed for the type of operation you are doing here.

      Note: it require you to have full control over the Entities collection you add to -* and it needs to be empty.

      I have also found that pre-populating the PolygonMesh with the points first and caching the index you get in a hash also gives a further improvement. But you probably want to try without first and see if it's sufficient.

      1 Reply Last reply Reply Quote 0
      • D Offline
        drewmorgan
        last edited by 8 Apr 2014, 13:35

        Hmm, ok, I will look into that and give it a shot..

        I wonder why it slows down though - generally that implies some data structure is getting larger and operations on that data structure are taking more time (to search for things, etc).. Is it just that the model is growing, and when I do add_face it uses the existing faces in the model somehow during that operation? I'm guessing that's what you meant by geometry merging.. Interesting..

        I'll give the PolygonMesh a shot, and report back.. Thanks for the quick response!

        1 Reply Last reply Reply Quote 0
        • J Offline
          jiminy-billy-bob
          last edited by 8 Apr 2014, 14:06

          add_face compare the added face to the existing geometry in the active context. So the more geometry in that context, the slower it gets.

          25% off Skatter for SketchUcation Premium Members

          1 Reply Last reply Reply Quote 0
          • T Offline
            tt_su
            last edited by 8 Apr 2014, 14:27

            @drewmorgan said:

            Is it just that the model is growing, and when I do add_face it uses the existing faces in the model somehow during that operation? I'm guessing that's what you meant by geometry merging..

            Yes. Entity.add_face (or line or anything) isn't just a simple method that blindly adds raw geometry, it tries to merge vertices and split edges etc - applying much of the SketchUp "stickyness" magic. It's good for tools where you add a face or edge at a time based on user input in a container with existing geometry, but for what you are going generating a large lump of geometry it quickly gets slow.

            PolygonMesh does less of this - though it will merge vertices.

            1 Reply Last reply Reply Quote 0
            • D Offline
              drewmorgan
              last edited by 8 Apr 2014, 14:40

              Ok, that makes sense.. Unforuntately, though, I am still seeing essentially the same behavior.. It does seem to be better - it doesn't hang until like 1200 now, but still slows down to a crawl and eventually a hang..

              I tried making a group for each item I want to add (which would be ideal) and also moved the group and fill_from_mesh call out of the loop, which didn't seem to make a difference..

              Just saw tt_su's post indicating that PolygonMesh still merges vertices.. That must be what's causing the problem, but is there any way around it? 20000 meshes doesn't seem like such a big number to me, so I'm surprised its so problematic.. Is this just something I can't do with SKP? I'm actually not trying to generate a "good SketchUp model", but rather just trying to get a visualization.. Maybe Rhino or something would be better at this type of thing?

              I could do some conversion outside of SKP I suppose, but it doesn't seem like a format conversion issue, it seems like a number of points issue, so I don't think that would provide any benefit.. Hmmm..

              1 Reply Last reply Reply Quote 0
              • A Offline
                Aerilius
                last edited by 8 Apr 2014, 15:38

                You mentioned you have the console open. At least in older versions, the console is slow. Sometimes I get the idea it rewrites the whole content with every line added. In SU14 it is supposed to be slightly better, and there is a secret clear command, but still it is best when you output only relevant data to the console and only during development (later only errors labeled with Our plugin's name).

                1 Reply Last reply Reply Quote 0
                • D Offline
                  drewmorgan
                  last edited by 8 Apr 2014, 17:42

                  Well, reducing the console output helped.. I printed out only every 500th item, and over about 30 or 45 minutes it made it to 5500 (of 20000), which is the farthest I've seen it get yet.. But it still ended up not responding and having to be killed.. Maybe it just simply too many points for it to handle? I have to believe SKP supports large facet-count models reasonably well.. Does SKP performance degrade that significantly when you want to add a face to a large facet/point count model, or is it something specific with the Ruby API? As I mentioned before, I'm talking a total of ~20,000 rectangular facets, so I wouldn't normally even consider that a high facet count or high point count..

                  I might play with it a little more, but I'll probably resort to Rhino.. If I make progress on the SKP front, I will update the thread, and if there's any other suggestions or thoughts you all have, I'd still be happy to hear them, because the SketchUp solution would be far preferred.. Thanks!

                  1 Reply Last reply Reply Quote 0
                  • T Offline
                    tt_su
                    last edited by 8 Apr 2014, 19:39

                    Can we see some sample code?

                    As Andreas mentioned, Ruby Console output significantly slow down code execution. Having the outliner open might also affect performance.

                    I'd suggest you test performance by not outputting to the console.

                    The absolutely fastest was I have found is:
                    Have a rough estimate of how many points and polygons you will generate and use this when you initialize the new PolygonMesh. Internally this reserves space for all the data so we avoid too many memory allocations.
                    Then generate all the unique points to be used, add them to the mesh and take the index returned by PolygonMesh.add_point and store it in a Hash for quick retrieval later on. Then add your polygons using the indices you built in the previous step.

                    You should be able to generate 20.000 triangles in less than a second or two.

                    1 Reply Last reply Reply Quote 0
                    • fredo6F Offline
                      fredo6
                      last edited by 8 Apr 2014, 19:43

                      You should post your data file, so that we can try.
                      Personally, I think you can probably generate the 20,000 faces in less than a minute in SU14

                      Fredo

                      1 Reply Last reply Reply Quote 0
                      • T Offline
                        tt_su
                        last edited by 9 Apr 2014, 09:21

                        @fredo6 said:

                        Personally, I think you can probably generate the 20,000 faces in less than a minute in SU14

                        For sure! But forget minutes - it should be around a second.

                        1 Reply Last reply Reply Quote 0
                        • D Offline
                          drewmorgan
                          last edited by 9 Apr 2014, 12:26

                          I made up a small test program that uses the same general flow as my real case, and it produces the same result, so I can post that.. I also generated a simple test file.. This is not the actual file I'm trying to use, but just a test case generated with a script, so its obviously very regular and not very interesting..

                          Given what you guys are saying, I'm more confident its something wrong with my code, so looking forward to your responses.. Thanks a lot!


                          The Ruby script from my plugins dir


                          A test input file

                          1 Reply Last reply Reply Quote 0
                          • fredo6F Offline
                            fredo6
                            last edited by 9 Apr 2014, 21:17

                            Thomthom was right, time is in seconds and the longest is to read the file and generate the Polygon Mesh (4 sec) whereas the face generation takes 0.5 seconds.

                            This is the output with your 20000 quad faces

                            [highlight=#ffff00:2915pieu]Reading File and creating Polygon Mesh....
                            Nfaces = 20000 - Creating Mesh time = 4.291246
                            Generation of Faces [20000] - Time = 0.517029[/highlight:2915pieu]

                            Here is the script for that. Note that it inserts a menu item in the Plugins menu
                            drewmorgan - test_skp_efficiency.rb

                            Fredo

                            1 Reply Last reply Reply Quote 0
                            • T Offline
                              tt_su
                              last edited by 10 Apr 2014, 09:40

                              It would probably be possible to squeeze even more performance out of it, but it'd get increasingly more challenging.

                              If you where able to pre-process all the 3d points first and index them it would save more time. As PolygonMesh is doing a linear search for each point you add.
                              However, as the data is structured in your sample file this would probably be too slow to do in Ruby. But if you wrote a small Ruby C Extension to read the file and compute the unique set of points needed in C before converting it to Ruby it should take a couple of seconds of the time to create the PolygonMesh.

                              1 Reply Last reply Reply Quote 0
                              • fredo6F Offline
                                fredo6
                                last edited by 10 Apr 2014, 15:36

                                I thought PolygonMesh would do that, based on C code. So doing it in Ruby would not help for performance (which is already quite good).

                                Fredo

                                1 Reply Last reply Reply Quote 0
                                • T Offline
                                  tt_su
                                  last edited by 17 Apr 2014, 20:44

                                  Yes, PolygonMesh will merge the points using C, but it's still doing it very inefficiently. It does a linear search each time I received a Point3d object. So when you grow to a large set of points this slows down noticeably.
                                  Which is why you will get best performance out of it if you use add_point first before using add_polygon (with indices).

                                  1 Reply Last reply Reply Quote 0
                                  • fredo6F Offline
                                    fredo6
                                    last edited by 17 Apr 2014, 21:04

                                    @tt_su said:

                                    Which is why you will get best performance out of it if you use add_point first before using add_polygon (with indices).

                                    Do you mean that add_polygon will accept indexes instead of points? This is not obvious from the doc!
                                    If so, then yes, we can probably improve performance.

                                    Thanks for the tip

                                    Fredo

                                    1 Reply Last reply Reply Quote 0
                                    • T Offline
                                      tt_su
                                      last edited by 17 Apr 2014, 22:08

                                      @fredo6 said:

                                      Do you mean that add_polygon will accept indexes instead of points? This is not obvious from the doc!

                                      Correct. I just checked the docs again - I was working from memory - and you're right. It's not obvious, it's only mentioned in add_point.

                                      Using indices you can also control the soft/smooth or hidden properties of the edges bounding the polygons you add. See Entities.fill_from_mesh for more info on how that is controlled.

                                      1 Reply Last reply Reply Quote 0
                                      • D Offline
                                        drewmorgan
                                        last edited by 18 Apr 2014, 14:02

                                        Sorry I've not gotten back to you lately - I got pulled off this for a while..

                                        The program you provided is definitely a lot faster and seems to work well, except for in the 20000 face test case, it causes SkecthUp to crash.. It seems to be during the fill_from_mesh call, because it prints out a message to console right before that happens.. But then I get a window that pops up saying "SketchUp Application has stopped working", and it crashes.. It seems to work on some test files I have with a lower number of faces, so maybe its just too many faces for a single polymesh? Does that seem reasonable, or do you think the crash is due to something else?

                                        A couple of observations / things I've learned:

                                        1. If I make each face be its own group (i.e. make a polymesh INSIDE the loop and fill_from_mesh inside the loop as well) it goes slooooooow. Ideally for my plugin, each one of these would be their own group (of course, what I originally was trying to do is a bit more complicated so each one would be more than a single plate).. I wonder if there's a way to get each one to be a separate group but still be reasonably fast? Based on our previous discussion, I'm guessing once you "fill_from_mesh", it then checks the points that are read in later will all the previous ones, whereas if you just put them all in the polymesh it doesn't do those checks?

                                        2. If there's more than one implementation of the ruby function in the same directory, SKP will not provide an error message, but will pick one and use that.. This made me think I was losing my mind as I was modifying stuff left and right and not seeing any updates. Turns out when I created the file to post here, I just wrote a subset of my main plugins file to the same dir. All my mods were to my main file, but apparently, SKP was using the (unmodified) implementation in the subset file.. Argh. That took me far longer to figure out than it should have.

                                        1 Reply Last reply Reply Quote 0
                                        • T Offline
                                          tt_su
                                          last edited by 18 Apr 2014, 15:56

                                          @drewmorgan said:

                                          The program you provided is definitely a lot faster and seems to work well, except for in the 20000 face test case, it causes SkecthUp to crash.. It seems to be during the fill_from_mesh call, because it prints out a message to console right before that happens.. But then I get a window that pops up saying "SketchUp Application has stopped working", and it crashes.. It seems to work on some test files I have with a lower number of faces, so maybe its just too many faces for a single polymesh? Does that seem reasonable, or do you think the crash is due to something else?

                                          20000 faces shouldn't crash SU, no.
                                          Did you submit the BugSplat? If you did, did you enter some details that I can use to look it up?

                                          1 Reply Last reply Reply Quote 0
                                          • 1
                                          • 2
                                          • 1 / 2
                                          • First post
                                            Last post
                                          Buy SketchPlus
                                          Buy SUbD
                                          Buy WrapR
                                          Buy eBook
                                          Buy Modelur
                                          Buy Vertex Tools
                                          Buy SketchCuisine
                                          Buy FormFonts

                                          Advertisement