Script slows down exponentially - help?

Chris Fullmer

Hi folks, I am re-writing my first script, which iterates through the model and takes the first entity, finds everything connected to it, and then groups that. Then moves on to the next entity.

My script is working great.

But the problem is that it gets slower the more entities I try to do at once.

500 entities < .1 seconds
2000 ents = 4 seconds
5000 ents = 20 seconds

So I thought I could break up my "grouping" into arrays of 500 elements, which goes so fast. So I send an array of 500 entities to my grouping method. The first 500 element array flys through, then the second takes 4 times as long, then the 3rd takes another 4 times as long. On and on. I'm unsure why the method processes more slowly the more times it gets run.

Does this make any sense? Here is the code. I've got a lote of "Time" stuff in there because I am testing it for speed:

module Clf_loosegroups

	# Make groups of all geometry in the selection set or active_entities.
	def Clf_loosegroups.main
		now = Time.now
		model = Sketchup.active_model
		@entities = model.active_entities
		sel = model.selection
		if sel[0]
			ents = sel.to_a
		else
			ents = @entities.to_a
		end
		
		if Sketchup.version_number < 7000000
			model.start_operation "Make Groups"
		else
			model.start_operation "Make Groups", true
		end
		
		chunk = 1000
		count = (ents.length/chunk)+1
		new_arr = []
		now21 = 0
		now22 = 0
		count.times do |e|
			place_number = (e*chunk)

			new_arr = ents[place_number,chunk]
			puts new_arr.length
			Clf_loosegroups.maker(new_arr)
		end
		
		model.commit_operation
		now2 = Time.now
		puts now2-now
	end


	def Clf_loosegroups.maker(ents)
		nowa = Time.now
		puts ents.length
		while ents.length > 0
			if (ents[0].is_a? Sketchup;;Edge) || (ents[0].is_a? Sketchup;;Face)
				to_group = ents[0].all_connected
				ents = ents - to_group
				group = @entities.add_group(to_group)
			else
				ents = ents - [ents[0]]
			end
		end
		nowb = Time.now
		puts "---------"
		puts nowb-nowa
		
		puts "---------"
	end # maker
end

Any thoughts?

Chris Fullmer

Hey, after some more playing, I think I've found the culprit.

SketchUp gets slower at creating groups the more groups it already has in the model. So where I said my script will work with 1000 entities in .1 seconds or whatever, well thats only true when SketchUp does not have any groups already in the model.

If SketchUp already had 5000 groups in the model, then that same process takes up to a minute (some of these times are only close approximations).

So it is not my script's fault, but it is in SketchUp's ability to create groups quickly.

Chris

AdamB

Have you tried (programmatically) hiding existing groups?

Adam

Chris Fullmer

No I have not. That might prove to be interesting! I'll give it a try and report back, thanks Adam.

Chris

Chris Fullmer

Bummer, that did not seem to help. I used group.hidden=true and adding groups still slowed down the more groups there were in the model.

Chris

thomthom

That's interesting... hmm...

AdamB

Just for completeness, try hiding them in a hidden Layer.

I'm presuming its not "creating groups quickly" per se but the need to check every existing group for interaction with the new one that is taking time.

So if you have N groups in a flat hierarchy is it as slow as N groups in a deeply nested hierarchy (where SU can early out on these tests).

Adam

todd burch

Do you have Outliner open while doing this?

Chris Fullmer

@Todd, nope, I've got it closed.

@Adam, I'll try putting them on a new hidden layer. That could be worth trying too.

I did try grouping them each, then adding the first chunk to a single group, hoping SU would only recognize the 1 big group and then go faster in the next chunk of groups to process - no luck. It was just as slow.

So I'll try the hidden layer too in an hour or two.

Chris

Didier Bur

Hi,
Maybe a problem with the memory. I recently developped a script that creates a bunch of groups on the fly, and first I noticed that entities.add_group is buggy, and second that the more groups there are, the slower the process is.
So I made an alias of the method "add_group" of the "entities" class, and added the garbage collector in this alias. No more bug splats and a quicker process.
Put that in your code so the memory is "cleaned" from time to time:

GC.start

Hope this helps,

Chris Fullmer

Hi Didier, is there also a file to include? or do I really just need to add the line

GC.start

somewhere in my script and it will just work?

Thanks so much, sounds like the problem I am experiencing and I'll be glad to see a fix for it!

Chris

Didier Bur

Hi Chris,

Putting GC.start just before the suspicious instructions will do.
No file to include, this is pure Ruby.
See: http://www.rubycentral.com/book/ref_m_gc.html
Regards,

thomthom

Components are not affected by this?

Didier Bur

Hi,
No, components aren't affected. The add_group method is buggy because when a script creates a lot of groups on the fly, sometimes the new empty group is stored (as usual) like a component definition, but has NO instance. This leads to a bug splat. Below is the alias method used to avoid the bug:


class Sketchup;;Entities
        alias add_group_su add_group
        def add_group(*args)
	GC.start
        g=self.add_group_su(*args)
        while g.class!=Sketchup;;Group
		puts "retry add_group" # trace
		g=self.add_group_su(*args)
 Sketchup.active_model.definitions.purge_unused
end
return g
end
end

Chris Fullmer

Thanks, I've been trying to figure out how to make this "alias" with no luck. I'll use your snippet.

So if you include it in your script and I include it mine, and other people put it in theirs, does it start doing weird things to the poor add_group method?

Chris

Chris Fullmer

Hmm, thats not helping speed things up for me. Iterating over 7,500 entities and grouping them in all_connected groups takes 187 seconds the way I have it right now.

adding the alias method above to my script makes it take 209 seconds. And as I've mentioned, I've got it spliiting the entities into groups of 1000 so I can see how the slow down is happening. And the method doesn't seem to help. Each block of 1000 takes twice as long as the previous block.

Here's my entire script if you care to look at how I put it together.

require 'sketchup.rb'

class Sketchup;;Entities
	alias add_group_su add_group
	def add_group(*args)
		GC.start
		g=self.add_group_su(*args)
		while g.class!=Sketchup;;Group
			#puts "retry add_group" # trace
			g=self.add_group_su(*args)
			Sketchup.active_model.definitions.purge_unused
		end
		return g
	end
end

module Clf_loosegroups

	# Make groups of all geometry in the selection set or active_entities.
	def Clf_loosegroups.main
		now = Time.now
		model = Sketchup.active_model
		@entities = model.active_entities
		sel = model.selection
		if sel[0]
			ents = sel.to_a
		else
			ents = @entities.to_a
		end
		
		if Sketchup.version_number < 7000000
			model.start_operation "Make Groups"
		else
			model.start_operation "Make Groups", true
		end
		
		#this divides the given entities into groups of 1000 and then sends them to be turned into groups
		chunk = 1000
		count = (ents.length/chunk)+1
		new_arr = []
		@top_level = ents[0].parent
		count.times do |e|
			place_number = (e*chunk)
			new_arr = ents[place_number,chunk]
			puts new_arr.length
			Clf_loosegroups.maker(new_arr)
		end
		model.commit_operation
		now2 = Time.now
		puts now2-now
	end
	
	#this takes each given entity, finds all_connected, then adds all that to a groups, unless the given entity is already a part of a group
	#That would happen when an entity it was connected to was already sent to this method in a previous block.
	def Clf_loosegroups.maker(ents)
		nowa = Time.now
		while ents.length > 0
			if ((ents[0].is_a? Sketchup;;Edge) || (ents[0].is_a? Sketchup;;Face)) && ents[0].parent == @top_level
				to_group = ents[0].all_connected
				ents = ents - to_group
				group = @entities.add_group(to_group)
			else
				ents = ents - [ents[0]]
			end
		end
		nowb = Time.now
		puts "---------"
		puts nowb-nowa
		puts "---------"
	end # maker
end




#----------Menu system----------
if( $submenu )
	organizer = $submenu 
	if !file_loaded?(__FILE__) then
		organizer.add_item("Loose Geometry to Groups") { Clf_loosegroups.main  }
	end
else
	if !file_loaded?('clf_menu_loader') then
		$clf_tools_menu = UI.menu("Plugins").add_submenu("Chris Fullmer Tools")
	end
	if !file_loaded?(__FILE__) then
		$clf_tools_menu.add_item("Loose Geometry to Groups") { Clf_loosegroups.main  }
	end
end

file_loaded('clf_menu_loader')
file_loaded(__FILE__)

Jim

How do I reproduce the bug?

Jim

Well, I do get a BugSplat every time I try to explode my groups made using your script, Chris.

I am only creating 100 groups total.

Chris Fullmer

Thats odd. I never get bugsplats. I'll see if I can make it splat.

I've attached my file - pre any tweaks I've trined from this thread. I've also included the file I used in the video.

(its large and might take a few moments to load - sorry!
[flash=944,759:2xbb34vu]http://www.chrisfullmer.com/chrisfullmer/forums/group_slowdown.swf[/flash:2xbb34vu]

And again, I'm not sure its a "bug". Seems more like a sketchup issue. It gets slower at making groups when there are already lots in the model. If anyone think s of anything else, great! But don't stay up late thinking about it

Chris

group slowdown.skp

clf_loose2groups.rb