Faster backup saves

jgb

I've harped on this before, but now I've thought it through and I have a way to do it.

The problem: Big files take a long time to do timed auto-saves (I'm not referring to manual saves).
On my PC my biggest file (35mb) takes about 2 full minutes to auto save every 1/2 hour.
75% of my files are over 10mb, of those 1/2 are over 20mb.
I fully realize the save time is largely controlled by the individual h/w in the O/S file support system, but SU is frozen during the save. Either of my 2 suggestions will either eliminate the freeze or bring it down to a very few seconds with even the largest files.

There are 2 solutions for SU-9.

Make use of PC's with dual/quad core to do the auto save (and the manual save) on the 2nd core in true background.
This should be relatively easy to implement, requiring a sub-program running independently in background in core 2 monitoring the memory area of SU running in core 1. Auto saves would occur as now, on a user defined timed basis. This interval can now be just a few minutes between.
This should eliminate any SU freeze time during a save.
Save ONLY the "UNDO" stack periodically till a manual save resets it. Here's how.

Keep in mind, the undo stack is actually a chronological list of all model changing operations.
The undo command simply reverses the changes back down to a previous state. So if you take a copy of the file from the initial load (or last save) and then apply the undo stack to it, or a "redo", you should end up with the same version as if no undo's were done.

The auto save remains event conditional on 3 points.
a) the user defined auto save interval is reached. (in my case, 30 minutes)
b) the undo stack runs to 100 (or whatever) operations. (SU6 is 100 steps, I've counted)
c) the undo stack exceeds a defined size. (a few mb)

The undo stack is still preserved in SU, but is modified with a pointer to keep tabs on where in the stack it last auto-saved to avoid double counting.

When SU first loads a model file, it creates an empty auto-save file on HD, in background.
When an auto-save is triggered (a,b or c) the undo stack is written out to disk, appended to the previous (this session only) auto-save file. This is very fast as the undo stack is relatively very small.

When a manual save is triggered, the save file is written out to a new file, and if the save is successful, the old file is deleted and the auto-save stack is zeroed out.

To recover a file (upto the last auto-save point) is simple.
Su loads the last saved file, then applies the saved undo stack (chronologically) to it, in effect a total "redo".

As is the case today, the recreated file is only as up to date as the last save or auto-save.
This way you can reduce the auto-save interval dramatically to a few minutes, preserving a greater amount of work in case of some crash or other event.

An alternate (optional) use of saving the undo stack is to preserve the detailed work history of the model over several versions, for audit use, training or documenting bugs. In this case, the undo stack file is preserved along with its base saved file, to be played back at some later date.

tfdesign

It's an interesting idea, but I hate to be a Mac-bore, but I can't see this ever being implemented in SU9 because Apple Mac's already have a back-up system built in to the OS called "Time Machine", which runs silently as well as effortlessly in the background, onto an external HD, while you sweat over what really matters!

Right, blatant plug for Apple over. Windows. Time Machine apart, there still doesn't look like there is a decent back up system built into Windows- even W7. That's a shame. But I did find this, which looks quite interesting.

http://gizmodo.com/5229397/seagate-replica-is-time-machine-for-windows-pcs

thomthom

@tfdesign said:

It's an interesting idea, but I hate to be a Mac-bore, but I can't see this ever being implemented in SU9 because Apple Mac's already have a back-up system built in to the OS called "Time Machine", which runs silently as well as effortlessly in the background, onto an external HD, while you sweat over what really matters!

It's not the same - he's talking about the Auto-save feature. Not the bakup files SU creates when you save a file.

thomthom

@jgb said:

The problem: Big files take a long time to do timed auto-saves (I'm not referring to manual saves).
On my PC my biggest file (35mb) takes about 2 full minutes to auto save every 1/2 hour.
75% of my files are over 10mb, of those 1/2 are over 20mb.

10-35MB isn't really that big. I got files that goes up to 100-200MB.
2min to save a 10-35MB file sounds odd - never experienced that. Are you by any chance saving to a network location?

jgb

@thomthom said:

@jgb said:

The problem: Big files take a long time to do timed auto-saves (I'm not referring to manual saves).
On my PC my biggest file (35mb) takes about 2 full minutes to auto save every 1/2 hour.
75% of my files are over 10mb, of those 1/2 are over 20mb.

10-35MB isn't really that big. I got files that goes up to 100-200MB.
2min to save a 10-35MB file sounds odd - never experienced that. Are you by any chance saving to a network location?

Well maybe, just maybe, I exaggerate a bit. I actually just timed it and it takes about 45 seconds on average. But when it happens while I am in a line draw or a pan, it is infuriating to be interrupted and put on hold for what FEELS like 2 or 3 minutes!!

My PC is about 6 1/2 years old with a 120gb IDE hard drive (big then! ), which probably accounts for its relative slowness compared to newer SATA based drives. But it has a fast Gigabyte board and Intel CPU with 1 gb RAM and a high end (then) ATI GPU with 512mb VRAM, so it performs pretty good with big SU files.

But still, saving a smaller file is faster than saving a bigger file in any system. Doing it as I described above results in a delay hit ONLY if recovery is required. Even today, if you need to recover from the backup file (.SKB), there is a delay in bringing it up compared to loading a .SKP file.

thomthom

@jgb said:

My PC is about 6 1/2 years old with a 120gb IDE hard drive (big then! ), which probably accounts for its relative slowness compared to newer SATA based drives.

This is true - your HD could very well be a bottleneck. then it would not matter how fast teh rest of your computer is.
Have you defragged you HD at all?

@jgb said:

Even today, if you need to recover from the backup file (.SKB), there is a delay in bringing it up compared to loading a .SKP file.

.skb? the autosave feature saves to an Autosave_<filename>.skp file - the .skb files are made when you do a manual save.
which feature are you talking about?

I share your frustration with the autosave though - I find it all too annoying - but it does have its usefulness.

jgb

@thomthom said:

.skb? the autosave feature saves to an Autosave_<filename>.skp file - the .skb files are made when you do a manual save.
which feature are you talking about?

I always thought the SKB was the autosave version, but checking into the file structure I see the "Autosave" version as a separate file, as you noted above. My error.

But THAT (Autosave_<filename>.skp) is the file I refer to, not the manual backup (SKB) which is the previous version manually saved.

As for defragging, not in a while. Just lazy. However, the partition that SU saves to is large and is only about 20% used, so defragging would not speed things up by much in this case. But yeah, I gotta do it, just it takes well over an hour to do the whole drive (not exaggerating here).

thomthom

You might very well find better performance with a newer SATA II drive. (In either case - 6,5 year old HD... hope you have backups...)

jgb

@thomthom said:

You might very well find better performance with a newer SATA II drive. (In either case - 6,5 year old HD... hope you have backups...)

I need to do that too!