I'm not an expert on this, but what machine are you running on? How many CPUs or cores and how much RAM would be the first relevant questions.
I'm on a quad core machine with 6GB of RAM and I notice a significant slow down in performance when turn on shadows for example. So, it may just be that your model is taking a lot of processing power.