1) Does the MAC version of SU also use the Internet Explorer Shell?
as Gaieus said on Mac there is Safari shell
2) What if you don't have Internet Explorer installed on your machine?
a form of IE is inside every Windows, mshtml.dll, that is used by other components and IE as well.
3) Is it possible that the javascript inside a Web Dialog can be blocked by a security feature of Windows Vista or Internet Explorer?
usually the javascript is allowed on files on the computer but can be disabled via registry
also check for antivirus/firewalls that can disable/change scripting