By default excel 2010 opens all documents in the same process and in the same window.
If you want to work in 2 separate processes you need to do the following:
Check the "Ignore other applications that use Dynamic Data Exchange (DDE)" box in Excel -> Excel Options -> Advanced
Open regedit, browse to
HKEY_CLASSES_ROOT\Excel.Sheet.8\shell\Open Rename the ddeexec key to whatever
Then click on the "command" key and replace the /e on the Default and command string in that key with "%1" Quotes around %1 are important.
Do the same for Excel.Sheet.12 Now Both .xls and .xlsx should open in new windows with no errors.
(source http://social.technet.microsoft.com/Forums/en-US/c42f825d-b7e8-48c8-b5af-30563b839f09/windows-7-cannot-run-multiple-instances-of-excel-using-ignore-other-applications-that-use-dde?forum=officesetupdeploylegacy)