On closer inspection it appears the bug is not related to NSIS but instead to any guest OS process that tries to output a large file to a Virtualbox shared folder.
See this thread http://www.virtualbox.org/ticket/5830 for more information.
I've not tested this in Virtualbox 4.x yet as I'm waiting for at least 4.0.6 before attempting an upgrade.
The workaround for the NSIS problem is to copy the files required to build the installer from the shared folder to a local drive on the guest OS (e.g. C:)
Edit: Bug still present in NSIS 2.46 on VirtualBox host 4.1.8