Example stack trace:
Caused by: java.io.IOException: Too many open files
at sun.nio.ch.IOUtil.initPipe(Native Method)
at sun.nio.ch.EPollSelectorImpl.<init>(Unknown Source)
at sun.nio.ch.EPollSelectorProvider.openSelector(Unknown Source)
Find your Java process ID:
pgrep java
15625
Use lsof with the -p option to list files, sockets, pipes associated with your Java process (pipe to more as you are expecting a lot of them):
lsof -p 15625 | more

COMMAND   PID       USER   FD   TYPE     DEVICE SIZE/OFF    NODE NAME
java    15625 chrisuser  cwd    DIR        8,1     4096 2351434 /home/chrisuser/insto
java    15625 chrisuser  rtd    DIR        8,1     4096       2 /
java    15625 chrisuser  txt    REG        8,1    47308 2359299 /home/chrisuser/jre1.6.0_29/bin/java
java    15625 chrisuser  mem    REG        8,1    71432 2982539 /lib/i686/cmov/libresolv-2.11.3.so
java    15625 chrisuser  mem    REG        8,1    22036 2982554 /lib/i686/cmov/libnss_dns-2.11.3.so
java    15625 chrisuser  mem    REG        8,1    96856 2359345 /home/chrisuser/jre1.6.0_29/lib/i386/libnet.so
java    15625 chrisuser  mem    REG        8,1    37406 2359346 /home/chrisuser/jre1.6.0_29/lib/i386/libnio.so
java    15625 chrisuser  mem    REG        8,1     8887 2351465 /home/chrisuser/insto/lib/slf4j-jdk14-1.6.3.jar
java    15625 chrisuser  mem    REG        8,1  1084061 2351470 /home/chrisuser/insto/lib/quickfixj-core-1.5.2.jar
java    15625 chrisuser  mem    REG        8,1  1527568 3195199 /usr/lib/locale/locale-archive
java    15625 chrisuser  mem    REG        8,1 46859519 2359429 /home/chrisuser/jre1.6.0_29/lib/rt.jar
java    15625 chrisuser  mem    REG        8,1   277723 2260993 /home/chrisuser/insto/lib/quickfixj-msg-fix42-1.5.2.jar
...
This information should tell you the type and location of file/socket/pipe is causing the resource exhaustion.
If you need to debug this error on a different OS then this page might help:
http://www-01.ibm.com/support/docview.wss?rs=0&uid=swg21067352