java.net.SocketException: Too many open files

We've recently moved our servers from windows to linux, and started getting the above error.

Its a well known scenario. Linux keeps a max number of open files limit, and we were exceeding it. Its possible to increase the limit, but first we should see if there is any process that may be causing the problem.

This is taken from http://doc.nuxeo.com/display/KB/java.net.SocketException+Too+many+open+files

Count File Descriptors in Use

Count Open File Handles

sudo lsof [-u user] | wc -l

Count File Descriptors in Kernel Memory

sudo sysctl fs.file-nr # => The number of allocated file handles # => The number of unused-but-allocated file handles # => The system-wide maximum number of file handles

There is a global limit and a per user limit

Raising the Global Limit

1. Edit /etc/sysctl.conf and add the following line:
   

   fs.file-max = 65536

2. Apply the changes with:
   

   sudo sysctl -p /etc/sysctl.conf

Raising the per-User Limit

On some systems it is possible to use the ulimit -Hn 8192 and ulimit -Sn 4096 commands. However most of the time this is forbidden and you will get an error such as:

ulimit: open files: cannot modify limit: Operation not permitted

In those cases, you must:

1. Edit as root the following system configuration file:
   

   % sudo vi /etc/security/limits.conf

2. Modify the values for user
   

   user           soft    nofile          4096 user           hard    nofile          8192

If you want to raise the limits for all users you can do instead:

*           soft    nofile          4096 *           hard    nofile          8192

To check whether changes are taken into account, open a new session w. And check that the change has been taken into account:

% su user % ulimit -n 1024

Here we can see that the new value has not been taken into account.
To fix this:

1. Edit /etc/pam.d/su:
   

   % sudo vi /etc/pam.d/su

2. Uncomment the line:
   

   session    required   pam_limits.so

   The change should now be taken into account the next time you login with your user:
   

   % su user % ulimit -n 4096

Log file rotate

Just to remind myself , that the linux logrotate daemon is very handy for those occasions when applications simply output to a single file that keeps growing over time.

THis will rotate daily (and add a date ext to the old file), after 31 days it will start to delete old files. It will not compres them (remove this line if you want it to compress the old log files). Not the size attribute is no longer been specified as it overrides the daily directive, and only rotates if files grow greater than10M. (Apparently there is a maxSize directive in newer versions of logrotate, that can be combined).. See http://serverfault.com/questions/391538/logrotate-daily-and-size

This job gets run nightly. If you want to run it immediately (e.g. to rotate a large file)  then,

>sudo /usr/sbin/logrotate /etc/logrotate.conf

e.g. > sudo vi /etc/logrotate.d/tomcat

/dirto/catalina.out /dirto/tomcat.log /dirto/admin.log /dirto/stacktrace.log{
  copytruncate
  daily
  rotate 31
  nocompress
  dateext
  missingok
  size 10M
}