I Recently got this issue when starting the App cluster Member for the first time ““java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 11″”

Issue :

a) The Dmgr started properly but when i when tried to start the cluster from the Dmgr for the first time i got this error in the System Out “java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 11”

Error Stack :

Steps Tried :

a) Tried to Start the cluster Members using ./startServer.sh server1 but still the error in the SystemOut.log
I felt the heap memory was not sufficient for the JVM increased by 512 MB but still same error 🙂

b) On Revisitng the SystemOutLogs and concluded that it seems to be Native Memory issue error from the Java Stack Error

c) Checked the System Resources like CPU , Memory but they were all fine . Finally i checked the Ulimit
All the Parameters were unlimited except the “Max no of Process” (nproc) which was 1024 .

d) The Value for ulimit nproc was very less than the recommended .
In Linux here is a new configurations (/etc/security/limits.d/90-nproc.conf ) which overrides the nproc setting in the /etc/security/limits.conf
Sample output of /etc/security/limits.d/90-nproc.conf
#* soft nproc 1024

WebSphere instance only uses a few hundred of threads and it will go on increasing depending on the Load and the other instances in the server .
We also have to take into account the other threads which are running on the Machine . So the nproc count may reach if too many threads are opened

Solution:

a) Edit the /etc/security/limits.d/90-nproc.conf to increase the nproc limit for all users .You can also set it for specific user too ..
    NOTE : A reboot is not required for this to take effect, however, the user must be completely logged out of the putty or the session and relogin to take the effect. The next time the user logs back in, the new value will be in place. This can be checked by running ‘ulimit -u’ as the user in question.

b) Update the /etc/security/limits.conf file for the nproc values to 131072 ( or any  higher value )

It is recommended to set the ulimit -u or nproc to a value of 131072 when running on Linux to safely account for all the forked threads within processes that could be created

There are many documents avaliable on the internet for the steps to change the ulimit . please go through them too .

c) After I made those changes to the ulimit & 90-nproc.conf

d) Now i restarted my cluster member and it started without any error now 🙂

NOTE : There could be other reasons though for the “java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 11” .. this is ONE of them