crashing jrockit while starting WebLogic server instance

Just today I spent some time working on a JVM (jrockit) while starting WebLogic with the default startWebLogic.sh script. The enviroment is rather straight forward:

• Jrockit (jrockit-jdk1.6.0_22-R28.1.1-4.0.1)
• WebLogic 10.3.4
• SOA Suite PS3 (11.1.1.4)

Now whenever I started WebLogic I ran into a rather lengthy core dump, which you can view: here (just rename to .txt). The interesting part of this core dump is this:


Loaded modules:
(* denotes the module where the exception occured)
0000000000400000-0000000000412703 /u01/app/oracle/product/1.6.0/jrockit-jdk1.6.0_22-R28.1.1-4.0.1/bin/java
0000003353200000-0000003353201f93 /lib64/libdl.so.2
0000003353600000-0000003353615a6f /lib64/libpthread.so.0
0000003352a00000-0000003352b4d067 */lib64/libc.so.6
0000003351a00000-0000003351a1bb7f /lib64/ld-linux-x86-64.so.2
00002ba4453b2000-00002ba4456b734b /u01/app/oracle/product/1.6.0/jrockit-jdk1.6.0_22-R28.1.1-4.0.1/jre/lib/amd64/jrockit/libjvm.so
00002ba445991000-00002ba4459af9eb /u01/app/oracle/product/1.6.0/jrockit-jdk1.6.0_22-R28.1.1-4.0.1/jre/lib/amd64/libosal.so
00002ba445bb2000-00002ba445bbe4fb /u01/app/oracle/product/1.6.0/jrockit-jdk1.6.0_22-R28.1.1-4.0.1/jre/lib/amd64/libutil.so
0000003352e00000-0000003352e8122f /lib64/libm.so.6
0000003354600000-0000003354606f5b /lib64/librt.so.1
00002aaaaabb4000-00002aaaaabe9c17 /u01/app/oracle/product/1.6.0/jrockit-jdk1.6.0_22-R28.1.1-4.0.1/jre/lib/amd64/libjdwp.so
00002aaaaacf3000-00002aaaaacf5307 /u01/app/oracle/product/1.6.0/jrockit-jdk1.6.0_22-R28.1.1-4.0.1/jre/lib/amd64/libnpt.so
00002aaaae3d0000-00002aaaae3dbd03 /u01/app/oracle/product/1.6.0/jrockit-jdk1.6.0_22-R28.1.1-4.0.1/jre/lib/amd64/libjfr.so
00002aaaf1a92000-00002aaaf1a9e337 /u01/app/oracle/product/1.6.0/jrockit-jdk1.6.0_22-R28.1.1-4.0.1/jre/lib/amd64/libverify.so
00002aaaf1ba1000-00002aaaf1bc8c73 /u01/app/oracle/product/1.6.0/jrockit-jdk1.6.0_22-R28.1.1-4.0.1/jre/lib/amd64/libjava.so
0000003355200000-00000033552140c7 /lib64/libnsl.so.1
00002aaaf1d10000-00002aaaf1d165bf /u01/app/oracle/product/1.6.0/jrockit-jdk1.6.0_22-R28.1.1-4.0.1/jre/lib/amd64/native_threads/libhpi.so
00002aaaf22dc000-00002aaaf22e98cb /u01/app/oracle/product/1.6.0/jrockit-jdk1.6.0_22-R28.1.1-4.0.1/jre/lib/amd64/libzip.so
00002aaaf2530000-00002aaaf2532e73 /u01/app/oracle/product/1.6.0/jrockit-jdk1.6.0_22-R28.1.1-4.0.1/jre/lib/amd64/libdt_socket.so
00002aaaf2643000-00002aaaf264c64b /lib64/libnss_files.so.2
00002aaaf2a4e000-00002aaaf2a6098f /u01/app/oracle/product/1.6.0/jrockit-jdk1.6.0_22-R28.1.1-4.0.1/jre/lib/amd64/libnet.so
00002aaaf3b26000-00002aaaf3b2bf43 /u01/app/oracle/product/1.6.0/jrockit-jdk1.6.0_22-R28.1.1-4.0.1/jre/lib/amd64/liborii.so
00002aaaf3ded000-00002aaaf3df38ab /u01/app/oracle/product/1.6.0/jrockit-jdk1.6.0_22-R28.1.1-4.0.1/jre/lib/amd64/libnio.so
00002aaaf8e80000-00002aaaf8e818dd /u01/app/oracle/product/11.1.1.4/wlserver_10.3/server/native/linux/x86_64/libweblogicunix1.so
Now this gave me a hook to start querying google and metalink, ending up with only one advice: downgrade jrockit. Obviously I tried this, but this and it gave me the root cause:

<Apr 12, 2011 2:47:54 PM CEST> <Critical> <WebLogicServer> <BEA-000252> <Cannot switch to group "dba". java.lang.IllegalArgumentException: Cannot setegid("dba"), system error: ''
java.lang.IllegalArgumentException: Cannot setegid("dba"), system error: ''
at weblogic.platform.Unix.setEGroup0(Native Method)
at weblogic.platform.Unix.setEffectiveGroup(Unix.java:73)
at weblogic.t3.srvr.SetUIDRendezvous.setEGroup(SetUIDRendezvous.java:156)
at weblogic.t3.srvr.SetUIDRendezvous.makeUnPrivileged(SetUIDRendezvous.java:174)
at weblogic.t3.srvr.SetUIDRendezvous.initialize(SetUIDRendezvous.java:88)
at weblogic.t3.srvr.BootService.start(BootService.java:48)
at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:461)
at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:166)
at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:879)
at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:566)
at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:467)
at weblogic.Server.main(Server.java:71)
>
Now this did help me. Suddenly I realized that I had been playing with machine configuration (uid and gid setting to be more specific)

<machine xsi:type="unix-machineType">
<name>testbox</name>
<node-manager>
<name>testbox</name>
<listen-address>localhost</listen-address>
</node-manager>
<post-bind-uid-enabled>true</post-bind-uid-enabled>
<post-bind-uid>testuser</post-bind-uid>
<post-bind-gid-enabled>true</post-bind-gid-enabled>
<post-bind-gid>dba</post-bind-gid>
</machine>
Now testuser doesn’t have dba as primary group:

$ id testuser
uid=10641(testuser) gid=100(users) groups=4006(dba),100(users)
So because of this jrockit crashes… (wrong group for the machine). The error is kind of unfriendly, downgrading seems to be a way of improving error messages.

Hope this helps.