NodeManager not Reachable: java.io.IOException: Invalid State File Format

Error :

NodeManager not Reachable: java.io.IOException: Invalid State File Format

If we get the above error, Managed Servers cannot be started as the nodemanager is not reachable to Admin server Console.

Stack trace from nodemanager.log file :

   

java.io.IOException: Invalid state file format. State file contents: 

at weblogic.nodemanager.common.StateInfo.load(StateInfo.java:135) 

at weblogic.nodemanager.server.ServerMonitor.loadStateInfo(ServerMonitor.java:475) 

at weblogic.nodemanager.server.ServerMonitor.isCleanupAfterCrashNeeded(ServerMonitor.java:139) 

at weblogic.nodemanager.server.ServerManager.recoverServer(ServerManager.java:255) 

at weblogic.nodemanager.server.DomainManager.initialize(DomainManager.java:103) 

at weblogic.nodemanager.server.DomainManager.(DomainManager.java:55) 

at weblogic.nodemanager.server.NMServer.getDomainManager(NMServer.java:257) 

at weblogic.nodemanager.server.Handler.handleDomain(Handler.java:224) 

at weblogic.nodemanager.server.Handler.handleCommand(Handler.java:108) 

at weblogic.nodemanager.server.Handler.run(Handler.java:70) at java.lang.Thread.run(Thread.java:619)

Reason :

The state file of the managed server is in an invalid state or Failed_not_Restartable.

Login to the server & navigate the Domain_Home, then move  inside to the servers directory

Under each managed server directory, there is a NodeManager directory containing a state file <managed_server_name>.state. If this file is empty or corrupt, then we may have a option to chance to get the errors o.

For example, 

Domain_Home/servers/Managed-server1/data/nodemanager/Managed-server1.state

We have couple of reasons to corrupt this Managed-server1.state file. 

     1) Unexpected physical server reboot

     2) Killing the running node manager process

How we can resolve this issue, if any target managed server is running, try to stop the instance using command line

Domain_Home/bin/stopManagedWeblogic.sh <Managed server Name>

The stop the Admin server

Domain_Home/bin/stopWebogic.sh 

Stop NodeManager process if it is running 

Finally verify the all process related to the domain & delete the below three files.

Domain_Home/servers/Managed-server1/data/nodemanager/Managed-server1.state

Domain_Home/servers/Managed-server1/data/nodemanager/Managed-server1.lck

Domain_Home/servers/Managed-server1/data/nodemanager/Managed-server1.pid

1.     Start the Admin Server .

2.     Start Node Manager.

3.     Verify the nodemanager staus in weblogic admin console(should be reachable)

4.     Start the managed server using the Admin Console.

weblogic.nodemanager.server.NMServer main SEVERE: Fatal error in node manager server java.net.BindException: Address already in use

Error :

Feb 5, 2014 2:45:02 AM weblogic.nodemanager.server.NMServer main SEVERE: Fatal error in node manager server java.net.BindException: Address already in use at 

Reason :

The error message : java.net.BindException: Address already in use at

Shows that the Port number used by the Node Manager process is being already  used by some other process .

Solution :

1 : First thing we need to check is whether Node Manager is already running on the machine :

For unix :

ps -ef | grep nodemanager

For windows :

netstat -a0 | findstr <Node_manager port>

tasklist | findstr nodemanager

2 : If you have not observe the running process then , check which process is using the using the port assigned to node manager using lsof command or other .

By default Weblogic Node Manager runs on port : 5556 if not just check nodemanager.properties

So, to check which process is using this port use the below command .

For unix :

netstat -an | grep 5556

For Windows :

netstat -ao | findstr 5556

3 : If this port is already used by some other process try to change the Node Manager Port number from file

Middleware_Home/wlserver_10.3/common/nodemanager/nodemanager.properties file :

ListenPort = XXXX

other wise before changing , verify the process & observe the wl server home... if it is same(which you are trying)... try to kill the process & start once again with below commands

4 : Now try to start the Node manager from :

Middleware_Home/wlserver_10.3/server/bin/startNodemanager.sh or

Middleware_Home/wlserver_10.3/server/bin/startNodemanager.cmd

Error NodeManager BEA-300033 Could not execute command “getVersion” on the node manager. Reason: “Access to domain ‘Base_Domain’ for user ‘DTXezFTI’ denied”.

<Error> <NodeManager> <BEA-300033> <Could not execute command “getVersion” on the node manager. Reason: “Access to domain ‘Test_Domain’ for user ‘DTXezFTI’ denied”.>

Resolution Steps:

By default node manager log level is Info .
First we have to set the log level to Finest.(if you configured the logging as finest... you will get more information in logs) 

We have few other log levels available for Node Manager :

SEVERE          
WARNING
INFO                
CONFIG
FINE
FINER
FINEST             

After  changes restart the Node Manager and check for the log file error message again.

Solution :

1 : Log on to the Admin console ( http://AdminserverIP:Port/console )

2 : click on  Domain Name (Test_Domain) in my case >>>>> Then to security >>>>

3 : Click on the Advance Option for change the username & password values to use nodemanager.

Let's an example

username is : weblogic 

password is : welcome123

4 : save the changes.

Then we have to edit the nodemanager properties of each remote host(with in domain):

·         Navigate to the folder %DOMAINHOME%\config\nodemanager

·         Edit the file: nm_password.properties with updated credentials

·         username=weblogic

·         password=welcome123

·         Save

·         Restart the Weblogic Node Manager

5 : Then navigate to the Test_Domain/servers/mannaged_server/data/nodemanager/ dir.

6 : open the boot.properties file and enter the below values :

username=weblogi & password= welcome123

7: Restart the your Admin Server.

8 : Now run the nmEnroll() wlst command for all the machines that has the Managed server.

wlst>connect(‘username’,'password’,'t3://admin_host:admin_port’)

online>>nmEnroll(‘Domain_dir_path’,'NodeManager_Home_Path’)

online>>exit()

Now you can check the node manager status from Admin Console .

left panel >> Machines >> Machine1 >> Monitoring >>

The Status should be reachable.

Configure node manager over SSL ( Custom Certifcates )

Below are the steps to configure Node Manager over SSL :

First create custom certificates using the commands below:

1) keytool -genkey -alias mykey -keyalg RSA -keysize 1024 -dname “CN=Puneeth, OU=Oracle, O=BEA, L=Denver, ST=Colorado, C=US” -keypass password -keystore identity.jks -storepass password

2) keytool -selfcert -v -alias mykey -keypass password -keystore identity.jks -storepass password -storetype jks

3) keytool -export -v -alias mykey -file rootCA.der -keystore identity.jks -storepass password

4) keytool -import -v -trustcacerts -alias mykey -file rootCA.der -keystore trust.jks –storepass password

.

Now configure “Custom Identity and Customer Trust “for Admin and managed servers as shown below:

.

.

Now enter the Key Alias and Private Key Passphrase under SSL tab for both the servers:

.

.

Make the following changes in nodemanager.properties file :

• AuthenticationEnabled=true
• ListenAddress=   ( Leave it blank if you are using localhost for server Listen Address )
• ListenPort=5556
• SecureListener=true
• StartScriptEnabled=true
• KeyStores=CustomIdentityAndCustomTrust
• CustomIdentityKeyStoreFileName=C\:\\bea10.3\\user_projects\\domains\\custom_certificate_nodemanager\\identity.jks
• CustomIdentityAlias=mykey
• CustomIdentityPrivateKeyPassPhrase=password
• CustomTrustKeyStoreFileName=C\:\\bea10.3\\user_projects\\domains\\custom_certificate_nodemanager\\trust.jks
• .
• .

Add the following flags in setDomainEnv.cmd:

-Dweblogic.security.IdentityKeyStore=CustomIdentity

-Dweblogic.security.CustomIdentityKeyStoreFileName=filename

-Dweblogic.security.CustomIdentityKeyStorePassPhrase=passphrase

-Dweblogic.security.Identity.KeyStoreType=type

-Dweblogic.security.TrustKeyStore=CustomTrust

-Dweblogic.security.CustomTrustKeyStoreFileName=filename

-Dweblogic.security.CustomTrustKeyStoreType=type

-Dweblogic.security.CustomTrustKeyStorePassPhrase=passphrase

.

.

Add the following flag in startNodeManager.cmd :

set JAVA_OPTIONS=-Dweblogic.nodemanager.sslHostNameVerificationEnabled=false%JAVA_OPTIONS%

.

.

Now check the status of NodeManager. It should be reachable:

.

.

Now start the server..!!

Node manager is now configured over SSL