Weblogic Autorestart - Managed Servers

Weblogic autorestart of managed servers might be required if there is a problem with it , like stuck thread , overloaded , aborted or suspended state. The below automation can help to restart based on a certain state of the manged server. In the below I am restarting the managed server when state is other than HEALTH_OK. feel free to customize it to your requirement.

oracle@marsweblogic:~$ cat /export/home/oracle/automation/autorestart/wls_autorestart.py

import os

import re

import string

import array

import java.lang

from jarray import array as jarray_c

from java.util import Hashtable

from javax.management import MBeanServerConnection

from javax.management import ObjectName

from java.lang import String

from java.lang import Object

connect('weblogic','xxxxxxxx','t3://192.168.0.1:7001')

domainRuntime();

cd('ServerRuntimes')

servers = domainRuntimeService.getServerRuntimes();

stoppedServers = []

for server in servers:

 try:

        cd ('/ServerRuntimes/' + server.getName())

        CS = get('OverallHealthState').getState()

        if CS == 0:

                print server.getName(), "= HEALTH_OK"

        elif CS == 1:

                print server.getName(), "= HEALTH_WARN :"

                ms = server.getName()

                shutdown(server.getName(), 'Server', force="true")

                start(ms, 'Server')

        elif CS == 2:

                print server.getName(), "= HEALTH_CRITICAL :"

                ms = server.getName()

                shutdown(server.getName(), 'Server', force="true")

                start(ms, 'Server')

        elif CS == 3:

                print server.getName(), "= HEALTH_FAILED :"

                ms = server.getName()

                shutdown(server.getName(), 'Server', force="true")

                start(ms, 'Server')

        elif CS == 4:

                print server.getName(), "= HEALTH_OVERLOADED :"

                ms = server.getName()

                shutdown(server.getName(), 'Server', force="true")

                start(ms, 'Server')

        else:

                print server.getName() + ': ' + get('State') + ': UNKNOWN HEALTH STATE (' + currentState + ')'

 except WLSTException, e:

                print server.getName() + " is not running."

                stoppedServers.append(server.getName())

disconnect()

exit()

oracle@marsweblogic:~$

Oracle Database TDE Enabling Automation

Oracle Database TDE Enabling Automation on a huge database can be made faster by parallelized tasks. The below automation will help to achieve it on Linux. 

The logic used here is

• build the sql for altering the datafile encrypt 
• find the total number of CPU
• split the workload - datafile/cpu
• submit all the jobs in the background
• monitor the job completion
• validate if all datafiles are encrypted

#Beginning of the Script

echo "TDE - PreStage - `date` - Started"

echo "TDE - PreStage - `date` -- Building SQL for Encrypting Tablespace - Started"

rm tablespace_tde_pre.sql tablespace_tde_exec.sql tablespace_tde_job_*

echo "set pagesize 1000" >  tablespace_tde_pre.sql

echo "set lines 200"     >> tablespace_tde_pre.sql

echo "set define off"    >> tablespace_tde_pre.sql

echo "set term off"      >> tablespace_tde_pre.sql

echo "set head off"      >> tablespace_tde_pre.sql

echo "set feedback off"  >> tablespace_tde_pre.sql

echo "spool tablespace_tde_exec.sql" >> tablespace_tde_pre.sql

echo "select 'alter database datafile '||chr(39)||df.name||chr(39)||' encrypt;' from v\$tablespace ts, v\$datafile df where ts.ts#=df.ts# and (ts.name not in ('SYSTEM','SYSAUX') and ts.name not in (select value from gv\$parameter where name='undo_tablespace'));" >> tablespace_tde_pre.sql

echo "spool off" >> tablespace_tde_pre.sql

echo "exit" >> tablespace_tde_pre.sql

sqlplus -s / as sysdba @tablespace_tde_pre.sql

sed -i '/^$/d' tablespace_tde_exec.sql

echo "TDE - PreStage - `date` -- Building SQL for Encrypting Tablespace - Completed"

echo "TDE - PreStage - `date` -- Building Parallel SQL for Encrypting Tablespace - Started"

cpuCount=`nproc`

echo "TDE - PreStage - `date` -- CPU Count is $cpuCount"

dbfileCount=`cat tablespace_tde_exec.sql|wc -l`

echo "TDE - PreStage - `date` -- DB File Count is $dbfileCount"

if [ $dbfileCount -lt $cpuCount ]

then

parallelJobCount=1

else

parallelJobCount=`expr $dbfileCount / $cpuCount`

fi

echo "TDE - PreStage - `date` -- Number of Datafiles in a Job is $parallelJobCount"

split -l $parallelJobCount tablespace_tde_exec.sql tablespace_tde_job_

echo "TDE - PreStage - `date` -- Created $cpuCount Jobs"

echo "TDE - PreStage - `date` -- Building Parallel SQL for Encrypting Tablespace - Completed"

echo "TDE - Execution - `date` -- Executing Parallel SQL for Encrypting Tablespace - Started"

echo "sqlplus / as sysdba << EOF"  > tablespace_tde_jobs.template

echo "set echo on               " >> tablespace_tde_jobs.template  

for i in `ls tablespace_tde_job_*`

do

cp tablespace_tde_jobs.template tablespace_tde_jobs.t

cat $i     >> tablespace_tde_jobs.t

echo "EOF" >> tablespace_tde_jobs.t

cp tablespace_tde_jobs.t $i

nohup sh $i > $i.log 2>&1 &

echo "TDE - Execution - `date` -- Job $i Submitted - Log $i.log"

done

echo "TDE - Execution - `date` -- Executing Parallel SQL for Encrypting Tablespace - Completed"

echo "TDE - Execution - `date` -- Monitoring Parallel SQL for Encrypting Tablespace - Started"

runningTDEJobs=`ps -ef|grep -i tablespace_tde_job|grep -v grep|wc -l`

while [ $runningTDEJobs -gt 0 ]

do

echo "TDE - Execution - `date` -- Running TDE Jobs"

ps -ef|grep -i tablespace_tde_job|grep -v grep

sleep 30

runningTDEJobs=`ps -ef|grep -i tablespace_tde_job|grep -v grep|wc -l`

done

echo "TDE - Execution - `date` -- Monitoring Parallel SQL for Encrypting Tablespace - Completed"

echo "TDE - PostStage - `date` -- Verification of Tablespace Encryption - Started"

sqlplus / as sysdba << EOF

alter database open;

select name from v\$tablespace where ts# not in (select ts# from v\$encrypted_tablespaces);

EOF

echo "Verify if all the tablespaces with the exception of TEMP/UNDO/SYSTEM/SYSAUX is encrypted"

echo "TDE - PostStage - `date` -- Verification of Tablespace Encryption - Completed" 

#End of the Script

Customize it to your need. 😵

Weblogic Managed Server - Rolling Restart Automation

Rolling Restart on Weblogic is sometimes done on a periodic manner to get things cleared. Well I am not favor of doing rolling restart but sometimes this is the solution. 

This handy script can help 😊

oracle@wls-ser1:~$ cat /export/home/oracle/automation/autorestart/wls_rollingrestart.py

import os

import re

import string

import array

import java.lang

import time as systime

from jarray import array as jarray_c

from java.util import Hashtable

from javax.management import MBeanServerConnection

from javax.management import ObjectName

from java.lang import String

from java.lang import Object

connect('weblogic','weblogic123','t3://localhost:7001')

domainRuntime();

cd('ServerRuntimes')

servers = domainRuntimeService.getServerRuntimes();

stoppedServers = []

for server in servers:

 try:

        cd ('/ServerRuntimes/' + server.getName())

        CS = get('OverallHealthState').getState()

        if CS == 0:

                print server.getName(), "= HEALTH_OK"

        elif CS == 1:

                print server.getName(), "= HEALTH_WARN :"

        elif CS == 2:

                print server.getName(), "= HEALTH_CRITICAL :"

        elif CS == 3:

                print server.getName(), "= HEALTH_FAILED :"

        elif CS == 4:

                print server.getName(), "= HEALTH_OVERLOADED :"

        else:

                print server.getName() + ': ' + get('State') + ': UNKNOWN HEALTH STATE (' + currentState + ')'

        ms = server.getName()

        if ms != "AdminServer":

                print ms, " Shutdown Initiated"

                systime.sleep(10)

                shutdown(server.getName(), 'Server', force="true")

                print ms, " Shutdown Completed"

                print ms, " Startup Initiated"

                start(ms, 'Server')

                systime.sleep(10)

                print server.getName(), " Startup Completed"

        else:

                print "AdminServer - no action performed"

        CS = get('OverallHealthState').getState()

        if CS == 0:

                print server.getName(), "= HEALTH_OK"

        elif CS == 1:

                print server.getName(), "= HEALTH_WARN :"

        elif CS == 2:

                print server.getName(), "= HEALTH_CRITICAL :"

        elif CS == 3:

                print server.getName(), "= HEALTH_FAILED :"

        elif CS == 4:

                print server.getName(), "= HEALTH_OVERLOADED :"

        else:

                print server.getName() + ': ' + get('State') + ': UNKNOWN HEALTH STATE (' + currentState + ')'

 except WLSTException, e:

                print server.getName() + " is not running."

                stoppedServers.append(server.getName())

disconnect()

exit()

oracle@wls-ser1:~$

 

oracle@wls-ser1:~$ cat /export/home/oracle/automation/autorestart/wls_rollingrestart.sh

log='/export/home/oracle/automation/autorestart/wls_rollingrestart.log'

date >> $log

echo >> $log

. /u01/app/oracle/product/fmw/middleware/wlserver/server/bin/setWLSEnv.sh >> $log

java weblogic.WLST -skipWLSModuleScanning /export/home/oracle/automation/autorestart/wls_rollingrestart.py >> $log

echo >> $log

echo >> $log

echo "-----------------------------------------------------------------------------" >> $log

oracle@wls-ser1:~$

oracle@wls-ser1:~$ crontab -l

10 0 * * 5   /export/home/oracle/automation/autorestart/wls_rollingrestart.sh

oracle@wls-ser1:~$

Execution Log

-----------------------------------------------------------------------------

Monday, July 13, 2020 at  5:20:12 PM +03

CLASSPATH=/usr/jdk/instances/jdk1.8.0/lib/tools.jar:/u01/app/oracle/product/fmw/middleware/wlserver/modules/features/wlst.wls.classpath.jar:/usr/jdk/instances/jdk1.8.0/lib/tools.jar:/u01/app/oracle/product/fmw/middleware/wlserver/modules/features/wlst.wls.classpath.jar:

PATH=/u01/app/oracle/product/fmw/middleware/wlserver/server/bin:/u01/app/oracle/product/fmw/middleware/wlserver/../oracle_common/modules/thirdparty/org.apache.ant/1.9.8.0.0/apache-ant-1.9.8/bin:/usr/jdk/instances/jdk1.8.0/jre/bin:/usr/jdk/instances/jdk1.8.0/bin:/usr/bin:/usr/sbin:/u01/app/oracle/product/fmw/middleware/wlserver/../oracle_common/modules/org.apache.maven_3.2.5/bin:/u01/app/oracle/product/fmw/middleware/wlserver/../oracle_common/modules/org.apache.maven_3.2.5/bin

Your environment has been set.

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Connecting to t3://localhost:7001 with userid weblogic ...

Successfully connected to Admin Server "AdminServer" that belongs to domain "ms_domain".

Warning: An insecure protocol was used to connect to the server.

To ensure on-the-wire security, the SSL port or Admin port should be used instead.

Location changed to domainRuntime tree. This is a read-only tree

with DomainMBean as the root MBean.

For more help, use help('domainRuntime')

ms_server02 = HEALTH_OK

ms_server02  Shutdown Initiated

Shutting down the server ms_server02 with force=true while connected to AdminServer ...

.

ms_server02  Shutdown Completed

ms_server02  Startup Initiated

Starting server ms_server02 ......................................................

Server with name ms_server02 started successfully

ms_server02  Startup Completed

ms_server02 = HEALTH_OK

ms_server01 = HEALTH_OK

ms_server01  Shutdown Initiated

Shutting down the server ms_server01 with force=true while connected to AdminServer ...

.

ms_server01  Shutdown Completed

ms_server01  Startup Initiated

Starting server ms_server01 ................................................

Server with name ms_server01 started successfully

ms_server01  Startup Completed

ms_server01 = HEALTH_OK

ms_server04 = HEALTH_OK

ms_server04  Shutdown Initiated

Shutting down the server ms_server04 with force=true while connected to AdminServer ...

.....

ms_server04  Shutdown Completed

ms_server04  Startup Initiated

Starting server ms_server04 ..................................................

Server with name ms_server04 started successfully

ms_server04  Startup Completed

ms_server04 = HEALTH_OK

ms_server03 = HEALTH_OK

ms_server03  Shutdown Initiated

Shutting down the server ms_server03 with force=true while connected to AdminServer ...

.

ms_server03  Shutdown Completed

ms_server03  Startup Initiated

Starting server ms_server03 ............................................

Server with name ms_server03 started successfully

ms_server03  Startup Completed

ms_server03 = HEALTH_OK

AdminServer = HEALTH_OK

AdminServer - no action performed

AdminServer = HEALTH_OK

Disconnected from weblogic server: AdminServer

Exiting WebLogic Scripting Tool.

Someone You Loved Cover | Keyboard Performance | Dharun at Improviser Music Studio

Cannot Stop Posting this from my Son

Someone You Loved Cover | Keyboard Performance | Dharun at Improviser Music Studio

EBS 12.2 FORMS CRASH FRM-92101 After 12.2.9 Upgrade when submitting Active User

If you are an EBS administrator the first thing some asks you to do a health check of an environment , you return to him with a Request ID of Active Users which is Completed. It ensures that you have gone through Login Page , OA pages , Forms , Concurrent Mangers , View Output. 80% of the EBS would be in a healthy state if you get a Request ID.

This case all went fine with the upgrade to 12.2.9 and then something strange happened when we Submit Active User or any other requests. In Oracle Support there are specific notes that are discussing this exception and it will direct you to apply patches. Well I did apply the patches but did not help

The below exception messages are seen in the forms immediately after clicking the Submit Button.

FRM-92101: There was a failure in the Firms Server during startup. This could happen due to invalid configuration. 

oracle.forms.net.ConnectionException: Forms session <3> aborted: unable to communicate with runtime process.

Errors seen in the forms logs are below

####<Jul 2, 2020 1:41:08 PM AST> <Error> <ServletContext-/forms> <bangalore.testserver.com> <forms_server1> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <00060qTRvSjFc5T_u98DyX0001Z400001Q> <1593686468004> <BEA-000000> <Forms session <3> exception stack trace:

java.io.IOException: FRM-93000: Unexpected internal error.

Details : No HTTP headers received from runform

        at oracle.forms.servlet.ListenerServlet.forwardResponseFromRunform(Unknown Source)

        at oracle.forms.servlet.ListenerServlet.doPost(Unknown Source)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)

        at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)

        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)

        at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)

        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)

        at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:138)

        at java.security.AccessController.doPrivileged(Native Method)

        at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:324)

        at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:464)

        at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:121)

        at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:211)

        at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)

        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)

Fix : after implementing many fixes including some known issue patches , this error did not change. Also it might be worth to validate Operating System patches , like motif and other libraries which are required for running EBS.

The fix for this exception in this environment was to compile FlexFields using adadmin. I hope this helps. 😉