Correct Way to Make Python3 Default on Linux

 Correct Way to Make Python3 Default on Linux

# update-alternatives --install /usr/bin/python python /usr/bin/python3 1

# update-alternatives --list|grep -i python

python  auto    /usr/bin/python3

# python

Python 3.6.8 (default, Oct  1 2020, 20:32:44)

EBS Upgrade Part 10 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

EBS Upgrade Part 10 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

Now the current state of the EBS and DB is 12.1.3 / 19c multi-tenant running on OEL6.

At this stage you can move upgrade the OS from OEL6 to OEL7 and then proceed further. If upgrade is not possible you can perform a outplace migration from OEL6 to OEL7 and then proceed further.

Upgrade from 12.1.3 to 12.2

• Software / Patch Download and Software Staging
• Source Environment Review
• Backup
• Gather Database Stats -- exec dbms_stats.gather_database_stats(estimate_percent => 15);
• Implement fix : Fresh Install of EBS 12.2 File System Failed With RW-50004 On /tmp Directory (Doc ID 2311816.1) -- correct the file system
• Gather System Stats  -- exec dbms_stats.gather_system_stats
• Rapid Insall - Create Upgrade File System  
• ETCC to the base 12.2 ( weblogic bsu slowness fixed by a patch)
• Gather SYS, Fixed Object and Dictionary Statistics (required) -- Page 4-39 -- run this task and perform the validation
• Gather Schema Stats - skip
• Drop event alert triggers in custom schemas - no custom top prefixed with XX found - skipping this step
• OATM verification - verified
• Application DBA (AD) Tasks Matching - $AD_TOP/sql/adutconf.sql - skipping because version is 12.1.3
• Oracle E-Business Suite Integrated SOA Gateway - skipping because integration is not in use
• GUEST user_id - matching to 6 and end_date is null
• Customer Relationship Management Tasks - does not apply to 12.1.3
• Financials and Procurement Tasks - Advanced Collections - IEX module is Inactive in FND_APPLICATIONS
• Financials and Procurement Tasks - Subledger Accounting - XLA.B.3 is Shared - apply Patch 13420532:R12.XLA.B for R12.1.x - Patched
• Human Resource Management (HRMS) - Object Name : INSERT_POSITION_TXN_INFO -- modified the package from LONG to CLOB
• Supply Chain Management Tasks - Install Base is Inactive --skipping
• Supply Chain Management Tasks - Order Management is Shared -- affected column is with NON-NULL status --- verified
• Supply Chain Management Tasks - Product Hub is Inactive --skipping
• Supply Chain Management Tasks - Packing Hierarchy Structure Type --skipping
• Supply Chain Management Tasks - Backing Up User Defined Attributes Data for Items --skipping
• Supply Chain Management Tasks - Checking for Duplicate Records --validated , as per note and patch instructions there are no duplication rows
• select 'exec dbms_ijob.broken('||job||',true);' from dba_jobs where broken='N';  , change db to spfile , restart , unlock all the locked stats table
• Database Initialization Parameters for Oracle E-Business Suite Release 12 (Doc ID 396009.1) - validation
• create table bkup_dba_objects_pre_upg_122 as select * from dba_objects;
• Disable AOL Audit Trail (conditional)
• Shutdown Apps
• Download PSU and Apply Latest PSU to Both the RAC Database [ p29141015_121020_Linux-x86-64.zip] and run datapatch post install
• Install ETCC and Apply all Patches p29408621_12102190416_Linux-x86-64.zip
• Run DB ETCC to verify recommendations
• JDK 1.7 Upgrade
• Init.ora verified
• Set FAILED_LOGIN_ATTEMPTS to UNLIMITED
• Maintenance Mode Enabled
• Trigger Disabled
• recycle bin verification
• Apply AD 12.2 upgrade driver , merge with AD Consolidated Upgrade Patch , for adgrants.sql use the one from the merge driver folder + adadmin post install 
• prepare the CPU + PreInstall driver and apply in preinstall mode
• Review  tablespace  and TEMP
• Apply known issues for 12.2 
• Apply Consolidated Upgrade Patch 12.2.0 after merging with preinstall drivers 
• create table bkup_dba_objects_post_upg_122 as select * from dba_objects; --- 263 invalids
• Disable Maintenance Mode
• DB appsutil and autoconfig
• JRE on DB home
• Configuring the Oracle E-Business Suite Release 12.2 Application Tier
• Enable Online Patching , ETCC ( db only , mt done during staging )
• Gather system stats
• Apply GSCC patch and report review and fix violation action
• create table bkup_dba_objects_post_adop_122 as select * from dba_objects;
• Review and Extend APPS_TS_TX_DATA & APPS_TS_TX_IDX
• Review and run the fix by note 2066607.1 for AD ZD invalid objects
• Apply Online Patching adpatch
• alter system set cluster_database=true scope=spfile; / srvctl stop database -d prod ; srvctl start database -p prod
• Page 4-64 Database Initialization Parameters
• Compile Invalids
• page 4-62 GSCC reports
• DB Parameters settings
• AD and TXK for Release 12.2
• RUP 12.2.10
• Product Interoperatability Patch - oracle workflow

Post Upgrade Task

• Start services , Health Check
• Concurrent Manager Post Upgrade Programs
• check for running job and make sure all finish
• Obsolete Schema drop
• Drop dangling synonyms
• shutdown application
• Online help patch 12.2.0
• Online help patch 12.2.10

NLS task

• apply base NLS patch 12.2.0
• apply NLS and TXK AR patch
• apply NLS patch 12.2.10
• apply base NLS help patch 12.2.0 
• apply base NLS help patch 12.2.10
• wf patch

Post Upgrade Release Task

• Application Customization Migration 
• Additional Licensing
• Patch 28984562  for workflow
• Patch as per 2357700.1  for workflow
• Connection Leak Patches
• Instance Stabilization - oacore sizing 4 per node with 2gb heap , 2 forms servlet engine per node with 2gb heap
• run fs_clone force=yes
• Create Mainfest File for NLS
• hrglobal
• dummy cutover
• Mailer Configuration
• UTL Folder Update
• Node Addition
• LB Configuration

With this we are done with the upgrade to EBS 12.2 on Database 19c

Refer to the Index Page to see the full flow of the upgrade. 

EBS Upgrade Part 9 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

EBS Upgrade Part 9 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

Now the current state of the EBS and DB is 12.1.3 / 11.2.0.4 running on OEL6. Well to get to EBS 12.2 Database 19c there are two ways. 

• The best way is to keep 11.2.0.4 database and upgrade EBS 12.1.3 to EBS 12.2. This is stable and kind of been done for some years now.
• The recent supported way is to get the database upgrade to 19c multi-tenant architecture on EBS 12.1.3 and then run upgrade EBS to 12.2

Well the second is being certified but not many customers have gone through this in 2020. So I would suggest to choose option 1. But what if you are in EBS 12.1.3 and you have upgraded your database to 19c. You have to go through option 2.

So I have decided to go by Option 2 to see what is the complication.

Here are the steps involved to upgrade to 19c

• Pre - Database Upgrade From 11.2.0.4 to 19c (19.3)
• Database Installaion 19c (19.3)
• Database Upgrade19c (19.3)
• Post Database Upgrade19c (19.3)
• Convert Database to Multitenant Architecture

Refer to the Index Page to see the full flow of the upgrade. 

EBS Upgrade Part 8 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

EBS Upgrade Part 8 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

Now we are in 12.1.1 , we can directly move to 12.1.3 or 12.2. This step can be differed if you want to move to 12.2

Upgrade Application 12.1.1 to 12.1.3

• Complete Action as per Document 761570.1
• Database Specific Patches Applied as per Document 761570.1
• Upgrading to the OracleAS 10g 10.1.3.4  Doc ID 728531.1
• Apply R12.AD.B.DELTA.3 Patch 9239089
• mv /dev/random /dev/random_original;ln -s /dev/urandom /dev/random
• Relink Error Fix 2057857.1
• Use AutoPatch to apply  12.1.3 Patch 9239090 + post Patch
• Online Help Patch 9239095
• Apply Patch 9817770 and Patch 9966055
• Run DB and MT Autoconfig with appsutil configuration
• Run adpreclone on DB and MT
• Disable Maintenance Mode
• Start Services and Health Check
• Gather Schema Stats - Run Now and Schedule Every Week
• Configure Database RMAN and MT Backup

Post Upgrade 12.1.3 Open Issues

• Missing System Administrator Responsibility for SYSADMIN , recompile JSP
• CM Not Coming up , rebuild CM view using cpadmin
• Slow Standard Manager , run GSS
• Tablespace Space Addition
• Apply Patch 9707056 as per 1559441.1
• Apply patch as per Note 1581955.1 and 1489097.1
• Set SGA to 8 GB , enable AMM
• WF_DEFERRED rebuild as per 286394.1
• WF_CONTROL rebuild as per 1663093.1
• Start Workflow Agents
• Run Bulk Responsibilities Sync Custom Program
• Run Sync Workflow Concurrent Program

With this EBS 12.1.3 migration is completed. We can open the system for users. I am taking this forward to EBS 12.2

Refer to the Index Page to see the full flow of the upgrade. 

EBS Upgrade Part 7 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

 EBS Upgrade Part 7 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

Now we have moved database form 9i RHEL 4 to 11gR2 OEL 6. We should be good to start with the 12.1 Upgrade. The below procedure will help Upgrading Application from 11.5.9 to 12.1.1

• Stage CD Creation
• Source adjkey –initialize
• Restart DB with 1000 as process parameter
• Apply 11i.AD.I.6 , Superseeded by 11i.AD.I.7
• Run TUMS utility , install patch 7705743 
• Convert to Multiple Organizations architecture
• Drop event alert triggers in custom schemas
• Review sizes of old and new tablespaces
• Run AD preparation scripts
• Convert Oracle Alert E-mail Processing to the WF Mailer 
• Customer Relationship Management Tasks
• Incentive Compensation
• TeleService
• Mobile Field Service
• Financials and Procurement Tasks
• General Ledger
• iPayments
• Internet Expenses
• Subledger Accounting
• Leasing and Finance Management Tasks
• Projects Tasks
• Public Sector/University Tasks
• Supply Chain Management Tasks
• Depot Repair
• Enterprise Asset Management
• Process Manufacturing
• Service Contracts
• Drop MRC Schema
• Enable Maintenance Mode
• Gather schema statistics
• Build Repo and  enable addons channel in .repo file
• yum install oracle-ebs-server-R12-preinstall -y
• mv /usr/bin/unzip /usr/bin/unzip-ver6
• replace unzip with https://oss.oracle.com/el4/unzip/unzip.tar
• Run Rapid Install to install the upgrade file system
• Enable Maintenance Mode
• Perform Task from 2057857.1 to avoid relink errors
• Merge Upgrade Driver 7461070,9474320,9179588
• ADPatch Merged Upgrade Driver from /u03/soft/upgrade_driver
• Download Apply PreInstall patch as per Note : 1448102.2
• mv /dev/random /dev/random_original;ln -s /dev/urandom /dev/random
• SQL> GRANT UNLIMITED TABLESPACE TO apps and applsys
• Apply PreInstall + u6678700.drv -- following Note : 1448102.2
• Disable Maintenance Mode
• Run Rapid Install to Configuration the Application 12.1.1
• Apply FIx 879522.1
• Copy JRE from MT to DB keep under appsutil folder
• Perform Finish the Upgrade Task
• Help Patch u6746000.drv
• Backup

Refer to the Index Page to see the full flow of the upgrade. 

EBS Upgrade Part 6 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

EBS Upgrade Part 6 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

With the Database now running on OEL5 , the 11g Database did not open properly so decided to move to OEL6 server. Now follow the below procedure to migrate Database 11gR2 from OEL5 64Bit to OEL6 64Bit. 

• Shutdown DB
• Backup Database and OraData
• Move Binaries from OEL5 to OEL6
• Create User/Group and RPM/Kernel Settings
• Unzip Oracle Home and OraData , Create User/Groups
• Cloning an Oracle Home
• Set Environment for SID EBSDB
• Create spfile to pfile
• Modify Refrences for file system mount
• Create pfile from spfile
• ReCreate Control File by taking trace
• Validate Alert.log
• Start Database
• alter system set compatible='11.2.0.4' scope=spfile;
• Restart Database and validate alert.log
• alter system set optimizer_features_enable='11.2.0.4' scope=spfile;
• Restart Database and validate alert.log
• Configure Listener
• Backup Database and OraData
• Transfer and untar appsutil from 9i MT to 11g MT , NO Autoconfig
• Update EBS 11.5.9 Context File with the New Database Hostname
• Update /etc/hosts with the IP and Hostname
• Run Autoconfig and verify logs for no errors
• Validate apps login from MT 11.5.9
• --DO NOT START APPLICATION 11.5.9 Services---
• Shutdown DB and Apply 11.2.0.4 RDBMS patch ( 7 patches )

Refer to the Index Page to see the full flow of the upgrade. 

EBS Upgrade Part 5 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

 EBS Upgrade Part 5 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

Well now we are on OEL5.11 Server and this would be a best option to move 10g to 11gR2. Here are the below steps that was performed to Upgrade Database 10gR2 to 11gR2

• Install 11.2.0.4 DB Home
• Install Sample CD for 11.2.0.4
• Create nls/data/9idata directory
• Apply 11.2.0.4 RDBMS patch ( 7 patches )
• Perform DB upgrade task as per note 1058763.1 (10-16)
• Skip Post Upgrade Task -- will perform after moving to OEL 6

Refer to the Index Page to see the full flow of the upgrade. 

EBS Upgrade Part 4 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

EBS Upgrade Part 4 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

This is going to be challenging. These decisions involve a lot of understanding about the complexity underlying. So we are on 10g 32-bit on RHEL 4 and want to move to 11g. The best possible route after reading a bunch of oracle documentation came to a conclusion that will build a RHEL 5 OS and do a OS migration. Then post the migration decided to upgrade the database to 11g.

Well there was a limitation for me and may not for all. I need a RHEL 5 server which I do not have. So I had another laptop where I spinned off a Virtual Box and Installed a Guest OS 5.11.

Migrate Database 10gR2 from RHEL4 32Bit to OEL5 64Bit

• Move the 10g64 Cold Backup RHEL to oel5 stage folder
• Install RPM and Kernel Settings
• Install Base 10.2.0.1 64 bit
• Install Companion 10.2.0.1 64 bit
• Install PatchSet 10.2.0.5
• Backup Oracle Home
• Copy OH,ORADATA,CONTROL file from 32bit to 64bit
• Pretask for 64bit Migration
• Set Environment ORACLE_HOME,ORACLE_SID,PATH
• Migrate to 64bit - recreate control file
• Recompile Invalids
• Post Activity
• Backup Oracle Home and OraData Cold Backup

Refer to the Index Page to see the full flow of the upgrade. 

EBS Upgrade Part 3 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

EBS Upgrade Part 3 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

Following Doc ID 761570.1 , Path A: Standard Upgrade Path , this path will take the database to 10g and then further to 11g

Upgrade Database from 9.2.0.8 to 10.2.0.5

• Install base 10.2.0.1 and then Patch to 10.2.0.5
• Migrate to Oracle Portal 10g - skip it
• Backup Context File , Update not required as per Note
• Export OLAP analytical workspaces
• DBUA Upgrade will fail with java exceptions - go Manual
• Follow Document 316889.1 for Manual Upgrade
• Upgrade
• Recompile Invalids
• Post Upgrade Steps 
• Backup

We are now at 10.2.0.5 and the next step is towards 11g. But wait RHEL 4 is 32 bit. What ? See you in the next post.

Refer to the Index Page to see the full flow of the upgrade.

EBS Upgrade Part 2 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

EBS Upgrade Part 2 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

Wow had a change to upgrade 9i database after a long time. 

Upgrade Database from 9.2.0.3 to 9.2.0.8

• Download and Unzip p4547809_92080_LINUX.zip 32bit
• Shutdown Application and Database
• Backup 9.2.0.3 Oracle Home
• Run 9.2.0.8 Patch Set Update on 9.2.0.3 Oracle Home
• Complete Post Install Action on for Patch 4547809
• Database and Server was Hung
• Changed Database to NoArchive Log Mode
• Restarted Post Install Action on for Patch 4547809
• Restart Database and Recompile Invalids
• Start Database and Application + Health Check
• Cold Backup with all Service Down

We are now at 9.2.0.8 which is eligible for a direct upgrade to 10.2.0.5. In the next step I will be upgrading the database to 10g.

Refer to the Index Page to see the full flow of the upgrade.

EBS Upgrade Part 1 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

EBS Upgrade Part 1 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

The first step towards getting this insance up was to build a RHEL 4. Hmm this was a challenge to bring up a OS with RHEL 4 in 2020. So had a old laptop and got the OS up and running then did the below task to get the EBS 11.5.9 Up and Running

• Validate and Untar DB and MT binaries
• perl adcfgclone.pl dbTechStack
• Manually Recover the Database , SID Change to EBSDB
• perl adcfgclone.pl dbconfig
• Database Validation
• Update MT Stage Context File with Fix for IAS/8.0.6 Home
• Update DB SID/Hostname in the Stage Context File
• Fix instconf.drv and remove em entries and files
• perl adcfgclone.pl appsTier
• Post Cloning Start Fails - apply patch and relink all
• Restart Services and Health Check

After restore figured out that the database is 9.2.0.3 and still needs a one layer up level to 9.2.0.4. In the next part I will be upgrading it. 

Refer to the Index Page to see the full flow of the upgrade.

EBS Upgrade 11.5.x Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7 - Index Page

EBS Upgrade 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

It is December and learning time. Was doing a cleanup of old drive and found a TAR Backup of Vision 11.5.9 with Database 9.2.0.3 which I took it 15 years back. I remember the Operating System was RHEL 4.

So this is what I am going to do ....

Going to take this instance to  EBS 12.2 Database 19c OEL 8

This page is going to be the Index Page and I will update all the links for the rest of the journey here. I hope this will help someone.

Index Page

• Part 1 - Build RHEL4 and clone 11.5.9
• Part 2 - Upgrade Database from 9.2.0.3 to 9.2.0.8
• Part 3 - Upgrade Database from 9.2.0.8 to 10.2.0.5
• Part 4 - Migrate Database 10gR2 from RHEL4 32Bit to OEL5 64Bit
• Part 5 - Upgrade Database 10gR2 to 11gR2
• Part 6 - Migrate Database 11gR2 from OEL5 64Bit to OEL6 64Bit
• Part 7 - Upgrading Application from 11.5.9 to 12.1.1
• Part 8 - Upgrade Application 12.1.1 to 12.1.3
• Part 9 - Upgrade EBS 12.1.3 to  Database 19c
• Part 10 - Upgrade from 12.1.3 to 12.2

EBS 12.1 and 12.2 + 19c Database Release

EBS 12.1 and 12.2 Product running 11.2.0.4 Database it is time to move forward to 19c Database Release.

19c CDB/PDB with EBS is a learning curve. Be it upgrade / migration / cloning -- we will have to face Container and Pluggable Database Feature.

So here is my recommendation to be competitive 

• Read EBS PDB/CDB Documents on Support Website 
• Read Cloning Documentation on Support Website
• Next Level : Read RAC PDB/CDB for EBS

I am going to start a series of blog post on these topic. 

Reference Document

• Information Center: Oracle Database 19c with Oracle E-Business Suite 12.2 and 12.1 (Doc ID 2655839.2)
• Using Oracle Database 19c RAC Multitenant (Single PDB) with Oracle E-Business Suite Release 12.1 (MOS Note 2530680.1)
• Using Oracle 19c RAC Multitenant (Single PDB) with Oracle E-Business Suite Release 12.2 (Doc ID 2530665.1)
• Interoperability Notes: Oracle E-Business Suite Release 12.1 with Oracle Database 19c (MOS Note 2580629.1)
• Database Initialization Parameter Settings for Oracle E-Business Suite Release 12 (MOS Note 396009.1)
• Cloning Oracle E-Business Suite Release 12.2 with Rapid Clone (Doc ID 1383621.1)
• Cloning Oracle E-Business Suite Release 12.1 with Multitenant Database using Rapid Clone (Doc ID 2560690.1)
• Cloning Oracle E-Business Suite Release 12.2 with Multitenant Database using Rapid Clone (Doc ID 2552208.1)
• Cloning Oracle E-Business Suite Release 12.2 RAC Enabled Systems with Rapid Clone (Doc ID 1679270.1)
• Oracle E-Business Suite Release 12.1: Adding or Deleting 19c Oracle RAC Nodes (Doc ID 2617780.1)
• Oracle E-Business Suite Release 12.2: Adding or Deleting 19c Oracle RAC Nodes (Doc ID 2608028.1)

vncserver: couldn't find "xauth" on your PATH.

 vncserver: couldn't find "xauth" on your PATH.

bash-3.2$ vncserver

vncserver: couldn't find "xauth" on your PATH.

bash-3.2$ PATH=$PATH:/usr/X/bin:/usr/X11/bin

bash-3.2$ export PATH

bash-3.2$ vncserver

New 'server:2 ()' desktop is server:2

Starting applications specified in /home/server/.vnc/xstartup

Log file is /home/server/.vnc/server:2.log

bash-3.2$

EBS Forms Spinning After Patching

 What a day to start with , got a escalation where customer applied patch over the weekend and then one particular form APXINWKB Invoice Workbench , when clicking the Lines Tab , it spins for ever.

The following is a best practice that can be done quickly to save time to deal with these kind of issues.

• check patch driver for errors
• check invalid objects and if any compile them
• run trace with binds from EBS forms and analyze it
• run FRD trace and analyze it
• use adadmin and compile all the forms/reports/pll for the relevant module.
• use adadmin and perform compile apps schema

sometimes generic adadmin task can fix complex problem

Grid 19c - PRVG-2031 - Grid Files Ownership Got Changed

PRVG-2031 : Owner of file "/u01/app/oracle/diag/crs/rac01/crs/lck" did not match the expected value on node "HOST1". [Expected = "grid(100)" ; Found = "oracle(0)"]

Fix

• # cd /u01/app/19c/grid/crs/install
• # ./rootcrs.sh -unlock -dstcrshome /u01/app/19c/grid
• # chown -Rf grid:oinstall /u01/app/19c/grid
• # ./rootcrs.sh -lock -dstcrshome /u01/app/19c/grid
• # reboot 

Weblogic Autorestart - Managed Servers

Weblogic autorestart of managed servers might be required if there is a problem with it , like stuck thread , overloaded , aborted or suspended state. The below automation can help to restart based on a certain state of the manged server. In the below I am restarting the managed server when state is other than HEALTH_OK. feel free to customize it to your requirement.

oracle@marsweblogic:~$ cat /export/home/oracle/automation/autorestart/wls_autorestart.py

import os

import re

import string

import array

import java.lang

from jarray import array as jarray_c

from java.util import Hashtable

from javax.management import MBeanServerConnection

from javax.management import ObjectName

from java.lang import String

from java.lang import Object

connect('weblogic','xxxxxxxx','t3://192.168.0.1:7001')

domainRuntime();

cd('ServerRuntimes')

servers = domainRuntimeService.getServerRuntimes();

stoppedServers = []

for server in servers:

 try:

        cd ('/ServerRuntimes/' + server.getName())

        CS = get('OverallHealthState').getState()

        if CS == 0:

                print server.getName(), "= HEALTH_OK"

        elif CS == 1:

                print server.getName(), "= HEALTH_WARN :"

                ms = server.getName()

                shutdown(server.getName(), 'Server', force="true")

                start(ms, 'Server')

        elif CS == 2:

                print server.getName(), "= HEALTH_CRITICAL :"

                ms = server.getName()

                shutdown(server.getName(), 'Server', force="true")

                start(ms, 'Server')

        elif CS == 3:

                print server.getName(), "= HEALTH_FAILED :"

                ms = server.getName()

                shutdown(server.getName(), 'Server', force="true")

                start(ms, 'Server')

        elif CS == 4:

                print server.getName(), "= HEALTH_OVERLOADED :"

                ms = server.getName()

                shutdown(server.getName(), 'Server', force="true")

                start(ms, 'Server')

        else:

                print server.getName() + ': ' + get('State') + ': UNKNOWN HEALTH STATE (' + currentState + ')'

 except WLSTException, e:

                print server.getName() + " is not running."

                stoppedServers.append(server.getName())

disconnect()

exit()

oracle@marsweblogic:~$

Oracle Database TDE Enabling Automation

Oracle Database TDE Enabling Automation on a huge database can be made faster by parallelized tasks. The below automation will help to achieve it on Linux. 

The logic used here is

• build the sql for altering the datafile encrypt 
• find the total number of CPU
• split the workload - datafile/cpu
• submit all the jobs in the background
• monitor the job completion
• validate if all datafiles are encrypted

#Beginning of the Script

echo "TDE - PreStage - `date` - Started"

echo "TDE - PreStage - `date` -- Building SQL for Encrypting Tablespace - Started"

rm tablespace_tde_pre.sql tablespace_tde_exec.sql tablespace_tde_job_*

echo "set pagesize 1000" >  tablespace_tde_pre.sql

echo "set lines 200"     >> tablespace_tde_pre.sql

echo "set define off"    >> tablespace_tde_pre.sql

echo "set term off"      >> tablespace_tde_pre.sql

echo "set head off"      >> tablespace_tde_pre.sql

echo "set feedback off"  >> tablespace_tde_pre.sql

echo "spool tablespace_tde_exec.sql" >> tablespace_tde_pre.sql

echo "select 'alter database datafile '||chr(39)||df.name||chr(39)||' encrypt;' from v\$tablespace ts, v\$datafile df where ts.ts#=df.ts# and (ts.name not in ('SYSTEM','SYSAUX') and ts.name not in (select value from gv\$parameter where name='undo_tablespace'));" >> tablespace_tde_pre.sql

echo "spool off" >> tablespace_tde_pre.sql

echo "exit" >> tablespace_tde_pre.sql

sqlplus -s / as sysdba @tablespace_tde_pre.sql

sed -i '/^$/d' tablespace_tde_exec.sql

echo "TDE - PreStage - `date` -- Building SQL for Encrypting Tablespace - Completed"

echo "TDE - PreStage - `date` -- Building Parallel SQL for Encrypting Tablespace - Started"

cpuCount=`nproc`

echo "TDE - PreStage - `date` -- CPU Count is $cpuCount"

dbfileCount=`cat tablespace_tde_exec.sql|wc -l`

echo "TDE - PreStage - `date` -- DB File Count is $dbfileCount"

if [ $dbfileCount -lt $cpuCount ]

then

parallelJobCount=1

else

parallelJobCount=`expr $dbfileCount / $cpuCount`

fi

echo "TDE - PreStage - `date` -- Number of Datafiles in a Job is $parallelJobCount"

split -l $parallelJobCount tablespace_tde_exec.sql tablespace_tde_job_

echo "TDE - PreStage - `date` -- Created $cpuCount Jobs"

echo "TDE - PreStage - `date` -- Building Parallel SQL for Encrypting Tablespace - Completed"

echo "TDE - Execution - `date` -- Executing Parallel SQL for Encrypting Tablespace - Started"

echo "sqlplus / as sysdba << EOF"  > tablespace_tde_jobs.template

echo "set echo on               " >> tablespace_tde_jobs.template  

for i in `ls tablespace_tde_job_*`

do

cp tablespace_tde_jobs.template tablespace_tde_jobs.t

cat $i     >> tablespace_tde_jobs.t

echo "EOF" >> tablespace_tde_jobs.t

cp tablespace_tde_jobs.t $i

nohup sh $i > $i.log 2>&1 &

echo "TDE - Execution - `date` -- Job $i Submitted - Log $i.log"

done

echo "TDE - Execution - `date` -- Executing Parallel SQL for Encrypting Tablespace - Completed"

echo "TDE - Execution - `date` -- Monitoring Parallel SQL for Encrypting Tablespace - Started"

runningTDEJobs=`ps -ef|grep -i tablespace_tde_job|grep -v grep|wc -l`

while [ $runningTDEJobs -gt 0 ]

do

echo "TDE - Execution - `date` -- Running TDE Jobs"

ps -ef|grep -i tablespace_tde_job|grep -v grep

sleep 30

runningTDEJobs=`ps -ef|grep -i tablespace_tde_job|grep -v grep|wc -l`

done

echo "TDE - Execution - `date` -- Monitoring Parallel SQL for Encrypting Tablespace - Completed"

echo "TDE - PostStage - `date` -- Verification of Tablespace Encryption - Started"

sqlplus / as sysdba << EOF

alter database open;

select name from v\$tablespace where ts# not in (select ts# from v\$encrypted_tablespaces);

EOF

echo "Verify if all the tablespaces with the exception of TEMP/UNDO/SYSTEM/SYSAUX is encrypted"

echo "TDE - PostStage - `date` -- Verification of Tablespace Encryption - Completed" 

#End of the Script

Customize it to your need. 😵

Weblogic Managed Server - Rolling Restart Automation

Rolling Restart on Weblogic is sometimes done on a periodic manner to get things cleared. Well I am not favor of doing rolling restart but sometimes this is the solution. 

This handy script can help 😊

oracle@wls-ser1:~$ cat /export/home/oracle/automation/autorestart/wls_rollingrestart.py

import os

import re

import string

import array

import java.lang

import time as systime

from jarray import array as jarray_c

from java.util import Hashtable

from javax.management import MBeanServerConnection

from javax.management import ObjectName

from java.lang import String

from java.lang import Object

connect('weblogic','weblogic123','t3://localhost:7001')

domainRuntime();

cd('ServerRuntimes')

servers = domainRuntimeService.getServerRuntimes();

stoppedServers = []

for server in servers:

 try:

        cd ('/ServerRuntimes/' + server.getName())

        CS = get('OverallHealthState').getState()

        if CS == 0:

                print server.getName(), "= HEALTH_OK"

        elif CS == 1:

                print server.getName(), "= HEALTH_WARN :"

        elif CS == 2:

                print server.getName(), "= HEALTH_CRITICAL :"

        elif CS == 3:

                print server.getName(), "= HEALTH_FAILED :"

        elif CS == 4:

                print server.getName(), "= HEALTH_OVERLOADED :"

        else:

                print server.getName() + ': ' + get('State') + ': UNKNOWN HEALTH STATE (' + currentState + ')'

        ms = server.getName()

        if ms != "AdminServer":

                print ms, " Shutdown Initiated"

                systime.sleep(10)

                shutdown(server.getName(), 'Server', force="true")

                print ms, " Shutdown Completed"

                print ms, " Startup Initiated"

                start(ms, 'Server')

                systime.sleep(10)

                print server.getName(), " Startup Completed"

        else:

                print "AdminServer - no action performed"

        CS = get('OverallHealthState').getState()

        if CS == 0:

                print server.getName(), "= HEALTH_OK"

        elif CS == 1:

                print server.getName(), "= HEALTH_WARN :"

        elif CS == 2:

                print server.getName(), "= HEALTH_CRITICAL :"

        elif CS == 3:

                print server.getName(), "= HEALTH_FAILED :"

        elif CS == 4:

                print server.getName(), "= HEALTH_OVERLOADED :"

        else:

                print server.getName() + ': ' + get('State') + ': UNKNOWN HEALTH STATE (' + currentState + ')'

 except WLSTException, e:

                print server.getName() + " is not running."

                stoppedServers.append(server.getName())

disconnect()

exit()

oracle@wls-ser1:~$

 

oracle@wls-ser1:~$ cat /export/home/oracle/automation/autorestart/wls_rollingrestart.sh

log='/export/home/oracle/automation/autorestart/wls_rollingrestart.log'

date >> $log

echo >> $log

. /u01/app/oracle/product/fmw/middleware/wlserver/server/bin/setWLSEnv.sh >> $log

java weblogic.WLST -skipWLSModuleScanning /export/home/oracle/automation/autorestart/wls_rollingrestart.py >> $log

echo >> $log

echo >> $log

echo "-----------------------------------------------------------------------------" >> $log

oracle@wls-ser1:~$

oracle@wls-ser1:~$ crontab -l

10 0 * * 5   /export/home/oracle/automation/autorestart/wls_rollingrestart.sh

oracle@wls-ser1:~$

Execution Log

-----------------------------------------------------------------------------

Monday, July 13, 2020 at  5:20:12 PM +03

CLASSPATH=/usr/jdk/instances/jdk1.8.0/lib/tools.jar:/u01/app/oracle/product/fmw/middleware/wlserver/modules/features/wlst.wls.classpath.jar:/usr/jdk/instances/jdk1.8.0/lib/tools.jar:/u01/app/oracle/product/fmw/middleware/wlserver/modules/features/wlst.wls.classpath.jar:

PATH=/u01/app/oracle/product/fmw/middleware/wlserver/server/bin:/u01/app/oracle/product/fmw/middleware/wlserver/../oracle_common/modules/thirdparty/org.apache.ant/1.9.8.0.0/apache-ant-1.9.8/bin:/usr/jdk/instances/jdk1.8.0/jre/bin:/usr/jdk/instances/jdk1.8.0/bin:/usr/bin:/usr/sbin:/u01/app/oracle/product/fmw/middleware/wlserver/../oracle_common/modules/org.apache.maven_3.2.5/bin:/u01/app/oracle/product/fmw/middleware/wlserver/../oracle_common/modules/org.apache.maven_3.2.5/bin

Your environment has been set.

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Connecting to t3://localhost:7001 with userid weblogic ...

Successfully connected to Admin Server "AdminServer" that belongs to domain "ms_domain".

Warning: An insecure protocol was used to connect to the server.

To ensure on-the-wire security, the SSL port or Admin port should be used instead.

Location changed to domainRuntime tree. This is a read-only tree

with DomainMBean as the root MBean.

For more help, use help('domainRuntime')

ms_server02 = HEALTH_OK

ms_server02  Shutdown Initiated

Shutting down the server ms_server02 with force=true while connected to AdminServer ...

.

ms_server02  Shutdown Completed

ms_server02  Startup Initiated

Starting server ms_server02 ......................................................

Server with name ms_server02 started successfully

ms_server02  Startup Completed

ms_server02 = HEALTH_OK

ms_server01 = HEALTH_OK

ms_server01  Shutdown Initiated

Shutting down the server ms_server01 with force=true while connected to AdminServer ...

.

ms_server01  Shutdown Completed

ms_server01  Startup Initiated

Starting server ms_server01 ................................................

Server with name ms_server01 started successfully

ms_server01  Startup Completed

ms_server01 = HEALTH_OK

ms_server04 = HEALTH_OK

ms_server04  Shutdown Initiated

Shutting down the server ms_server04 with force=true while connected to AdminServer ...

.....

ms_server04  Shutdown Completed

ms_server04  Startup Initiated

Starting server ms_server04 ..................................................

Server with name ms_server04 started successfully

ms_server04  Startup Completed

ms_server04 = HEALTH_OK

ms_server03 = HEALTH_OK

ms_server03  Shutdown Initiated

Shutting down the server ms_server03 with force=true while connected to AdminServer ...

.

ms_server03  Shutdown Completed

ms_server03  Startup Initiated

Starting server ms_server03 ............................................

Server with name ms_server03 started successfully

ms_server03  Startup Completed

ms_server03 = HEALTH_OK

AdminServer = HEALTH_OK

AdminServer - no action performed

AdminServer = HEALTH_OK

Disconnected from weblogic server: AdminServer

Exiting WebLogic Scripting Tool.

Someone You Loved Cover | Keyboard Performance | Dharun at Improviser Music Studio

Cannot Stop Posting this from my Son

Someone You Loved Cover | Keyboard Performance | Dharun at Improviser Music Studio

EBS 12.2 FORMS CRASH FRM-92101 After 12.2.9 Upgrade when submitting Active User

If you are an EBS administrator the first thing some asks you to do a health check of an environment , you return to him with a Request ID of Active Users which is Completed. It ensures that you have gone through Login Page , OA pages , Forms , Concurrent Mangers , View Output. 80% of the EBS would be in a healthy state if you get a Request ID.

This case all went fine with the upgrade to 12.2.9 and then something strange happened when we Submit Active User or any other requests. In Oracle Support there are specific notes that are discussing this exception and it will direct you to apply patches. Well I did apply the patches but did not help

The below exception messages are seen in the forms immediately after clicking the Submit Button.

FRM-92101: There was a failure in the Firms Server during startup. This could happen due to invalid configuration. 

oracle.forms.net.ConnectionException: Forms session <3> aborted: unable to communicate with runtime process.

Errors seen in the forms logs are below

####<Jul 2, 2020 1:41:08 PM AST> <Error> <ServletContext-/forms> <bangalore.testserver.com> <forms_server1> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <00060qTRvSjFc5T_u98DyX0001Z400001Q> <1593686468004> <BEA-000000> <Forms session <3> exception stack trace:

java.io.IOException: FRM-93000: Unexpected internal error.

Details : No HTTP headers received from runform

        at oracle.forms.servlet.ListenerServlet.forwardResponseFromRunform(Unknown Source)

        at oracle.forms.servlet.ListenerServlet.doPost(Unknown Source)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)

        at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)

        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)

        at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)

        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)

        at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:138)

        at java.security.AccessController.doPrivileged(Native Method)

        at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:324)

        at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:464)

        at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:121)

        at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:211)

        at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)

        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)

Fix : after implementing many fixes including some known issue patches , this error did not change. Also it might be worth to validate Operating System patches , like motif and other libraries which are required for running EBS.

The fix for this exception in this environment was to compile FlexFields using adadmin. I hope this helps. 😉