Diagnosing Problems with SOA Composite Applications on Exalogic

I am not going to write a story on the issue but it is very much described on the forum post SOA Suite 11g composite application deployment to SOA cluster?

"when customer deploys SOA composite it only gets deployed to one of the SOA managed servers in the cluster and the only way for them to sync up SOA1 and SOA2 is by bouncing the servers. Is there a best practice on how to deploy composite apps to SOA clustered env?"

This issue happened on an Exalogic X3-2 Quarter Rack. When composite got deployed on SOA1 it did not get deployed on SOA2 and the only work around was to restart SOA2 and it gets deployed. Though the workaround did not stop the golive but it happened right on the production.

Exalogic administrators discovered the issue to be Duplicate IP deduction and they brought down the duplicate. A complete restart and Virtual IP were up and deployment issue was resolved.

Now I will throw some information on how an exalogic/weblogic administrator can find this  kind of conflict without having a need of Machine Administrator.

The /etc/hosts on the two exalogic compute nodes looks like below for the virtual IP's
10.200.10.101     exalogic001-admin.exadomain.com        exalogic001-admin
10.200.10.102     exalogic001-soa1.exadomain.com         exalogic001-soa1
10.200.10.103     exalogic001-bam1.exadomain.com         exalogic001-bam1
10.200.10.104     exalogic001-osb1.exadomain.com         exalogic001-osb1
10.200.10.105     exalogic002-soa2.exadomain.com         exalogic002-soa2
10.200.10.106     exalogic002-osb2.exadomain.com         exalogic002-osb2

From compute node 1 and 2 if the command wlsifconfig.sh is invoked with the listif option it gives a formatted output of "/sbin/ip -o addr"

ComputeNode1>$WL_HOME/common/bin/wlsifconfig.sh -listif bond1
bond1 10.200.10.100
bond1:1 10.200.10.101
bond1:3 10.200.10.105
bond1:4 10.200.10.103
bond1:5 10.200.10.104
bond1:6 10.200.10.102
ComputeNode1>

ComputeNode2>$WL_HOME/common/bin/wlsifconfig.sh -listif bond1
bond1 10.200.10.100
bond1:1 10.200.10.106
bond1:2 10.200.10.105
ComputeNode2>

The interface name can be found on the nodemanager.properties file here it is bond1
10.200.10.105 belongs to exalogic002-soa2.exadomain.com
10.200.10.102 belongs to exalogic001-soa1.exadomain.com

Understanding why 10.200.10.105 is up on both the compute node is a different story but you can clearly see that 105 is up on both the nodes. This could be the reason why SOA composite deployment on compute node 1 could not automatically deploy on compute node 2. The solution was to bring down 105 on compute node 1 because it does not belong there. Later a complete restart of the SOA cluster was made to ensure that the VIP's are started in the correct compute nodes and the deployments were successful.


ComputeNode1>$WL_HOME/common/bin/wlsifconfig.sh -listif bond1
bond1 10.200.10.100
bond1:1 10.200.10.101
bond1:4 10.200.10.103
bond1:5 10.200.10.104
bond1:6 10.200.10.102
ComputeNode1>

ComputeNode2>$WL_HOME/common/bin/wlsifconfig.sh -listif bond1
bond1 10.200.10.100
bond1:1 10.200.10.106
bond1:2 10.200.10.105
ComputeNode2>

Hope a Root Cause Analysis will have a permanent fix.

Post 12c Upgrade - Converting Non-CDB as PDB

The upgrade process is getting fine tuned on every release and in 12c we are getting a much better features. A lot of automation has been put in the 12c upgrade process. The pre-upgrade phase has now more validation with the apply action option where the assistance generates a fix script and that can be run to fix the issues. If you have more CPU cores and want to make use of it then 12c upgrade assistance has an answer for it. Assistance allows you to select the parallelism which will speed up the upgrade process. 

Now we have a database upgraded to 12c from 11g. The upgraded database is not a PDB it is a NON-CDB. I will go ahead and convert into a PDB.

So the process is simple , from the 12c Oracle Home start DBCA and create a CDB Instance with one PDB. Then you have all the necessary setup for the migration. Now with few simple steps one can migrate to the CDB instance. 

In this demo my Non-CDB name is UPGDB and the newly created CDB instance name is UPGCDB , it has one PDB named PDB001 and will later add UPGDB into the CDB as a PDB.

With the new CDB created let us start adding the UPGDB (Non-CDB) into it.

Startup the Non-CDB database and make sure that you also have the CDB running.

Now connect to the Non-CDB and use the PDB Describe package to create the xml file with all the necessary information. You can take a look at the xml file and understand its internal specifications.

Review the xml file and Shutdown the Non-CDB database

Now connect to the CDB database and use the dbms_pdb database package to check the compatibility of the Non-CDB database by passing the xml file generated.

Use the pdb_plug_in_violations view to know the status.

Refer to Oracle 12c upgrade guide for any errors reported.

Now issue the create pluggable database command using the xml file with COPY option.

COPY option will copy the data files to the new location , if you do not want to then use the MOVE option which will move the data files. If you want to retain the data files in the same location then use NOCOPY option. The default one is COPY.

Once the PDB is created make sure that there are no errors in the alert.log

Verify the status of the new PDB , it is NEW. Now run the noncdb_to_pdb.sql to integrate the converted PDB into the container CDB.

Open the PDB to complete the integration.

Verify the same from alert log.

Check the pdb_plug_in_violations view to make sure that the Non-CDB to PDB has RESOLVED status.

 With this we have successfully converted Non-CDB to PDB

Later you can invoke DBCA to delete the Non-CDB entries.

Independence Day Wishes

Managing 12c with SQL Developer : RMAN Recover and Restore

Read me before you start 

Restoring and Recovering a 12c Database is simple and easy. I am here restoring complete database including CDB and PDB. Will cover a separate topic on restoring PDB alone. But this one is simple. The only difficulty I had was after I initiated restore I cannot get any status since the script was running in the backend server. So I had to monitor the state of the database and some custom scripts on the RMAN status.