EBS 12.2 APPL_TOP Only Clone - ERRORMSG: Invalid APPS database user credentials.

[oracle@testebsop3app01 ~]$ perl /u01/install/APPS/fs1/EBSapps/comn/clone/bin/adcfgclone.pl appltop /u01/install/APPS/fs1/inst/apps/SATURN_testebsop3app01/appl/admin/SATURN_testebsop3app01.xml

Appl_top only clone might result in the below error

START: Instantiate the txkWfClone.sql...
instantiate file:
   source : /u01/install/APPS/fs1/EBSapps/appl/fnd/12.0.0/admin/template/txkWfClone.sql
   dest   : /u01/install/APPS/fs1/inst/apps/SATURN_testebsop3app01/admin/install/txkWfClone.sql
Adding execute permission to : /u01/install/APPS/fs1/inst/apps/SATURN_testebsop3app01/admin/install/txkWfClone.sql

END: Instantiated txkWfClone.sql...

START: Executing /u01/install/APPS/fs1/inst/apps/SATURN_testebsop3app01/admin/install/txkWfClone.sh -nopromptmsg
txkWfClone.sh exited with status 1
ERROR: txkWfClone.sh execution failed, exit code 1
[oracle@testebsop3app01 clone_10091511]$


if you scroll up the logs you may find some more errors

TXK Script Name: txkChkFormsLaunchMethod.pl
Enter the APPS user password:
*******FATAL ERROR*******
PROGRAM : (/u01/install/APPS/fs1/EBSapps/appl/fnd/12.0.0/patch/115/bin/txkChkFormsLaunchMethod.pl)
TIME    : Wed Oct  9 14:50:36 2019
FUNCTION: main::validateAppsSchemaCredentials [ Level 1 ]
ERRORMSG: Invalid APPS database user credentials.
ERRORCODE = 1 ERRORCODE_END
.end std out.
stty: standard input: Inappropriate ioctl for device
stty: standard input: Inappropriate ioctl for device
*******FATAL ERROR*******
PROGRAM : (/u01/install/APPS/fs1/EBSapps/appl/fnd/12.0.0/patch/115/bin/txkChkFormsLaunchMethod.pl)
TIME    : Wed Oct  9 14:50:36 2019
FUNCTION:  [ Level 1 ]
ERRORMSG: *******FATAL ERROR*******
PROGRAM : (/u01/install/APPS/fs1/EBSapps/appl/fnd/12.0.0/patch/115/bin/txkChkFormsLaunchMethod.pl)
TIME    : Wed Oct  9 14:50:36 2019
FUNCTION: main::validateAppsSchemaCredentials [ Level 1 ]
ERRORMSG: Invalid APPS database user credentials.


The problem is tnsnames.ora does not have an entry for TWO_TASK


from MT , sqlplus apps/apps@TWO_TASK will fail to connect.

Fix :
run autoconfig on the DB node
run adcfgclone.pl appltop

Not All Endpoints Registered

Not All Endpoints Registered

crsctl stat res -t

 --------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_LSTNSOLTP .lsnr
     1        ONLINE  INTERMEDIATE orcl1           Not All Endpoints Registered,STABLE
ora.LISTENER_LSTNSOLTP .lsnr
     1        ONLINE  ONLINE       orcl2           STABLE
ora.LISTENER_LSTNSOLTP .lsnr
     1        ONLINE  ONLINE       orcl3           STABLE

"Not All Endpoints Registered"

To fix the problem
- compare the listener.ora of all the nodes
- find the differences and sync it up

if you find static entries make sure you stop the listener running with "LSNRCT STOP " otherwise you might end up with duplicate processes

$  ps -ef|grep -i LSTNSOLTP
grid     312080      1  0 03:17 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/tnslsnr LSTNSOLTP -no_crs_notify -inherit
$

-- found static entry in listener.ora

-- fixed listener.ora by removing the static entry

-- used crsctl to start the listener

$ ps -ef|grep -i LSTNSOLTP
grid     166779      1  0 Sep18 ?        00:14:53 /u01/app/12.1.0.2/grid/bin/tnslsnr LSTNSOLTP -no_crs_notify -inherit
grid     267334      1  0 03:07 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/tnslsnr LSTNSOLTP -no_crs_notify -inherit
$

correct way to approach this problem


$  ps -ef|grep -i LSTNSOLTP
grid     312080      1  0 03:17 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/tnslsnr LSTNSOLTP -no_crs_notify -inherit
$

$ lsnrctl stop LSTNSOLTP


$  ps -ef|grep -i LSTNSOLTP
$

Make sure no processes are running

-- fix listener.ora by removing the static entry

-- use crsctl to start the listener

$ ps -ef|grep -i LSTNSOLTP
grid     267334      1  0 03:07 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/tnslsnr LSTNSOLTP -no_crs_notify -inherit
$


crsctl stat res -t

 --------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_LSTNSOLTP .lsnr
     1        ONLINE  ONLINE       orcl1           STABLE
ora.LISTENER_LSTNSOLTP .lsnr
     1        ONLINE  ONLINE       orcl2           STABLE
ora.LISTENER_LSTNSOLTP .lsnr
     1        ONLINE  ONLINE       orcl3           STABLE