Let me start telling a story , "Once upon a time there lived a great ♚EBS 12.2 instance , it was very kind to the DBAs and one fine day adpreclone appsTier got hung for unknown reason" interesting ???
$ perl adpreclone appsTier
....
....
....
hung 🙈🙉🙊
[oracle@kingdom <RUN> ]$ pwd
/u01/install/APPS/fs1/inst/apps/KING_kingdom/admin/log/clone
[oracle@kingdom <RUN> ]$ ls -lrt
total 26
drwxr-xr-x. 2 oracle oinstall 4 Jun 26 11:04 wlsT2PStage
drwxr-xr-x. 2 oracle oinstall 2 Jun 26 11:04 txkSetOHSConfig_06261104
drwxr-xr-x. 2 oracle oinstall 4 Jun 26 11:04 ohsT2PStage
-rw-r--r--. 1 oracle oinstall 20445 Jun 26 11:04 StageAppsTier_06261009.log
drwxr-xr-x. 2 oracle oinstall 2 Jun 29 01:17 fmwT2PStage
[oracle@kingdom <RUN> ]$
looking at StageAppsTier_datetime.log the below copyBinary was running
/u01/install/APPS/fs1/FMW_Home/oracle_common/bin/copyBinary.sh -javaHome /u01/install/APPS/fs1/EBSapps/comn/clone/FMW/t2pjdk -al /u01/install/APPS/fs1/EBSapps/comn/clone/FMW/FMW_Home1234.jar -smw /u01/install/APPS/fs1/FMW_Home -ldl /u01/install/APPS/fs1/inst/apps/KING_kingdom/admin/log/clone/fmwT2PStage -invPtrLoc /etc/oraInst.loc -silent true -debug true
getting deeper into fmwT2PStage found the below
FINE : Jun 26, 2020 10:55:25 - CLONE-22065 The statistics of files whose permission will be handled:
FINE : Jun 26, 2020 10:55:25 - CLONE-22066 The number of files with root privilege=0.
FINE : Jun 26, 2020 10:55:25 - CLONE-22067 The number of symbolic link files=12.
...
...
...
...
INFO : Jun 26, 2020 10:55:25 - CLONE-21118 Adding symbolic link files detail to the archive started ...
FINE : Jun 26, 2020 10:55:25 - CLONE-22209 Number of entries in the file, which will contain all valid symbolic link files detail=12.
FINE : Jun 26, 2020 10:55:25 - [FileUtil:deleteDir] /u01/install/APPS/fs1/FMW_Home/webtier/clone/CloningService_OH_All_Simlink_file path does not exist, so it will not be deleted.
FINE : Jun 26, 2020 10:55:25 - [JarUtilArchiveForOracleHome:addAllFileNamesFromSimlinkListToJarStream] Archiving file which contains simlink file information...
FINE : Jun 26, 2020 10:55:25 - CLONE-22061 The archive entry point=oraclehomes/ORACLEHOME_WEBTIER_CLONE20200626101420859-oraclehome2@webtier/clone/.
FINE : Jun 26, 2020 10:55:25 - [JarUtilArchiveForOracleHome:addAllFileNamesFromSimlinkListToJarStream] Archive of file which contains no read access file information completed.
FINE : Jun 26, 2020 10:55:25 - [FileUtil:deleteDir] Deleting /u01/install/APPS/fs1/inst/apps/KING_kingdom/logs/appl/rgf/TXK/CLONINGCLIENT-8495823447040830722/1593186925232_allsimlinkFilePath/CloningService_OH_All_Simlink_file
FINE : Jun 26, 2020 10:55:25 - [FileUtil:deleteDir] Deleting /u01/install/APPS/fs1/inst/apps/KING_kingdom/logs/appl/rgf/TXK/CLONINGCLIENT-8495823447040830722/1593186925232_allsimlinkFilePath
INFO : Jun 26, 2020 10:55:25 - CLONE-21119 Adding symbolic link files detail to the archive finished. Operation complete in "23" milliseconds.
It got stuck with the symbolic link processing. Analyzing the logs found that there are 12 files but there were more files under the FMW_HOME. After checking all the symlinks found few strange links that were created as root user.
[oracle@kingdom <RUN> ]$ ls -lR . | grep ^l
The above command will list down all the symlinks , carefully review them to find the problematic one. So in my case there was a symlink that is created as root. Recreated the symlink as oracle user and there was another broken symlink , deleted it.
Restarted the adpreclone tool
INFO : Jun 26, 2020 10:55:25 - CLONE-21118 Adding symbolic link files detail to the archive started ...
FINE : Jun 26, 2020 10:55:25 - CLONE-22209 Number of entries in the file, which will contain all valid symbolic link files detail=12.
FINE : Jun 26, 2020 10:55:25 - [FileUtil:deleteDir] /u01/install/APPS/fs1/FMW_Home/webtier/clone/CloningService_OH_All_Simlink_file path does not exist, so it will not be deleted.
FINE : Jun 26, 2020 10:55:25 - [JarUtilArchiveForOracleHome:addAllFileNamesFromSimlinkListToJarStream] Archiving file which contains simlink file information...
FINE : Jun 26, 2020 10:55:25 - CLONE-22061 The archive entry point=oraclehomes/ORACLEHOME_WEBTIER_CLONE20200626101420859-oraclehome2@webtier/clone/.
FINE : Jun 26, 2020 10:55:25 - [JarUtilArchiveForOracleHome:addAllFileNamesFromSimlinkListToJarStream] Archive of file which contains no read access file information completed.
FINE : Jun 26, 2020 10:55:25 - [FileUtil:deleteDir] Deleting /u01/install/APPS/fs1/inst/apps/KING_kingdom/logs/appl/rgf/TXK/CLONINGCLIENT-8495823447040830722/1593186925232_allsimlinkFilePath/CloningService_OH_All_Simlink_file
FINE : Jun 26, 2020 10:55:25 - [FileUtil:deleteDir] Deleting /u01/install/APPS/fs1/inst/apps/KING_kingdom/logs/appl/rgf/TXK/CLONINGCLIENT-8495823447040830722/1593186925232_allsimlinkFilePath
INFO : Jun 26, 2020 10:55:25 - CLONE-21119 Adding symbolic link files detail to the archive finished. Operation complete in "23" milliseconds.
FINE : Jun 26, 2020 10:55:27 - [JarUtilArchiveForOracleHome:addSrcDirWithExcludeList] Size of the Oracle home =2476042252
INFO : Jun 26, 2020 10:55:27 - CLONE-21053 Total size of the file(s), to be included in the main jar from source location /u01/install/APPS/fs1/FMW_Home/webtier=2.476 GB.
5823447040830722/1593186925232_allsimlinkFilePath
INFO : Jun 26, 2020 10:55:25 - CLONE-21119 Adding symbolic link files detail to the archive finished. Operation complete in "23" milliseconds.
FINE : Jun 26, 2020 10:55:27 - [JarUtilArchiveForOracleHome:addSrcDirWithExcludeList] Size of the Oracle home =2476042252
INFO : Jun 26, 2020 10:55:27 - CLONE-21053 Total size of the file(s), to be included in the main jar from source location /u01/install/APPS/fs1/FMW_Home/webtier=2.476 GB.
FINE : Jun 26, 2020 10:55:27 - CLONE-22061 The archive entry point=oraclehomes/ORACLEHOME_WEBTIER_CLONE20200626101420859-oraclehome2@webtier/.
FINE : Jun 26, 2020 10:55:27 - [FileUtil:deleteDir] /u01/install/APPS/fs1/FMW_Home/webtier/clone/CloningService_OH_All_File_Path path does not exist, so it will not be deleted.
INFO : Jun 26, 2020 10:56:03 - CLONE-21055 In 36 seconds, 12% has completed.
INFO : Jun 26, 2020 10:56:09 - CLONE-21055 In 42 seconds, 23% has completed.
INFO : Jun 26, 2020 10:56:52 - CLONE-21055 In 85 seconds, 39% has completed.
INFO : Jun 26, 2020 10:57:19 - CLONE-21055 In 112 seconds, 50% has completed.
INFO : Jun 26, 2020 10:57:26 - CLONE-21055 In 119 seconds, 61% has completed.
INFO : Jun 26, 2020 10:57:31 - CLONE-21055 In 124 seconds, 73% has completed.
INFO : Jun 26, 2020 10:57:46 - CLONE-21055 In 138 seconds, 89% has completed.
this time it passed that stage and it was successful.... ✌