APR2016 combo ojvm + psu patch

Strange behaviour we noticed yesterday. we downloaded and installed the combo PSU OJVM patch p22738657 using opatch version 12.1.0.1.10 we deployed it both on Linux x86_64 as on Solaris SPARC. using opatchauto in the case of linux and using a generated oplan in the case of SPARC amazed to see that it looks like in both cases the OJVM patch wasn't install

[grid@charel008 ~]$ opatch lspatches
23006522;OCW Interim patch for 23006522
22806133;DATABASE BUNDLE PATCH: 12.1.0.2.160419 (22806133)
22674709;Database PSU 12.1.0.2.160419, Oracle JavaVM Component (Apr2016)
22502518;ACFS Patch Set Update : 12.1.0.2.160419 (22502518)
21436941;WLM Patch Set Update: 12.1.0.2.5 (21436941)
 
OPatch succeeded.

  
[oracle@charel008 ~]$ opatch lspatches
23006522;OCW Interim patch for 23006522
22806133;DATABASE BUNDLE PATCH: 12.1.0.2.160419 (22806133)
 
OPatch succeeded.

I checked a bit the usual directories to check and checked in

$ORACLE_HOME/.patch_storage/NApply/2016-05-25_10-48-41AM
 

And I don’t see the ojvm patch there However I see it here ….in grid home

 $GRID_HOME/.patch_storage/NApply/2016-05-25_11-06-16AM/
[ogrd@bert007 2016-05-25_11-06-16AM]$ ls
patchlist.txt
[ogrd@bert007 2016-05-25_11-06-16AM]$ cat patchlist.txt
User invokes OPatch via "NApply" to process the following patches:
 
snipped...

Patch ID: 22674709
Patch Location: "/u01/stage/patch_apr/22738657/22674709"
Patch storage area: "/u01/app/grid/product/12.1.0/grid/.patch_storage/22674709_Apr_5_2016_08_56_18"
 

then In bundle.xml in the patch directory

            
                
                    Oracle Database
                    Oracle 12.1.0.2.0
                
                
                    Oracle Database
                    Oracle 12.1.0.2.0
                
            
        
        
            
                
                    Oracle Database
                    Oracle 12.1.0.2.0
                
                
                    Oracle Database
                    Oracle 12.1.0.2.0
                
                
                    Oracle Database
                    Oracle 12.1.0.2.0
                
                
                    Oracle Database
                    Oracle 12.1.0.2.0
                
            
        
 

While for the patch straight under it is put type=”oracle_datababase” so I think the issue is there in the bundle.xml in the patch directory. Let's see what support says UPDATE : Support wasn't very clear about this and kept on saying that this patch shouldn't be installed on a non Exadata system, while this isn't the case. " Overview of Database Patch Delivery Methods ( Doc ID 1962125.1 ) Database Patch for Engineered Systems and Database In-Memory Short Name DBBP Target environment/s 12c DB installs on Exadata. Also applicable to any 12c system using Database In-Memory (DBIM) features, RAC or non-RAC. Applicable to both Database and Grid homes. DB Versions 12.1.0.2 onwards only Platform/s Unix platforms, including Exadata systems, but also for any system using DB In-Memory (DBIM) Released Released monthly. Patch Content Includes miscellaneous fixes for Engineered Systems along with fixes for Database In-Memory. Critical Patch content is included quarterly. Also includes fixes for the Grid home for Exadata and RAC systems. See Note:888828.1 Exadata Recommended Patch Information Note:756671.1 Current Recommended Patches Recommended Version Recommend to be on the latest quarterly release Notes The patch name sometimes causes confusion - the patches can be used in both Exadata and non-Exadata systems. " this sr took really a long time and came to the wrong conclusion focussing on the workaround which was manually deinstall GI OJVM patch and re-apply the ovm on DB home. After closure we had a meeting with oracle support manager and they think that indeed the issue with the xml file which i pointed out from the beginning ;-)

installing em13c : errors not captured ....

I installed Em 13c last week on Redhat 7.2 the installation was on a vmware vm at the customer that was given to me without explanation. After going through the prereqs and installing missing packages I started the installation of this "temporary" setup. what is OEL so much easier .... Everything went fine up to the configuration of Bi publisher that step failed miserably. Log files showed that the oms which is stopped before didn't come up. digging a bit deeper showed following in the logs :

2016-05-06 17:08:31,778 INFO  [75] oracle.sysman.oii.oiix.OiixSpawner$StreamReader - (if [ "compile" = "compile" ] ; then \
  /u01/app/oem/middelware/bin/gennttab > ntcontab.c ;\
  gcc -m64  -fPIC -c ntcontab.c ;\
  rm -f /u01/app/oem/middelware/lib/ntcontab.o ;\
  mv ntcontab.o /u01/app/oem/middelware/lib/ ;\
          /usr/bin/ar rv /u01/app/oem/middelware/lib/libn11.a /u01/app/oem/middelware/lib/ntcontab.o ; fi)
 
2016-05-06 17:08:32,093 INFO  [76] oracle.sysman.oii.oiix.OiixSpawner$StreamReader - gcc: error trying to exec 'as': execvp: Permission denied
 
2016-05-06 17:08:32,095 INFO  [76] oracle.sysman.oii.oiix.OiixSpawner$StreamReader - mv: cannot stat ‘ntcontab.o’: No such file or directory
 
2016-05-06 17:08:32,102 INFO  [76] oracle.sysman.oii.oiix.OiixSpawner$StreamReader - /usr/bin/ar: /u01/app/oem/middelware/lib/ntcontab.o: No such file or directory
 
2016-05-06 17:08:32,102 INFO  [76] oracle.sysman.oii.oiix.OiixSpawner$StreamReader - make: *** [ntcontab.o] Error 1
 
2016-05-06 17:08:32,103 INFO  [64] oracle.sysman.oii.oiix.OiixSpawner - End output from spawned process.
2016-05-06 17:08:32,103 INFO  [64] oracle.sysman.oii.oiix.OiixSpawner - ----------------------------------
2016-05-06 17:08:32,103 INFO  [64] oracle.sysman.oii.oiis.OiisCompActions - Exception thrown from action: make
Exception Name: MakefileException
Exception String: Error in invoking target 'mkldflags ntcontab.o nnfgt.o' of makefile '/u01/app/oem/middelware/network/lib/ins_net_client.mk'. See '/u01/stage/em/disks/OraInstall2016-05-06_04-01-07PM/install2016-05-06_04-01-07PM.log' for details.
Exception Severity: 1
2016-05-06 17:08:32,103 INFO  [64] oracle.sysman.oii.oiic.OiicAPISessionInterfaceManager - Retry Handler not registered, using Super class functionality
2016-05-06 17:08:32,104 OFF   [64] oracle.sysman.oii.oiio.oiiol.OiiolTextLogger - Error in invoking target 'mkldflags ntcontab.o nnfgt.o' of makefile '/u01/app/oem/middelware/network/lib/ins_net_client.mk'. See '/u01/stage/em/disks/OraInstall2016-05-06_04-01-07PM/install2016-05-06_04-01-07PM.log' for details.
2016-05-06 17:08:32,104 INFO  [64] oracle.sysman.oii.oiis.OiisCompActions - Calling Action unixActions10.2.0.3.0  make  registerOnly = false    installMakePath = /usr/bin/make installMakeFileName = /u01/app/oem/middelware/rdbms/lib/ins_rdbms.mk    installTarget = client_sharedlib        undoMakeFileName =      installArguments = ORACLE_HOME=/u01/app/oem/middelware  logFile = /u01/app/oem/middelware/install/make.log      undoTarget =    progMsg = Building Client Shared Libraries
2016-05-06 17:08:32,105 INFO  [64] oracle.sysman.oii.oiio.oiiol.OiiolTextLogger - The output of this make operation is also available at: '/u01/app/oem/middelware/install/make.log'
2016-05-06 17:08:32,106 INFO  [64] oracle.sysman.oii.oiix.OiixSpawner - Start output from spawned process:
2016-05-06 17:08:32,106 INFO  [64] oracle.sysman.oii.oiix.OiixSpawner - ----------------------------------
2016-05-06 17:08:32,114 INFO  [77] oracle.sysman.oii.oiix.OiixSpawner$StreamReader - /u01/app/oem/middelware/bin/genclntsh
 
2016-05-06 17:08:32,159 INFO  [78] oracle.sysman.oii.oiix.OiixSpawner$StreamReader - collect2: fatal error: cannot find 'ld'
compilation terminated.
 
2016-05-06 17:08:32,159 INFO  [78] oracle.sysman.oii.oiix.OiixSpawner$StreamReader - genclntsh: Failed to link libclntsh.so.11.1
 
2016-05-06 17:08:32,159 INFO  [78] oracle.sysman.oii.oiix.OiixSpawner$StreamReader - make: *** [client_sharedlib] Error 1
 
2016-05-06 17:08:32,159 INFO  [64] oracle.sysman.oii.oiix.OiixSpawner - End output from spawned process.
2016-05-06 17:08:32,159 INFO  [64] oracle.sysman.oii.oiix.OiixSpawner - ----------------------------------
2016-05-06 17:08:32,160 INFO  [64] oracle.sysman.oii.oiis.OiisCompActions - Exception thrown from action: make
Exception Name: MakefileException
Exception String: Error in invoking target 'client_sharedlib' of makefile '/u01/app/oem/middelware/rdbms/lib/ins_rdbms.mk'. See '/u01/stage/em/disks/OraInstall2016-05-06_04-01-07PM/install2016-05-06_04-01-07PM.log' for details.
Exception Severity: 1
2016-05-06 17:08:32,160 INFO  [64] oracle.sysman.oii.oiic.OiicAPISessionInterfaceManager - Retry Handler not registered, using Super class functionality
2016-05-06 17:08:32,160 OFF   [64] oracle.sysman.oii.oiio.oiiol.OiiolTextLogger - Error in invoking target 'client_sharedlib' of makefile '/u01/app/oem/middelware/rdbms/lib/ins_rdbms.mk'. See '/u01/stage/em/disks/OraInstall2016-05-06_04-01-07PM/install2016-05-06_04-01-07PM.log' for details.
2016-05-06 17:08:32,160 INFO  [64] oracle.sysman.oii.oiis.OiisCompActions - Calling Action unixActions10.2.0.3.0  make  registerOnly = false    installMakePath = /usr/bin/make installMakeFileName = /u01/app/oem/middelware/sqlplus/lib/ins_sqlplus.mk        installTarget = install undoMakeFileName =      installArguments = ORACLE_HOME=/u01/app/oem/middelware  logFile = /u01/app/oem/middelware/install/make.log      undoTarget =    progMsg = Linking SQLPLUS Executables
2016-05-06 17:08:32,161 INFO  [64] oracle.sysman.oii.oiio.oiiol.OiiolTextLogger - The output of this make operation is also available at: '/u01/app/oem/middelware/install/make.log'
2016-05-06 17:08:32,162 INFO  [64] oracle.sysman.oii.oiix.OiixSpawner - Start output from spawned process:
2016-05-06 17:08:32,162 INFO  [64] oracle.sysman.oii.oiix.OiixSpawner - ----------------------------------
2016-05-06 17:08:32,165 INFO  [79] oracle.sysman.oii.oiix.OiixSpawner$StreamReader - mkdir -p /u01/app/oem/middelware/sqlplus/bin
16-05-06 17:08:32,166 INFO  [79] oracle.sysman.oii.oiix.OiixSpawner$StreamReader - cp -f /u01/app/oem/middelware/lib/libsqlplus.so \
        /u01/app/oem/middelware/sqlplus/lib/libsqlplus.so
 
2016-05-06 17:08:32,175 INFO  [79] oracle.sysman.oii.oiix.OiixSpawner$StreamReader - Linking /u01/app/oem/middelware/sqlplus/bin/sqlplus
 
2016-05-06 17:08:32,175 INFO  [79] oracle.sysman.oii.oiix.OiixSpawner$StreamReader - rm -f /u01/app/oem/middelware/sqlplus/bin/sqlplus
 
2016-05-06 17:08:32,176 INFO  [79] oracle.sysman.oii.oiix.OiixSpawner$StreamReader - gcc -o /u01/app/oem/middelware/sqlplus/bin/sqlplus -m64 -L/u01/app/oem/middelware/sqlplus/lib/ -L/u01/app/oem/middelware/lib/ -L/u01/app/oem/middelware/lib/stubs/  /u01/app/oem/middelware/sqlplus/lib/s0afimai.o -lsqlplus -lclntsh  `cat /u01/app/oem/middelware/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /u01/app/oem/middelware/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11 -lztkg11 -lclient11 -lnnetd11  -lvsn11 -lcommon11 -lgeneric11 -lmm -lsnls11 -lnls11
2016-05-06 17:08:32,176 INFO  [79] oracle.sysman.oii.oiix.OiixSpawner$StreamReader -  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 `cat /u01/app/oem/middelware/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /u01/app/oem/middelware/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lclient11 -lnnetd11  -lvsn11 -lcommon11 -lgeneric11   -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lclient11 -lnnetd11  -lvsn11 -l
2016-05-06 17:08:32,176 INFO  [79] oracle.sysman.oii.oiix.OiixSpawner$StreamReader - common11 -lgeneric11 -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11   `cat /u01/app/oem/middelware/lib/sysliblist` -Wl,-rpath,/u01/app/oem/middelware/lib -lm -lpthread   `cat /u01/app/oem/middelware/lib/sysliblist` -ldl -lm -lpthread  -L/u01/app/oem/middelware/lib
 
2016-05-06 17:08:32,186 INFO  [80] oracle.sysman.oii.oiix.OiixSpawner$StreamReader - collect2: fatal error: cannot find 'ld'
compilation terminated.

So basically ld and ar where not allowed to execute due to some acl .... once fixed as root with

setfacl -m u:oem:rx /usr/bin/ld.bfd
setfacl -m u:oem:rx /usr/bin/as
 

and I restarted the installation it went on smoothly. the sad point is that this error which happened way before configuring the BI publisher wasn't caught earlier...

kfod corrupt after patching

Short post that hopefully will save you time After patching to QFSDP 2016 jan it looked like dbca wasn't seeing ASM disks any more. Quickly @vanpupi found that it was related to kfod. apparently on a linux system it uses % instead of $ in $ORACLE_HOME/bin/kfod both in GI as in the DB home change the kfod file to following

OHOME=%ORACLE_HOME%

to 


OHOME=$ORACLE_HOME

did the trick good find pieter

using oplan

While preparing for the QFSDP of january 2016 we ran into an issue with oplan. We wanted to use oplan to generate the steps for patching since opatchauto couldn't be used in this case. but ran into this on a exadata OVM machine with plenty of rac databases

oracle.oplan.sdk.OPlanException: There is no RAC DB Instance running onexa01adm01vm01
 at oracle.oplan.db.cmdtranslator.commands.SqlPatchCommand.getOracleSIDForActiveInstance(SqlPatchCommand.java:272)
 at oracle.oplan.db.cmdtranslator.commands.SqlPatchCommand.getExecutionStep(SqlPatchCommand.java:104)
 at oracle.oplan.db.cmdtranslator.commands.SqlPatchCommand.generateExecutionSteps(SqlPatchCommand.java:64)
 at oracle.oplan.sdk.cmdtranslator.Command.getExecutionSteps(Command.java:101)
 at oracle.oplan.core.engine.SequencingEngine.expandRollingPhase(SequencingEngine.java:113)
 at oracle.oplan.core.engine.SequencingEngine.expandPhases(SequencingEngine.java:250)
 at oracle.oplan.core.engine.SequencingEngine.getExecutionPlan(SequencingEngine.java:271)
 at oracle.oplan.sdk.oplan.OPlan.process(OPlan.java:528)
 at oracle.oplan.sdk.oplan.OPlan.generateApplySteps(OPlan.java:705)
 at oracle.oplan.sdk.oplan.OPlan.main(OPlan.java:653)

after scratching our head we figured out that this was due to a database that was not running. since it wasn't needed anymore we removed it from the cluster and then everything went fine. The problem is that the error message is not clear at all. hope this helps when you encounter this .

ORA-00600 [ktfbns_update_ilmstat2] during impdp

I was asked to do an datapump import from a 11.2.0.4 BP16 to 12.1.0.2.7 from a couple of schema's. everything went fine but all of a sudden during the import following happened

Processing object type SCHEMA_EXPORT/SEQUENCE/SEQUENCE
Processing object type SCHEMA_EXPORT/TABLE/TABLE
ORA-39014: One or more workers have prematurely exited.
ORA-39029: worker 1 with process name "DW00" prematurely terminated
ORA-31671: Worker process DW00 had an unhandled exception.
ORA-00600: internal error code, arguments: [ktfbns_update_ilmstat2], [262161], [], [], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.DBMS_SQL", line 1185
ORA-06512: at "SYS.KUPW$WORKER", line 22791
ORA-06512: at "SYS.KUPW$WORKER", line 21996
ORA-06512: at "SYS.KUPW$WORKER", line 21553
ORA-06512: at "SYS.KUPW$WORKER", line 4516
ORA-06512: at "SYS.KUPW$WORKER", line 12063
ORA-06512: at "SYS.KUPW$WORKER", line 2081
ORA-06512: at line 2
 

ORA-0600 .... generally that is not a good sign, luckily the instance was still up. the ORA-0600 checker on metalink ORA-600/ORA-7445/ORA-700 Error Look-up Tool (Doc ID 153788.1) didn't know this argument. I opened an SR in parallel with all the possible info and I started to have a look at the dump files but that didn't help either. It occurred to me that ilm, Information Life Cycle Management or Heat Map is a new 12c feature. However this feature wasn't enabled on our Target.

heat_map                       OFF


So this was a surprise.


Still the ORA-0600 seemed to give an error while calculating these stats, I quickly read some docs about this feature.

select * from dba_ilmparameters;
 
NAME                                VALUE
------------------------------ ----------
ENABLED                                 1
RETENTION TIME                         30
JOB LIMIT                               2
EXECUTION MODE                          2
EXECUTION INTERVAL                     15
TBS PERCENT USED                       85
TBS PERCENT FREE                       25
POLICY TIME                             0

I used the procedure to disable

 
DBMS_ILM_ADMIN
 
 
sys@WDWSI> begin
  2  dbms_ilm_admin.DISABLE_ILM;
  3  end;
  4  /
 
PL/SQL procedure successfully completed.
 
sys@WDWSI> select * from dba_ilmparameters;
 
NAME                                VALUE
------------------------------ ----------
ENABLED                                 2
RETENTION TIME                         30
JOB LIMIT                               2
EXECUTION MODE                          2
EXECUTION INTERVAL                     15
TBS PERCENT USED                       85
TBS PERCENT FREE                       25
POLICY TIME                             0
 
8 rows selected.

Retried the import => NOK I checked which undocumented parameters where set with regards to ILM in the description

Parameter                     Instance  Description
----------------------------- --------- -------------------------------------------
_ILM_FILTER_TIME              0         Upper filter time for ILM block compression
_ILM_FILTER_TIME_LOWER        0         Lower filter time for ILM block compression
_ILM_POLICY_NAME              FALSE     User specified ILM policy name
_create_stat_segment          0         create ilm statistics segment
_disable_12cbigfile           FALSE     DIsable Storing ILM Statistics in 12cBigFiles
_drop_stat_segment            0         drop ilm statistics segment
_enable_ilm_flush_stats       TRUE      Enable ILM Stats Flush
_enable_ilm_testflush_stats   FALSE     Enable Test ILM Stats Flush
_flush_ilm_stats              0         flush ilm stats
_ilmflush_stat_limit          0         ILM flush statistics limit - Internal testing only
_ilmset_stat_limit            0         ILM set statistics limit - Internal testing only
_ilmstat_memlimit             10        Percentage of shared pool for use by ILM Statistics
_ktilmsc_exp                  600       expiration time of ktilm segment cache (in second)
_print_inmem_heatmap          0         print inmem ilm heatmap
_print_stat_segment           0         print ilm statistics segment
_trace_ktfs                   FALSE     Trace ILM Stats Tracking
heat_map                      OFF       ILM Heatmap Tracking

It occurred to me that BIGFILE TABLESPACES were used so because of the urgency of the IMPORT i decided to try to set the parameter _disable_12cbigfile to TRUE, since I didn't get any update from support and the customer was waiting to do his tests .... And Bingo the import Worked as expected.... Big disclaimer : putting _ parameters is not something you do without the blessing of support !!! I provided support with my workaround and they are checking for further potential issues and impacts

evmd not starting in oracle restart

Monday two weeks ago I patched the DEV Super Cluster System to GI 12.1.0.2 BP10 together with DB 11.2.0.4 BP17. To win time I created a seperate oracle database home and upgraded that one to BP17. Then it would be a matter to stop the db's change their homes and do catbundle exa apply.... So far so good... Then came the time to upgrade GI everything went pretty smooth.

opatchauto -oh $GI_HOME -ocmrf /export/home/grid/ocm.rsp ....

and there we go ... however the last last step post patch took a long long time. looking at the traces evm didn’t want to start I remembered that last time i had the same issue and cleaning up /var/tmp/.oracle solved the issue …. so i interrupted this step,disabled automatic has start, cleaned up /var/tmp/.oracle and rebooted the zone. ok all perfect however relaunching the step still didn’t help, evmd still didn’t want to start

grid 3493 566 0 15:18:41 pts/17 0:00 grep d.bin 
grid 1380 27879 0 15:11:57 ? 0:07 /u01/app/grid/product/12.1.0/grid/bin/ohasd.bin reboot 
grid 1708 27879 0 15:12:10 ? 0:07 /u01/app/grid/product/12.1.0/grid/bin/oraagent.bin 
root 1355 1161 0 15:11:57 pts/19 0:02 /u01/app/grid/product/12.1.0/grid/bin/crsctl.bin start has 

I also saw plenty of errors regarding evmd in oohs_oraagent_grid.trc

"2015-09-01 15:19:59.298545 :GIPCXCPT:13:  gipcInternalConnectSync: failed sync request, ret gipcretConnectionRefused (29)

2015-09-01 15:19:59.298700 :GIPCXCPT:13:  gipcConnectSyncF [EvmConConnect : evmgipcio.c : 205]: EXCEPTION[ ret gipcretConnectionRefused (29) ]  failed sync connect endp 102d76690 [00000000000050c4] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=)(GIPCID=00000000-00000000-0))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=SYSTEM.evm.acceptor.auth)(GIPCID=00000000-00000000-0))', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef 0, ready 0, wobj 1030d1d40, sendp 102d76190 status 13flags 0xa008871a, flags-2 0x1, usrFlags 0x30020 }, addr 1033b1290 [00000000000050cb] { gipcAddress : name 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=SYSTEM.evm.acceptor.auth)(GIPCID=00000000-00000000-0))', objFlags 0x0, addrFlags 0x4 }, flags 0x8000000

2015-09-01 15:19:59.299375 : CLSCEVT:13: (:CLSCE0017:)clsce_subscribe 10212ee50 EvmConnCreate failed with status = 13

2015-09-01 15:19:59.299798 :  CRSEVT:13: {0:0:2} ClusterPubSub::subscribe clsce_subscribe failed [4]

2015-09-01 15:19:59.299917 : USRTHRD:13: {0:0:2} LsnrAgentSub-LISTENER_CLONE ClusterReconnectingSubscriber::subscribe Exception ClusterConnectException : CRS-10203: (:CLSCE0017:)  Could not connect to the Event Manager daemon

2015-09-01 15:19:59.300001 : CLSCEVT:13: (:CLSCE0028:)clsce_unsubscribe 10212ee50 successfully unsubscribed : 0

2015-09-01 15:20:00.301266 : CLSCEVT:13: clsce_subscribe 10226cad0 filter='^CRS_RESOURCE_PROFILE_CHANGE.*NAME='ora\.(scan|ssc02dbdat05z01\.vip).*RESOURCE_CLASS='(scan_vip|vip)'', flags=1, handler=100b26978, arg=102f928e0

2015-09-01 15:20:00.303161 :GIPCXCPT:13:  gipcInternalConnectSync: failed sync request, ret gipcretConnectionRefused (29)

2015-09-01 15:20:00.303300 :GIPCXCPT:13:  gipcConnectSyncF [EvmConConnect : evmgipcio.c : 205]: EXCEPTION[ ret gipcretConnectionRefused (29) ]  failed sync connect endp 102d76690 [00000000000050d5] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=)(GIPCID=00000000-00000000-0))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=SYSTEM.evm.acceptor.auth)(GIPCID=00000000-00000000-0))', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef 0, ready 0, wobj 1030d1d40, sendp 102d76190 status 13flags 0xa008871a, flags-2 0x1, usrFlags 0x30020 }, addr 1033b0990 [00000000000050dc] { gipcAddress : name 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=SYSTEM.evm.acceptor.auth)(GIPCID=00000000-00000000-0))', objFlags 0x0, addrFlags 0x4 }, flags 0x8000000

2015-09-01 15:20:00.303791 : CLSCEVT:13: (:CLSCE0017:)clsce_subscribe 10226cad0 EvmConnCreate failed with status = 13

2015-09-01 15:20:00.304118 :  CRSEVT:13: {0:0:2} ClusterPubSub::subscribe clsce_subscribe failed [4]

2015-09-01 15:20:00.304204 : USRTHRD:13: {0:0:2} LsnrAgentSub-LISTENER ClusterReconnectingSubscriber::subscribe Exception ClusterConnectException : CRS-10203: (:CLSCE0017:)  Could not connect to the Event Manager daemon

2015-09-01 15:20:00.304257 : CLSCEVT:13: (:CLSCE0028:)clsce_unsubscribe 10226cad0 successfully unsubscribed : 0

2015-09-01 15:20:00.304304 : CLSCEVT:13: clsce_subscribe 10212ee50 filter='^CRS_RESOURCE_PROFILE_CHANGE.*NAME='ora\.(scan|ssc02dbdat05z01\.vip).*RESOURCE_CLASS='(scan_vip|vip)'', flags=1, handler=100b26978, arg=1033406a0

2015-09-01 15:20:00.305574 :GIPCXCPT:13:  gipcInternalConnectSync: failed sync request, ret gipcretConnectionRefused (29)

2015-09-01 15:20:00.305675 :GIPCXCPT:13:  gipcConnectSyncF [EvmConConnect : evmgipcio.c : 205]: EXCEPTION[ ret gipcretConnectionRefused (29) ]  failed sync connect endp 102d76690 [00000000000050e6] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=)(GIPCID=00000000-00000000-0))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=SYSTEM.evm.acceptor.auth)(GIPCID=00000000-00000000-0))', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef 0, ready 0, wobj 1030d1d40, sendp 102d76190 status 13flags 0xa008871a, flags-2 0x1, usrFlags 0x30020 }, addr 1033b1290 [00000000000050ed] { gipcAddress : name 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=SYSTEM.evm.acceptor.auth)(GIPCID=00000000-00000000-0))', objFlags 0x0, addrFlags 0x4 }, flags 0x8000000

2015-09-01 15:20:00.306108 : CLSCEVT:13: (:CLSCE0017:)clsce_subscribe 10212ee50 EvmConnCreate failed with status = 13

2015-09-01 15:20:00.306376 :  CRSEVT:13: {0:0:2} ClusterPubSub::subscribe clsce_subscribe failed [4]

2015-09-01 15:20:00.306470 : USRTHRD:13: {0:0:2} LsnrAgentSub-LISTENER_CLONE ClusterReconnectingSubscriber::subscribe Exception ClusterConnectException : CRS-10203: (:CLSCE0017:)  Could not connect to the Event Manager daemon

2015-09-01 15:20:00.306752 : CLSCEVT:13: (:CLSCE0028:)clsce_unsubscribe 10212ee50 successfully unsubscribed : 0

2015-09-01 15:20:01.308000 : CLSCEVT:13: clsce_subscribe 10226cad0 filter='^CRS_RESOURCE_PROFILE_CHANGE.*NAME='ora\.(scan|ssc02dbdat05z01\.vip).*RESOURCE_CLASS='(scan_vip|vip)'', flags=1, handler=100b26978, arg=102f928e0

2015-09-01 15:20:01.309869 :GIPCXCPT:13:  gipcInternalConnectSync: failed sync request, ret gipcretConnectionRefused (29)

2015-09-01 15:20:01.309994 :GIPCXCPT:13:  gipcConnectSyncF [EvmConConnect : evmgipcio.c : 205]: EXCEPTION[ ret gipcretConnectionRefused (29) ]  failed sync connect endp 102d76690 [00000000000050f7] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=)(GIPCID=00000000-00000000-0))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=SYSTEM.evm.acceptor.auth)(GIPCID=00000000-00000000-0))', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef 0, ready 0, wobj 1030d1d40, sendp 102d76190 status 13flags 0xa008871a, flags-2 0x1, usrFlags 0x30020 }, addr 1033b0990 [00000000000050fe] { gipcAddress : name 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=SYSTEM.evm.acceptor.auth)(GIPCID=00000000-00000000-0))', objFlags 0x0, addrFlags 0x4 }, flags 0x8000000

2015-09-01 15:20:01.310445 : CLSCEVT:13: (:CLSCE0017:)clsce_subscribe 10226cad0 EvmConnCreate failed with status = 13

2015-09-01 15:20:01.310779 :  CRSEVT:13: {0:0:2} ClusterPubSub::subscribe clsce_subscribe failed [4]”

I opened an sr and after killing ohasd and oraagent.bin process the GI and evmd came up. Later after transferring the SR to my timezone support came back with a suspicion of a couple of bugs.

“

1. Unpublished  BUG 21484367 12.1.0.2 SIHA UPGRADE HANG INDEFINITELY IF MORE SERVICES REGISTERED 
2.  BUG 20620033 AIX ISSUES WITH GI 12.1.0.2 UPGRADE FINE, IF DON'T CONFIGURE > 34 OR 35 SERVICES 
-> not AIX specific 

“

on another system I could reproduce the issue the problem indeed appeared in my case when 35 services where created in total on the machine that is services you add with

“
srvctl add service 

then the GI didn't come up. one way circumvent is to put the services in MANUAL but that is not really a solution for us >80 services or put the db in MANUAL. Dev is working on a patch currently. Hope this help when you get these errors.

E4 2015 it is a wrap

Long overdue but I had a very busy month of june and july sofar.

 This year I was invited at E4 as a speaker. This was my second time in the USA and
 I really enjoyed every bit of it, a great conference , great people nice interactions unsurpassed hospitality. Andy Colvin and his wife Kate came to pick me up at the airport, the belt where I had to get my luggage was very appropriate.

Once the luggage was there it was the turn of Martin Bach to get picked up at the airport he landed at about the same time at another terminal.
Andy together with the amazing Veronica arranged a great Mexican Restaurant for us that night it was delicious and we finished the evening in a great beer pub in Southlake .


The great thing about E4 is that it is completely focussed on engineered systems and how they work, the other great thing is that there is one track which makes you attend plenty of interesting session you might miss otherwise.

 I really enjoyed it and learned a lots of engineered systems I didn't even know before such as the VCA, Virtual Compute Appliance, James Anthony really nailed this presentation it was very interesting and well presented.


 Other highlights for me were :

•  Tanel Poder & Kerry Osborne's keynote about the hybrid world, Conventional RDBMS and the NOSQL movement will continue to coexist together and are complimentary 

• Tyler Muth about Super Cluster and in-memory very interesting since I didn't have not a lot of exposure of ORACLE 12c so far 

•  Roger MacNicol What’s new in Exadata Smart Scan 12.1.0.1 and 12.1.0.2 which was the version we just upgraded to, so this was a great refresher

•  Sue Lee "Tips and Tricks for Successful Consolidation on Exadata" I had a lot of eye openers there and got really good info about resource manager and IORM. Some interesting new things in Storage Cell version 12.1.2.1 such as Flash Cache Min and Flash Cache Limit are available now. 

•  Martin Bach's talk about HCC Compression, I saw this one already a couple of times and it is really a presentation like wine, it gets better and and better, I you are able to attend this talk at some other conference please do so it is really very world while and Martin knows his stuff and has very good presentation skills.

•  Frits Hoogland, in Frits' own style he did a superb presentation about memory SGA the differences in allocation between AMM and ASMM and manual, PGA how it allocated, etc very interesting but have to go through the presentation again to grasp everything 

• Kodi's Umamageswaran session he pointed out what is in store on the technology level: Networking 40Gbps Ethernet, 100Gbps infiniband Persistent memory etc. 

•  Mauro Pagano's session about health checks was really incredible what a horror story very entertaining, if ever a book should be written about worst practices this use case is a good candidate.

 My talk was about the Sparc Super Cluster "Super Cluster: the SWISS army knife of the Engineered Systems", I had the first slot on Tuesday and had about 100 people in my session,

I was bit nervous since a lot of my hero's were in the room and also the product manager of Super Cluster Michael Palmeter and Tyler Muth Super Cluster affiando.

The talk went ok and I did get good feedback and enjoyed it a lot.

 The conference is one thing but being able to talk offline with people like Karl Arao, Tanel Põder, Kerry Osborne, Mauro Pagana, Cary Millsap, Carlos Sierra, Andy Colvin, Tyler Muth, Michael Palmeter, Frits Hoogland, Martin Bach, Tarmo Alasoo, Radu Parvu, Jason Arneil, Hank   and all the passionate attendees just feels great and that is what attending this conference

I really enjoyed this conference a lot and got to know some nice people, thank you Veronica , Chanea and all the others of the organisation, you really made us speakers feel like kings.

A big thank you to Andy and Kate,  thanks for you warm welcome and even warmer hospitality really appreciated.

Next year there will be a European E4 as well in April, just a couple hours of flight for most of the Europeans, so you have no excuse to not attend this great engineered systems conference. check enkitecs website for more info


and yes it is true everything is bigger in Texas :-)