Wednesday, May 15, 2019

patching Exadata from 18.1.12.0.0.190111 version to 19.2.1.0.0.190419

As you know the 19 release of Exadata is a big one, it upgrades the linux distribution from Oracle Enterprise Linux 6 to 7. We have an OVM Based Exadata and in the progress of testing it. I will write some more this week about the process to upgrade to 19c so far we upgraded : * our cells * dom0 and are now in the process of upgrading the domU's however today we ran into a funny issue, for which we are waiting for the solution, 18.1.12 comes with linux release cat /etc/redhat-release on the upgraded system shows us
Red Hat Enterprise Linux Server release 6.10 (Santiago) 
However the pre upgrade check tool during the upgrade complains on this Make sure you Read Andy Colvin's blog about this release you might need to redownload the patch as it was re-released More to come about this patching later UPDATE : in the meantime yesterday 15 may 2019 a new QFSDP was released

Thursday, May 2, 2019

moving Exadata vm to a new node

The customer I am working for has OVM on Exadata for the dev qualification and testing Exadata we recently added two compute nodes. The main driver was RAM, it was less ( or equally priced) expensive to add two new nodes then extending the RAM ( removing existing DIMM and change them with higher capacity ones)


 We now have : 4 X6 nodes and two brand new X7 nodes (we just got them when X8 was released :-( ) there are quite some database running here and we since they are using the same cells no need to duplicate them.

 My colleague Freek D'hooge pointed me to this document : Moving a User Domain to a Different Database Server


 Ok that procedure worked as a charm, at first sight, only infiniband in the vm didn't come up. lspci didn't show us the IB card. luckily there was one vm during the installation of the extra nodes on this vm and we could start comparing the configuration : in the vm.cfg of the vm we "copied" from the X6" we saw following


ib_pfs = ['03:00.0']
ib_pkeys = [{'pf':'03:00.0','port':'1','pkey':['0xffff',]},{'pf':'03:00.0','port':'2','pkey':['0xffff',]},]


on the vm that was working on the X7
ib_pfs = ['3b:00.0']
ib_pkeys = [{'pf':'3b:00.0','port':'1','pkey':['0xffff',]},{'pf':'3b:00.0','port':'2','pkey':['0xffff',]},]


once we put this in the copied vm everything booted afterwards we understood why, my colleague Fred pointed out on the source dom0 (X6)


lspci |grep -i infiniband
03:00.0 InfiniBand: Mellanox Technologies MT27500 Family [ConnectX-3]
03:00.1 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
03:00.2 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
03:00.3 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
03:00.4 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
03:00.5 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
03:00.6 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
03:00.7 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
03:01.0 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

....

on target dom0 (X7)


lspci | grep -i 'infiniband'
3b:00.0 InfiniBand: Mellanox Technologies MT27500 Family [ConnectX-3]
3b:00.1 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
3b:00.2 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
3b:00.3 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
3b:00.4 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
3b:00.5 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
3b:00.6 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
3b:00.7 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
3b:01.0 InfiniBand: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]



on an X4 it is even differnt


so basically I think this should be amended to the otherwise flawless document mentioned before

Hope this helps