Oracle Corner

Cloning a database where source Oracle Home has a higher DST patch than a destination Oracle home 

Symptoms

Solution

Labels: ORA-01804 RMAN-06429 timezone DST

OEM13c occasionally leaves orphaned alerts or targets in suspended mode, that clearstate can't remove.  After some investigation, I've created and used these commands for few years, and they are working well.  

Get targets out of suspended mode

select * from sysman.em_current_availability  where current_status in (4,5,6) and SEVERITY_GUID is not null;
update sysman.em_current_availability set CURRENT_STATUS=1,SEVERITY_GUID=null  where current_status in (4,5,6) and SEVERITY_GUID is not null;

Clear up orphaned alerts

col msg for a50
select TARGET_GUID, EVENT_SEQ_ID, msg, REPORTED_DATE,CLOSED_DATE  from sysman.GC$EVENTS where event_seq_id in (select event_seq_id from sysman.EM_EVENT_SEQUENCES_E where open_status=1 and msg like '%&msg_text%'); update sysman.EM_EVENT_SEQUENCES_E set open_status=0, closed_date=sysdate, severity=0, prev_severity=16 where (TARGET_GUID = '&target_id' or  EVENT_SEQ_ID='&event_seq_id' ) and open_status=1; 
--or
update sysman.EM_EVENT_SEQUENCES_E set open_status=0, closed_date=sysdate, severity=0, prev_severity=16 where open_status=1 and event_seq_id in (select EVENT_SEQ_ID from sysman.GC$EVENTS where event_seq_id in  (select event_seq_id from sysman.EM_EVENT_SEQUENCES_E where open_status=1 and msg like '%&msg_text%'));

Labels: OME 13c Suspended target repository update

Clusterize the database

oh; cd dbs
cp initDBNAME.ora initDBNAME.ora.singleinstance

create pfile='/oracle/DBNAME/121/dbs/initDBNAME.ora' from spfile;

vi initDBNAME.ora
Add:
*.cluster_database = TRUE
*.remote_listener='scanename:xxx'
DBNAME1.instance_number=1
DBNAME2.instance_number=2
DBNAME1.thread=1
DBNAME2.thread=2
DBNAME1.undo_tablespace='UNDO1'
DBNAME2.undo_tablespace='UNDO2'
If you see andy DBNAME.__ parameters, change them to DBNAME1 and DBNAME2. Example:
DBNAME1__db_cache_size=1895825408
DBNAME2__db_cache_size=1895825408
Important: Remove *.undo_tablespace parameter

create undo tablespace PUNDO1 datafile '+ORADATA3' SIZE 2G AUTOEXTEND ON MAXSIZE 5G;
create undo tablespace PUNDO2 datafile '+ORADATA3' SIZE 2G AUTOEXTEND ON MAXSIZE 5G;

Add log files for thread 2

List log files from v$logfile;

#Add as many groups as you have in thread1, with the same size.
Example:
alter database add logfile thread 2 group 31 size 200M;
alter database add logfile thread 2 group 32 size 200M;
alter database add logfile thread 2 group 33 size 200M;
...
alter database enable public thread 2;

oh; cd dbs
cp initDBNAME.ora initDBNAME1.ora
cp initDBNAME.ora initDBNAME2.ora
cp orapwDBNAME orapwDBNAME1
cp orapwDBNAME orapwDBNAME2
vi ../network/admin/tnsnames.ora
Change DBNAME now to be:
DBNAME=
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = scanname)(PORT = xxxx))
    (CONNECT_DATA =  (SERVER = DEDICATED)  (SERVICE_NAME = DBNAME)  ) )

Start the new cluster database

shutdown immediate
#Start the both instance individually
export ORACLE_SID=DBNAME2
startup nomount pfile='/oracle/DBNAME/121/dbs/initDBNAME.ora'
create spfile='+ORADATA3/DBNAME/spfileDBNAME.ora' from pfile='/oracle/DBNAME/121/dbs/initDBNAME.ora';
shutdown immediate
startup pfile='/oracle/DBNAME/121/dbs/initDBNAME2.ora'

ssh to other node
export ORACLE_SID=DBNAME1
startup pfile='/oracle/DBNAME/121/dbs/initDBNAME1.ora'
!! shutdown immediate on BOTH nodes !!

srvctl add database -d DBNAME -o /oracle/DBNAME/121
srvctl add instance -d DBNAME -i DBNAME1 -n node1
srvctl add instance -d DBNAME -i DBNAME2 -n node2
srvctl modify database -d DBNAME -p '+ORADATA3/DBNAME/spfileDBNAME.ora'
srvctl config database -d DBNAME

srvctl start instance -i DBNAME2 -d DBNAME
srvctl start instance -i DBNAME1 -d DBNAME
srvctl stop database -d DBNAME
srvctl start database -d DBNAME

Voila!

Prepare OLD database

cd $ORACLE_HOME/dbs; cp orapwDBNAME orapwDBNAME.beforemigration
orapwd file=/oracle/DBNAME/12102/dbs/orapwDBNAME sysdg=yes sysbackup=yes force=y
Stop all cronjobs on OLD server

Prepare  NEW database

Prepare all directories
oh; cd dbs
orapwd file=/oracle/DBNAME/121/dbs/orapwDBNAME sysdg=yes sysbackup=yes force=y

Update oratab

vi /etc/oratab
Add line:
DBNAME:/oracle/DBNAME/121:N
. oraenv

Create a separate listener for duplicate process

We are doing this so we don't have to touch SCAN or VIP listeners and disturb existing cluster
oh; cd network/admin
lsnrctl stop listener_duplicate

vi listener.ora
Add this:
LISTENER_DUPLICATE=
  (ADDRESS_LIST =
    (ADDRESS = (PROTOCOL = TCP)(HOST = servername)(PORT = xxxx)) )

ADR_BASE_LISTENER_DUPLICATE = /oracle/DBNAME/saptrace
CONNECT_TIMEOUT_LISTENER_DUPLICATE = 10
TRACE_LEVEL_LISTENER_DUPLICATE = OFF

SID_LIST_LISTENER_DUPLICATE =
  (SID_LIST =
    (SID_DESC =
      (GLOBAL_DBNAME = DBNAME)
      (ORACLE_HOME = /oracle/DBNAME/121)
      (SID_NAME = DBNAME)  ) )

lsnrctl start listener_duplicate

Set up init.ora

Ensure these parameters are changed:
*.compatible=12.1.0.2.0
*.control_files
*.db_create_file_dest
*.db_recovery_file_dest
*.db_recovery_file_dest_size
*.db_create_online_log_dest_1
*.db_create_online_log_dest_2
*.dg_broker_start=FALSE
*.db_unique_name=DBNAME
*.log_archive_dest_1
*.log_archive_dest_state_1='ENABLE'

Set up TNS connectivity between servers

Add to tnsnames.ora on NEW server:

DBNAME_OLD=
  (DESCRIPTION =
    (ADDRESS_LIST =(ADDRESS = (PROTOCOL = TCP)(HOST = oldhost)(PORT = xxxx) ))
    (CONNECT_DATA = (SID = DBNAME) (GLOBAL_NAME = DBNAME.WORLD)))

DBNAME=
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST=servername)(PORT = xxxx))
    (CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = DBNAME)(UR = A) )
  )

LISTENER_DUPLICATE =(DESCRIPTION =(PROTOCOL = TCP)(HOST = servername)(PORT = xxxx))

Check all with:
sqlplus sys@DBNAME_OLD as sysdba
sqlplus sys@DBNAME as sysdba
tnsping listener_duplicate

Startup nomount NEW database

startup nomount pfile='/oracle/DBNAME/121/dbs/initDBNAME.ora';
create spfile from pfile='/oracle/DBNAME/121/dbs/initDBNAME.ora';

Startup mount OLD database

We are doing this so we don't have to do recovery on the NEW db
STARTUP MOUNT;

Duplicate

#No need to allocate channels as we are going directly over the network
rman AUXILIARY sys@DBNAME target sys@DBNAME_OLD
RUN
{
DUPLICATE TARGET DATABASE TO DBNAME FROM ACTIVE DATABASE;
}

Check the health of the database before commencing.

If standby is considerably lagging behind production, “roll it forward” using the incremental backup.

DR

rman target /
catalog start with '/backup/DBNAME_inc';
recover database noredo;

DGMGRL> edit database 'DBNAMEDR_A' set state='APPLY-ON';

This is a simple scenario, which assumes no new datafile have been created on the primary since the given SCN.

OMSCA Recover failed

Labels: "weblogic.nodemanager.NMConnectException: Connection refused. Could not connect to NodeManager."

OEM target stuck in "Notification Blackout"

This is OMS 13c bug.

As there is no emctl or emcli command that can fix this problem, update the brownout status in EM repository.

update sysman.MGMT_CURRENT_AVAILABILITY
set brownout_status=0
where target_guid in ('107F8CCD2C86105AE0531D5A710A8A7F','4D4FA30DBA388074E6821EDE4DB4B031');

Symptoms

Solution

Lesson learned

Labels: ORA-15137 force patch rollback

Cause

Workaround

From

To

This is a pretty simple problem, and there is an Oracle bug explained in Doc ID 2145349.1, but Oracle support doesn't offer solution, so I thought I should connect those two.

Symptoms

Mon Jul 17 21:50:44 2017ERROR: Unable to get logical block size for spfile '+ORADATA3/PPEN/spfilePPEN.ora'.
Mon Jul 17 21:50:45 2017ERROR: Unable to get logical block size for spfile '+ORADATA1/PPEN/spfilePPEN.ora'.
Cause


The problem is caused by init*.ora file pointing to a non-existing spfile, while spfile instance parameter is pointing to the correct one.

Solution

Create an alias in ASM that matches the name in init*.ora file.

ASMCMD> pwd+oradata1/PPEN
ASMCMD> ls PARAMETERFILE/spfile.268.948615175
ASMCMD> mkalias PARAMETERFILE/spfile.268.948615175 spfilePPEN.ora

Labels: "ERROR: Unable to get logical block size for spfile" solution

When you use OEM to create a standby on RAC, it will create a single instance on one standby node, which you later need to convert to a clustered database.

In my example that instance is called PPEN (playpen).

I converted PPEN to a clustered database, and I could see 2 instances: PPEN1 and PPEN2. Previous "PPEN" instance has been automatically removed from the cluster node, and I removed it manually from OEM. The problem is that it wasn't removed within Data Guard configuration.

Symptoms

When you go to OEM, and chose 'Availability'/'Verify Database Data Guard Configuration', you will get warnings:

Initializing
Connected to instance host2:PPEN2
Starting alert log monitor...
Updating Data Guard link on database homepage...
WARNING: Cluster database instance PPEN is not discovered.
WARNING: Switchover or failover may not succeed as a result.
Skipping verification of fast-start failover static services check.

Data Protection Settings:
  Protection mode : Maximum Performance
  Redo Transport Mode settings:
    PPEN: ASYNC
    PPENDR_A: ASYNC
... etc...

Solution

Fix is to delete the initial PPEN instance from Data Guard configuration.

DGMGRL>  show database verbose "PPENDR_A";
Database - PPENDR_A

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-ON
  Transport Lag:      0 seconds (computed 0 seconds ago)
  Apply Lag:          0 seconds (computed 0 seconds ago)
  Average Apply Rate: 4.00 KByte/s
  Active Apply Rate:  781.00 KByte/s
  Maximum Apply Rate: 1.01 MByte/s
  Real Time Query:    OFF
  Instance(s):
    PPEN  -- this instance shouldn't be here
    PPEN1 (apply instance)
    PPEN2

DGMGRL> remove instance "PPEN" on database "PPENDR_A";

This looks like a bug, but search through Oracle support didn't show it, and google-ing didn't help me find this warning, so hopefully this helps other DBAs.

Labels: Verify Database Data Guard Configuration, WARNING: Cluster database instance is not discovered., WARNING: Switchover or failover may not succeed as a result.

Note
After upgrading to 12.1.0.2 on RAC cluster, we had a very slow SQL below that was reading column group statistics:


     select count(*) from sys.col_group_usage$ where obj# = :1 and cols = :2
     and trunc(sysdate) = trunc(timestamp) and bitand(flags, :3) = :3"  (Doc ID 2013851.1 ).


According to Oracle support site, this was a known bug and you could work around this by deleting extended statistics, or by setting cursor_sharing to FORCE. After experimenting with both options, we switched the cursor_sharing to 'FORCE'. That caused another bug. A SQL below started running in 2.5 seconds per execution, and was waiting on "GV$: slave acquisition retry wait time". 

     select inst_id, instance_name, host_name from gv$instance
     where instance_mode in (:"SYS_B_0",:"SYS_B_1") order by inst_id 


Solution

Oracle had a suggestion on how to work around this, but that didn't apply to us, as we already had parallel_execution_message_size=16K (Doc ID 1589592.1)

Oracle released a new merged patch for us: 
Bug 24434288 - MERGE REQUEST ON TOP OF DATABASE PSU 12.1.0.2.160719 FOR BUGS 19543384 20907061.

Create wallet

Create requirement file

Add trusted certificates

Secure OMS with third-party certificates 

Re-secure all agents

BEGIN 
DBMS_AUTO_SQLTUNE.set_auto_tuning_task_parameter( 
parameter => '_sqltune_control', 
value => 15); 
END; 
/ 

Symptoms

SQL Tuning Advisor recommends lots of plans with PX hints, claiming that parallelism puts the cost down. That might be a case, but in the case you don't want to use parallelism, there is a way to remove those recommendations.

Solution

BEGIN
DBMS_AUTO_SQLTUNE.set_auto_tuning_task_parameter(
parameter => '_sqltune_control',
value => 1+2+4+8+32);
END;
/  

To switch it back:
value => 1+2+4+8+10+32);

The upgrade had quite a few bugs/problems, for which we couldn't find solutions either on Oracle support site or Google, therefore listing them here:

Error

Error during "Repository upgrade" with ORA-00955 with file severity_indexes.sql failing when command "create index em_violations_idx_08 ..." was run.

Workaround

drop index sysman.em_violations_idx_08;

Error

java.sql.SQLSyntaxErrorException "The host on which you are performing the Two system upgrade does not match with the host name you entered in the Enterprise Manager 12c Upgrade Console"

Workaround

Although Oracle support site said to check PRE_UPGC_MASTER_INFO table, the hostname in that table was correct. Problem was that we had lots of bugs previously in the installation, so somehow the whole installation got confused. We stopped that runInstaller, ran another, and the error didn't show up again.

Error

ORA-00600: internal error code, arguments: [qcisSetPlsqlCtx:tzi init], [], [], [], [], [], [], [], [], [], [], []

Workaround

Increase parameters to at least 3G SGA, 1G PGA, 250 session cashed cursors.

Error

Empty temporary tablespace: TEMP

Workaround

Manually add TEMP tablespace (we had to add TEMP1, switch to that one, drop TEMP, create TEMP, switch back)
Error

ORA-00942 SCHEMA_VERSION_REGISTRY is missing. One of the log files are showing that OBG$MIG was missing, but that was a red herring.
Workaround

create or replace force view "SYSTEM"."SCHEMA_VERSION_REGISTRY"
(
  "COMP_ID",
  "COMP_NAME",
  "MRC_NAME",
  "MR_NAME",
  "MR_TYPE",
  "OWNER",
  "VERSION",
  "STATUS",
  "UPGRADED",
  "START_TIME",
  "MODIFIED"
)
as
    selectcomp_id,
           comp_name,
           mrc_name,
           mr_name,
           mr_type,
           owner,
           version,
           status,
           upgraded,
           start_time,
           modified
      from SYSTEM.SCHEMA_VERSION_REGISTRY$
  order by comp_id;

Error

MDS Schema Creation status: FAILURE 
MDS Schema Creation is failed
Failed to create OPSS schema

$more emmdscreate.err
ERROR:MDS: ORA-00955: name is already used by an existing object
File:/data1/opt/oracle/middleware/oms/rcu/integration/mds/sql/cremdcmtbs.sql
Statement:create table MDS_PARTITIONS (
 PARTITION_ID                  NUMBER NOT NULL,
 PARTITION_NAME                VARCHAR2(200) NOT NULL,
 PARTITION_LAST_PURGE_TIME     TIMESTAMP,
 PARTITION_MT_STATE            NUMBER
 )
Creation of MDS schema fails

Workaround

Press "Retry". This solution was in Oracle Support.
Error

ERROR: Invalid username and/or password
LOG: Local Authentication Failed...Attempt PAM authentication...PAM failed with error: No account present for user
Workaround

After you click "Deploy and Configure Agents", chose "Override Oracle Home Preferred Credentials" instead of preferred. Probably due to new OMS 12c not having default preferred credentials.

I couldn't find this error anywhere on Google, and IBM Support has been looking into it for 2 days without help. I found the workaround, it would be good to share it with community.

The errors showed up during the upgrade of TSM TDP for SAP for Oracle v5.5 to v6.3. We restored v5.5 back, deinstalled, installed, and at the end finished the errors below.

Error

When running tdpoconf or backup:

ld.so.1: tdpoconf: fatal: relocation error: file /opt/tivoli/tsm/client/oracle/bin64/tdpoconf: symbol tdpoQueryVersion: referenced symbol not found
Killed

RMAN-03009: failure of allocate command on sbt_1 channel at 10/18/2014 18:15:55
ORA-19554: error allocating device, device type: SBT_TAPE, device name:
ORA-27211: Failed to load Media Management Library

BKI9009W: The following products are not compatible: ESD_DataMover (5.5.0.0) and ESD_Prole (6.3.0.0).

I couldn't find this anywhere else on the Internet, therefore worth posting.

Default value of  _sys_logon_delay 12c parameter is 1.

Value of zero will switch off this feature, so this proves it is enabled by default. One Oracle blog said the higher the values, the bigger the delay is.


NAME      Type  Value Is Def ISSES ISSYS_MOD DESCRIPTION
---------------- ----- ------ ----- ---------------------
_sys_logon_delay  3 1 TRUE   FALSE FALSE failed logon delay for sys

2 products exist for managing: Cc + Ops Center (host management)

Oracle HTTP server is started up implicitly when OMS is started. Managed by opmnctl process (if we need explicitly to run something).

Discovery
=======
Agent-based (self explanatory) or Agent-less (scans all IP n/w addresses. Then you can review and promote what you want)

Backup
=====
Backup s/w home. That will cover WLS admin server, the whole web logic, OMs and webtier configuration files.
Backup the instance home: emctl exportconfig oms command.

Monitor
=====
What happens during the upgrade to user-defined metrics? They are called metric extensions now.
Linux and Solaris hosts can be monitored but YaST needs to be installed.
Targets can be grouped in a "group". Notification can be done 'per group'.
Part of service Level Mgmt pack:
  -- Define "SYSTEM" for bunching up all targets that are the infrastructure for the services (like SAP, PS, etc...)
  -- One SYSTEM can be an infrastructure for multiple services.
Use SYSTEMs and beacons for end-to-end monitoring.
  -- Service test is end-to-end test.
  -- Availability can be defined per service

Some events are statefull and some stateless. Statefull
emcli clear_stateless_elements -older_then=7 -target_type=oracle_database  (older than 7 days)

Events/Incidents/Problems
====================
For Enterprise Manager 12c, the concept and function of notification rules has been replaced with a two-tier stem consisting of Event Rules and Incident Rule Sets.
Create rules on any of these, to email or do something.
Problem is a type of incident with the same root cause
Incident is something you need to act on
Event Rules: Operate at the lowest level granularity (on discrete events) and performs the same role as notification rules from earlier releases.

Software Lifecycle Management
=======================
Roles can be more granular: admin, designer (designs workflows), operators
DBA needs to configure the software library, put gold images inside
  -- OMS and all agents need to be able to access the shared file system
Deployment:
  -- create a database profile
  -- create a procedure workflow (Like "Clone Database", "Patch database")
  -- publish it and Launch it
  -- Ability to pause and resume upgrade (by setting breakpoints

Reports
======
Lots of Predefined reports
!! We can replace our daily report with one of the OEM reports
https://zpgrid1:7799/public/reports,

Oracle Enterprise Manager Connectors Integration
=====================================
Guide number E25163-4

OMS is a J2EE app deployed via Web Logic. It has 3 apps:
1 - The Console serves up all the /em URLs.
2 - PBS (Platform Background Services) serves up all the /empbs URLs. It is where agents
upload their metrics.
3 - OCMRepeater is the link between EM CC and My Oracle Support for consolidating
configuration data collected from agents.

12 new:
Each agent has plug-ins. They are automatically discovered.

Backup/Restore OMS:
/u01/app/oracle/product/middleware/oms/bin/emctl exportconfig oms
emctl importconfig oms –file /.../EMGC_OMS1/sysman/backup/OMS_backups.bka

Default ports:
2889/2900 or 3872/3870 between agent and OMS
7788/7799  between OMS and CC
JDBC:1521 from OMS to Rep

-Installation from s/w distribution is possible.
-11g required a pre-existing WLS installation; 12c does not.
-12c metrics are gathered by agent plug-ins
-OMs replication means copy the config files only (OMS needs to be shut down)
-OMS and OMR can run on the same server. No advantage to separating it.

Enterprise Manager startup sequence:
1. OMR listener and database instance
2. OMS
3. Agent on OMS server
• Shutdown is in the reverse order.

Installation:
1. First install OREP database, as normal
2. Download software from http://www.oracle.com/technetwork/oem/grid-control/downloads/linuxx8664soft-085949.html
3. Unpack
4. Run prerequsite on OREP
5. cd to it, ./runInstaller
6. Choose medium size db...
7. Backup OMS
8. Check that https://host:7799/em is working. Secure it with certificate from the trusted domain.

Upgrade:

1-system upgrade:
- The Upgrade Console is installed into your current OMS by applying Patch 14375077
- Follow the process.
- Grid is down all the time.
- Agents first, then OMS

2-system upgrade:
- No loss of metric data
- We can have 11g and 12c running in parallel
- The Upgrade Console is installed into your current OMS by applying Patch 14375077
- Execute post-installation scripts
$OH/bin/rcuJDBCEngine sys/Oracle123@host:1521:orep JDBC_SCRIPT post_install_script.sql $PWD $ORACLE_HOME
$OHbin/rcuJDBCEngine  sys/Oracle123@host:1521:em11rep JDBC_SCRIPT post_install_patch.sql $PWD $ORACLE_HOME
Apply latest PSU
- Install the 12c agent software
   -- Put it in the directory where console can see it
   -- Copy the emkey from oms to repository
- Configure Upgrade console
  -- In "Identify Host.." specify new hostname
  -- In Manage Softare, specify the link to the software
  -- In "Provide Backup.." put date of the templates, or date after
- Clone the current repository
  -- Create a template (with data) from your existing 11g repository
  -- Start the 11g OMS and remove the emkey from the repository
        ##The emkey is a random number that is generated during the installation of the Repository and is stored in a table. When the OMS is installed, the emkey is copied from the  Rep to the $OH/sysman/config/emkey.ora file. After the emkey has been copied, you must remove it from the  Repository as it is not considered secure. If it is not removed, data such as database passwords, server passwords and other sensitive information can be easily decrypted.
  -- Scp the template accross
  -- Use dbca to create a new repository from the template
  -- Other option is to clone it any other way, as long as data is the same #What about removing the emkey?
  -- grant execute on dbms_random to sysman, dbsnmp; #additional privileges needed
- Check the new 12c repository database
  -- emprereq command
  -- If changes are required, run the emprereqkit again and let it make changes that can be auto-corrected.
- Install OMS 12c
  -- ./runINstaller on the new host
     # Chose an option "Upgrade an Existing OEM with 2-Server" not the "Create new OEM"
- In "Link to old OMR" part:
  --grep CONNECTDESCRIPTOR /u01/app/oracle/product/middleware/gc_inst/em/EMGC_OMS1/emgc.properties
    and attach that into the first field
- Use the Upgrade console to deploy 12c agents (check space first)
  --Agent on zpgrid1 can be skipped
  -- Add/Select other agents in “Deploy and Configure Agents”, follow the instructions and Submit a Job
- Agent Health check
  -- Click the “Generate Health Report of Deployed Agents” link, add the targets that have 12c agent deployed
  -- Sign off
- Switch to 12c agent
  -- "Switch Agents" and add the target
  -- This taks supposedlyalso runs Accrued data
- Access Post-Upgrade Console
  -- Navigate to Setup > Manage Cloud Control > Post Upgrade Tasks \ Accrued Data and Submit job for all Components
  -- This is just a read-only view (apparently).
  -- Run what you need in the Deferred Data.
  -- Sign off : This task also de-installs the 11g agent
- Wait for all agents to be activated on 12c before decommissioning the current OMS #if that's what you want to do
  -- While waiting, 11g agents will be active
- If all agents can't be activated, revert to 11g

How to discover targets:
1. Manually discover the host
2. Set up ssh to it from OMS host
3. Push the agent to it
4. Do auto discover

Maintenance
=========
Web Logic has to be patched separately

Security
======
- Backup the emkey emctl exportconfig oms -dir  
- Remove emkey from the 11g repository #don't remove till you backup
- Assuming that you use the default secure communications between the agents and the OMS,
disable unsecure communications with the emctl command: emctl secure lock.
- Secure the console with an SSL certificate from a trusted authority # check what has been done
- Check OEM DR whether it's working

Latest patch right now: 12.1.0.2

OEM CC

Chargeback basaed on capacity

SQL Developer

Multi Tenant

Creating

Plugging in

Dropping

Migration 11G--> 12C

Managing

Backup/Recovery

HEAT MAP or ILM – Advanced Compression option

Move DF online

IN-DB ARCHIVING

Temporal Validity

Unified Auditing – an option

Privileges

Redaction – option (hiding data) !

RMAN

Monitoring

Schema Comparison DBLM Pack !

Adaptive SQL Plan Mgmt – Tuning pack !

Adaptive Executions Plan

Stats

Histograms

Emergency Monitoring – Diag pack

ASH

Multi-threaded architecture

Flash Cache

Paralel In-memory on RAC

Temporary UNDO

Multiple Indexes

Invisible Columns

Online Redefinition

Compression

ONLINE DMLs

ADR

Network performance: Compression – option!

Data Pump, Loader & External Tables

Partitioning

Various