Tuesday, January 24, 2012

EM on Oracle Fail Safe on Windows

Oracle Fail Safe is the oldest fail-over techniques ... invented much much more before RAC. The matter of fact it was only solution before RAC come to stable phase. Because of easiness of implementation and stability in work, Oracle on Oracle Fail Safe on any OS environment was the biggest advantage for a long time. How ever, if you wanted to use EM (Enterprise Manager) on such a platform, until Grid came, you were helpless. So I'll explain how to implement EM on OFS. Maybe you may ask why is this a topic, relatively easy answer is that EM is very sensitive in IP addresses change, what in OFS is more than normal case!

Keep in mind that such a configuration is not officially certified nor recommended) from Oracle, so use it on own risk.

Software requirements

To be able to perform next steps, some requirements must be obeyed.
  • OS
    • All newer Windows and Linux versions are fully supported (in all CPU versions) and this example will be done on Windows 2008 R2 x64
  • Oracle database
    • Almost all version are supported from 9i to 11g, and this example will be on latest 11g R2 (11.2.0.3)
  • Oracle Fail Safe
    • I have successfully tested this examples on OFS 3.4.2 releases and this particular case was on latest possible OFS 3.4.2.3
As you see, all versions are latest possible releases, what might sound bad in some ears, but first impressions are more then OK. Also, this example presume that you have successfully installed and tested OFS fail-over on Windows and attached database to OFS group. On OFS with two nodes configuration, fully configured OFS console looks like:
To make more understandable next steps, let me say that OFS DB_group has virtual IP address 192.168.168.158, which is attached to virtual cluster IP address 192.168.168.157. Of course, real nodes has own real IP addresses, but for this case this is unimportant. All IP addresses use IPV4 protocol.
Additionally, database (SID) attached to cluster is TRA. Listener is configured on standard Oracle address-1521. Oracle database is installed in "c:\oracle\product\11.2.0\dbhome_2" on both nodes.

Let me show this with real data together in one place:
C:\Users\teb>ipconfig

Windows IP Configuration


Ethernet adapter HPTeam:

   Connection-specific DNS Suffix  . :
   IPv4 Address. . . . . . . . . . . : 192.168.168.149
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   IPv4 Address. . . . . . . . . . . : 192.168.168.157
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   IPv4 Address. . . . . . . . . . . : 192.168.168.158
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.168.1

Ethernet adapter Local Area Connection* 9:

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::3024:711b:32b1:b57f%14
   IPv4 Address. . . . . . . . . . . : 169.168.2.53
   Subnet Mask . . . . . . . . . . . : 255.255.0.0
   Default Gateway . . . . . . . . . :

Tunnel adapter isatap.{C9E1C140-6F0F-411C-BD16-C7D94EF093AD}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :

Tunnel adapter isatap.{FAA77D5D-C68B-425A-B3CE-F3CED3951328}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :

C:\Users\teb>ping -a 192.168.168.157

Pinging NAPTRAFC [192.168.168.157] with 32 bytes of data:
Reply from 192.168.168.157: bytes=32 time<1ms TTL=128
Reply from 192.168.168.157: bytes=32 time<1ms TTL=128
Reply from 192.168.168.157: bytes=32 time<1ms TTL=128
Reply from 192.168.168.157: bytes=32 time<1ms TTL=128

Ping statistics for 192.168.168.157:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

c:\oracle\product\11.2.0\dbhome_2\BIN>tnsping TRA

TNS Ping Utility for 64-bit Windows: Version 11.2.0.3.0 - Production on 23-SIJ-2012 21:38:01

Copyright (c) 1997, 2011, Oracle.  All rights reserved.

Used parameter files:
c:\oracle\product\11.2.0\dbhome_2\network\admin\sqlnet.ora


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION= (ADDRESS_LIST= (ADDRESS= (PROTOCOL=TCP) (Host=192.168.168.158) (Port=1521))) (CONNECT_DATA= (SID=TRA)))
OK (0 msec)

Step 1-setting environment

Initial step is done on node where DB_group reside (has connection to database resources). Before starting DB control repository creation, ensure that any previous DB repository is removed and that next three env. variables are set:
C:\Users\teb>set ORACLE_HOSTNAME=192.168.168.158

C:\Users\teb>set ORACLE_SID=TRA

C:\Users\teb>set ORACLE_HOME=c:\oracle\product\11.2.0\dbhome_2
Importance of ORACLE_HOSTNAME variable is in that this is the only way to tell Oracle which host IP address should be used for repository. And this is only DB_Group virtual IP address. Do not mix that with cluster virtual IP address.

Step 2-Create DB control repository

After setting environment, DB control repository< creation is straight forward action. Just to mention that DBSNMP user has to unlocked and altered with some password. Here is log of that action:
c:\oracle\product\11.2.0\dbhome_2\BIN>emca -config dbcontrol db -repos create



STARTED EMCA at 2012.01.23 21:38:49

EM Configuration Assistant, Version 11.2.0.3.0 Production

Copyright (c) 2003, 2011, Oracle.  All rights reserved.



Enter the following information:

Database SID: TRA

Listener port number: 1521

Listener ORACLE_HOME [ c:\oracle\product\11.2.0\dbhome_2 ]:

Password for SYS user:

Password for DBSNMP user:

Password for SYSMAN user:

Password for SYSMAN user: Email address for notifications (optional):

Outgoing Mail (SMTP) server for notifications (optional):

-----------------------------------------------------------------



You have specified the following settings



Database ORACLE_HOME ................ c:\oracle\product\11.2.0\dbhome_2



Local hostname ................ 192.168.168.158

Listener ORACLE_HOME ................ c:\oracle\product\11.2.0\dbhome_2

Listener port number ................ 1521

Database SID ................ TRA

Email address for notifications ...............

Outgoing Mail (SMTP) server for notifications ...............



-----------------------------------------------------------------

Do you wish to continue? [yes(Y)/no(N)]:Y

2012.01.23 21:39:48 oracle.sysman.emcp.EMConfig perform

INFO: This operation is being logged at c:\oracle\cfgtoollogs\emca\TRA\emca_2012_01_23_21_38_49.log.

2012.01.23 21:39:50 oracle.sysman.emcp.EMReposConfig createRepository

INFO: Creating the EM repository (this may take a while) ...

2012.01.23 21:46:13 oracle.sysman.emcp.EMReposConfig invoke

INFO: Repository successfully created

2012.01.23 21:49:13 oracle.sysman.emcp.EMReposConfig uploadConfigDataToRepository

INFO: Uploading configuration data to EM repository (this may take a while) ...

2012.01.23 21:50:26 oracle.sysman.emcp.EMReposConfig invoke

INFO: Uploaded configuration data successfully

2012.01.23 21:50:28 oracle.sysman.emcp.ParamsManager getLocalListener

WARNING: Error retrieving listener for 192.168.168.158

2012.01.23 21:50:33 oracle.sysman.emcp.util.DBControlUtil secureDBConsole

INFO: Securing Database Control (this may take a while) ...

2012.01.23 21:50:45 oracle.sysman.emcp.util.DBControlUtil secureDBConsole

INFO: Database Control secured successfully.

2012.01.23 21:50:45 oracle.sysman.emcp.util.DBControlUtil startOMS

INFO: Starting Database Control (this may take a while) ...

2012.01.23 21:52:01 oracle.sysman.emcp.EMDBPostConfig performConfiguration

INFO: Database Control started successfully

2012.01.23 21:52:02 oracle.sysman.emcp.EMDBPostConfig performConfiguration

INFO: >>>>>>>>>>> The Database Control URL is https://192.168.168.158:1158/em <<<<<<<<<<<
2012.01.23 21:52:04 oracle.sysman.emcp.EMDBPostConfig invoke
WARNING:
************************  WARNING  ************************

Management Repository has been placed in secure mode wherein Enterprise Manager data will be encrypted.  The encryption key has been placed
in the file: c:/oracle/product/11.2.0/dbhome_2/192.168.168.158_TRA/sysman/config/emkey.ora. Ensure this file is backed up as the encrypted dat
a will become unusable if this file is lost.

***********************************************************
Enterprise Manager configuration completed successfully
FINISHED EMCA at 2012.01.23 21:52:04

c:\oracle\product\11.2.0\dbhome_2\BIN>

Step 3-checking installed EM on first node

Before going to any further step, verification that you have correctly installed EM is vital. For that just logon to mentioned addres through WEB url:

Step 4-Understand EM location directories

DB control repository is, beside database storage, saved on two directories in active ORACLE_HOME:
  1. %ORACLE_HOME%\<hostname_sid>
  2. %ORACLE_HOME%\oc4j\j2ee\OC4J_DBConsole_<hostname_sid>
In our case this is:
  1. %ORACLE_HOME%\192.168.168.158_TRA
  2. %ORACLE_HOME%\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA
As you see these two locations are on local and not shared cluster disks. So what we need to do is to create NTFS junction point (symbolic link in Linux) which will point to some location on shared disk. In mine case this would be disk E, "e:\oracle\EM\TRA\" directory.

Step 5-creating NTF junction points

Stop DB console:
c:\oracle\product\11.2.0\dbhome_2\BIN>emctl stop dbconsole
Oracle Enterprise Manager 11g Database Control Release 11.2.0.3.0
Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.
https://192.168.168.158:1158/em/console/aboutApplication
The OracleDBConsoleTRA service is stopping.........
The OracleDBConsoleTRA service was stopped successfully.
Create directory on shared disk:
mkdir e:\oracle\EM\TRA\192.168.168.158_TRA
mkdir e:\oracle\EM\TRA\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA
Copy mentioned two directories in ORACLE_HOME to share disks location:
xcopy %ORACLE_HOME%\192.168.168.158_TRA e:\oracle\EM\TRA\192.168.168.158_TRA /E /Y
xcopy %ORACLE_HOME%\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA e:\oracle\EM\TRA\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA  /E /Y
Here is the full log of all operations:
c:\oracle\product\11.2.0\dbhome_2\BIN>xcopy %ORACLE_HOME%\192.168.168.158_TRA e:\oracle\EM\TRA\192.168.168.158_TRA /E /Y
Does E:\oracle\EM\TRA\192.168.168.158_TRA specify a file name
or directory name on the target
(F = file, D = directory)? D
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\config\b64InternetCertificate.txt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\config\b64LocalCertificate.txt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\config\classpath.lst
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\config\emagentlogging.properties
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\config\emd.properties
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\config\emd.properties.2012-01-23-21-50-51
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\config\emd.properties.bak.2012_01_23_21_50_45_612
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\config\emd.properties.tzbak
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\config\emkey.ora
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\config\emoms.properties
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\config\emoms.properties.bak.2012_01_23_21_50_45_612
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\config\emomsintg.xml
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\config\emomslogging.properties
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\config\OUIinventories.add
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\config\monwallet\cwallet.sso
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\config\monwallet\ewallet.p12
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\config\server\ewallet.p12
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\config\server\repoconn.ora
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\agntstmp.txt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\emagent_storage.config
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\lastupld.xml
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\protocol.ini
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\targets.xml
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\WMITestAutomation.out
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0000.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0001.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0002.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0003.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0004.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0005.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0006.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0007.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0008.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0009.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0010.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0011.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0012.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0013.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0014.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0015.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0016.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0017.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0018.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0019.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0020.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0021.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\0022.dlt
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\91E46007A92B3E6CB73B18053C07BBD0_seg_adv_count.log
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\corr.save
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\snapshot
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\adr\91E46007A92B3E6CB73B18053C07BBD0.adr.base
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\adr\91E46007A92B3E6CB73B18053C07BBD0.adr.home
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\state\adr\91E46007A92B3E6CB73B18053C07BBD0.alert.log.xml.state
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\upload\192.168.168.158_host_host_configuration_old
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\upload\errors.dat
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\upload\mgmt_db_featureusage.dat
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\upload\rawdata.dat
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\upload\X2.uod
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\emd\upload\X3.uod
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\log\emagent.log
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\log\emagent.trc
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\log\emagentfetchlet.log
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\log\emagentfetchlet.trc
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\log\emagent_perl.trc
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\log\emdb.nohup
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\log\emdctl.log
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\log\emdctl.trc
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\log\emoms.log
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\log\emoms.trc
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\log\nfsPatchPlug.log
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\log\OracleDBConsoleTRAexit.log
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\log\OracleDBConsoleTRAsrvc.log
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\log\patchAgtStPlugin.log
C:\oracle\product\11.2.0\dbhome_2\192.168.168.158_TRA\sysman\log\secure.log
74 File(s) copied

c:\oracle\product\11.2.0\dbhome_2\BIN>xcopy %ORACLE_HOME%\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA e:\oracle\EM\TRA\oc4j\j2ee\OC4J_DBConso
le_192.168.168.158_TRA  /E /Y
Does E:\oracle\EM\TRA\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA specify a file name
or directory name on the target
(F = file, D = directory)? D
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\application-deployments\default\datasources\oc4j-ra.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\application-deployments\default\defaultWebApp\orion-web.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\application-deployments\default\defaultWebApp\persistence\_TldC
ache
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\application-deployments\default\OracleASjms\oc4j-ra.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\application-deployments\dms\orion-web.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\application-deployments\em\orion-application.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\application-deployments\em\principals.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\application-deployments\em\datasources\oc4j-ra.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\application-deployments\em\em\orion-web.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\application-deployments\em\em\persistence\_TldCache
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\application-deployments\em\em\persistence\tlds\db
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\application-deployments\em\em\persistence\tlds\oem
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\application-deployments\em\em\persistence\tlds\oemjsp
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\application.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\data-sources.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\default-web-site.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\entity-resolver-config.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\global-web-application.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\http-web-site.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\http-web-site.xml.bak.2012_01_23_21_50_45_612
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\internal-settings.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\j2ee-logging.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\java2.policy
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\javacache.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\jazn-data.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\jazn.security.props
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\jazn.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\jms.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\mime.types
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\oc4j-connectors.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\oc4j.properties
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\orb-config.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\principals.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\rmi.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\server.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\system-application.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\system-jazn-data.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\transaction-manager.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\database-schemas\hypersonic.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\database-schemas\ms-access.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\database-schemas\ms-sql.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\database-schemas\oracle.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\database-schemas\postgresql.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\database-schemas\sybase.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\config\server\ewallet.p12
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\connectors\README.txt
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\connectors\datasources\datasources.rar
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\connectors\datasources\datasources\datasources.jar
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\connectors\datasources\datasources\META-INF\MANIFEST.MF
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\connectors\datasources\datasources\META-INF\oc4j-ra.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\connectors\datasources\datasources\META-INF\ra.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\connectors\OracleASjms\OracleASjms.rar
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\connectors\OracleASjms\OracleASjms\gjra.jar
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\connectors\OracleASjms\OracleASjms\META-INF\MANIFEST.MF
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\connectors\OracleASjms\OracleASjms\META-INF\oc4j-ra.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\connectors\OracleASjms\OracleASjms\META-INF\ra.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\log\em-application.log
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\log\global-application.log
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\log\rmi.log
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\log\server.log
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\log\system-application.log
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\log\oc4j\log.xml
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\META-INF\MANIFEST.MF
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\tldcache\_GlobalTldCache
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\tldcache\ojsputil_jar\ejbtaglib
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\tldcache\ojsputil_jar\email
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\tldcache\ojsputil_jar\fileaccess
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\tldcache\ojsputil_jar\jesitaglib
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\tldcache\ojsputil_jar\jwcache
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\tldcache\ojsputil_jar\personalization
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\tldcache\ojsputil_jar\sqltaglib
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\tldcache\ojsputil_jar\utiltaglib
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\tldcache\ojsputil_jar\wstaglib
C:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA\tldcache\ojsputil_jar\xml
74 File(s) copied
Rename two copied directories (you can delete them later when you are sure that all went OK!):
c:\oracle\product\11.2.0\dbhome_2>dir 10*
 Volume in drive C has no label.
 Volume Serial Number is B862-204E

 Directory of c:\oracle\product\11.2.0\dbhome_2

23.01.2012.  22:00    <dir>          192.168.168.158_TRA
               0 File(s)              0 bytes
               1 Dir(s)  26.643.705.856 bytes free

c:\oracle\product\11.2.0\dbhome_2>rename 192.168.168.158_TRA 192.168.168.158_TRA-old

c:\oracle\product\11.2.0\dbhome_2>dir 10*
 Volume in drive C has no label.
 Volume Serial Number is B862-204E

 Directory of c:\oracle\product\11.2.0\dbhome_2

23.01.2012.  22:00    <dir>          192.168.168.158_TRA-old
               0 File(s)              0 bytes
               1 Dir(s)  26.641.604.608 bytes free

c:\oracle\product\11.2.0\dbhome_2>
Create NTFS junction point from the local disk to the shared disk:
c:\oracle\product\11.2.0\dbhome_2>mklink /J 192.168.168.158_TRA e:\oracle\EM\TRA\192.168.168.158_TRA
Junction created for 192.168.168.158_TRA <<===>> e:\oracle\EM\TRA\192.168.168.158_TRA

c:\oracle\product\11.2.0\dbhome_2>
Repeat similar two steps for other directory. Here is action log :
c:\oracle\product\11.2.0\dbhome_2>cd %ORACLE_HOME%\oc4j\j2ee

c:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee>dir
 Volume in drive C has no label.
 Volume Serial Number is B862-204E

 Directory of c:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee

23.01.2012.  21:50    <dir>          .
23.01.2012.  21:50    <dir>          ..
16.01.2012.  15:37             1.025 deploy_db_wf.ini
16.01.2012.  15:32    <dir>          home
16.01.2012.  15:30    <dir>          oc4j_applications
16.01.2012.  15:30    <dir>          OC4J_DBConsole
23.01.2012.  21:50    <dir>          OC4J_DBConsole_192.168.168.158_TRA
16.01.2012.  15:30    <dir>          OC4J_Workflow_Component_Container
16.01.2012.  15:30    <dir>          OC4J_Workflow_Management_Container
16.01.2012.  15:32    <dir>          utilities
               1 File(s)          1.025 bytes
               9 Dir(s)  26.641.440.768 bytes free

c:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee>rename OC4J_DBConsole_192.168.168.158_TRA OC4J_DBConsole_192.168.168.158_TRA-old

c:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee>mklink /J OC4J_DBConsole_192.168.168.158_TRA e:\oracle\EM\TRA\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA
Junction created for OC4J_DBConsole_192.168.168.158_TRA <<===>> e:\oracle\EM\TRA\oc4j\j2ee\OC4J_DBConsole_192.168.168.158_TRA

c:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee>
From Explorer point of view placed junction points looks like:


Check that junction points are correctly pointing to real directory folder by clicking on them from explorer for both junction points.

Step 6-Check EM is running on NTFS junction points

Now you have fully replaced original DB Control repository to location on shared disks. Start DB Console service and verify that everything is still working fine, with this mounted location.
c:\oracle\product\11.2.0\dbhome_2\oc4j\j2ee>emctl start dbconsole
Oracle Enterprise Manager 11g Database Control Release 11.2.0.3.0
Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.
https://192.168.168.158:1158/em/console/aboutApplication
Starting Oracle Enterprise Manager 11g Database Control ...The OracleDBConsoleTRA service is starting................
The OracleDBConsoleTRA service was started successfully.
Check EM connectivity through WEB url as described before.

Step 7-Create DB repository on other node

Now similar task has to be done on other node of cluster. If you have more nodes, repeat them on each one.
  1. Stop the DB Console service on current node.
  2. Failover DB_Group to other node.
Once, when database is fully opened, repeat steps 1, 2.
C:\Users\teb>emca -config dbcontrol db

STARTED EMCA at 2012.01.23 23:07:08
EM Configuration Assistant, Version 11.2.0.3.0 Production
Copyright (c) 2003, 2011, Oracle.  All rights reserved.

Enter the following information:
Database SID: TRA
Listener port number: 1521
Listener ORACLE_HOME [ c:\oracle\product\11.2.0\dbhome_2 ]:
Password for SYS user:  y
Password for DBSNMP user:
Password for SYSMAN user:
Password for SYSMAN user: Email address for notifications (optional):
Outgoing Mail (SMTP) server for notifications (optional):
-----------------------------------------------------------------

You have specified the following settings

Database ORACLE_HOME ................ c:\oracle\product\11.2.0\dbhome_2

Local hostname ................ 192.168.168.158
Listener ORACLE_HOME ................ c:\oracle\product\11.2.0\dbhome_2
Listener port number ................ 1521
Database SID ................ TRA
Email address for notifications ...............
Outgoing Mail (SMTP) server for notifications ...............

-----------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: Y
2012.01.23 23:07:40 oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at c:\oracle\cfgtoollogs\emca\TRA\emca_2012_01_23_23_07_08.log.
2012.01.23 23:07:44 oracle.sysman.emcp.EMReposConfig uploadConfigDataToRepository
INFO: Uploading configuration data to EM repository (this may take a while) ...
2012.01.23 23:09:48 oracle.sysman.emcp.EMReposConfig invoke
INFO: Uploaded configuration data successfully
2012.01.23 23:09:49 oracle.sysman.emcp.ParamsManager getLocalListener
WARNING: Error retrieving listener for 192.168.168.158
2012.01.23 23:09:53 oracle.sysman.emcp.util.DBControlUtil secureDBConsole
INFO: Securing Database Control (this may take a while) ...
2012.01.23 23:09:57 oracle.sysman.emcp.util.PlatformInterface executeCommand
WARNING: Error executing CMD /C c:\oracle\product\11.2.0\dbhome_2\bin\emctl.bat config emkey -repos
2012.01.23 23:09:57 oracle.sysman.emcp.util.DBControlUtil secureDBConsole
WARNING: Configuring EM-Key failed.
2012.01.23 23:09:57 oracle.sysman.emcp.EMDBPostConfig performConfiguration
WARNING: Error securing Database control.
2012.01.23 23:09:57 oracle.sysman.emcp.EMDBPostConfig setWarnMsg
INFO: Error securing Database Control. Database Control has been brought-up in non-secure mode. To secure the Database Contro
l execute the following command(s):

 1) Set the environment variable ORACLE_UNQNAME to Database unique name
 2) c:\oracle\product\11.2.0\dbhome_2\bin\emctl.bat stop dbconsole
 3) c:\oracle\product\11.2.0\dbhome_2\bin\emctl.bat config emkey -repos -sysman_pwd < Password for SYSMAN user >
 4) c:\oracle\product\11.2.0\dbhome_2\bin\emctl.bat secure dbconsole -sysman_pwd < Password for SYSMAN user >
 5) c:\oracle\product\11.2.0\dbhome_2\bin\emctl.bat start dbconsole

 To secure Em Key, run c:\oracle\product\11.2.0\dbhome_2\bin\emctl.bat config emkey -remove_from_repos -sysman_pwd < Password
 for SYSMAN user >
2012.01.23 23:09:57 oracle.sysman.emcp.util.DBControlUtil startOMS
INFO: Starting Database Control (this may take a while) ...
2012.01.23 23:11:13 oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: Database Control started successfully
2012.01.23 23:11:13 oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: >>>>>>>>>>> The Database Control URL is http://192.168.168.158:1158/em <<<<<<<<<<<


Error securing Database Control. Database Control has been brought-up in non-secure mode. To secure the Database Control exec
ute the following command(s):

 1) Set the environment variable ORACLE_UNQNAME to Database unique name
 2) c:\oracle\product\11.2.0\dbhome_2\bin\emctl.bat stop dbconsole
 3) c:\oracle\product\11.2.0\dbhome_2\bin\emctl.bat config emkey -repos -sysman_pwd < Password for SYSMAN user >
 4) c:\oracle\product\11.2.0\dbhome_2\bin\emctl.bat secure dbconsole -sysman_pwd < Password for SYSMAN user >
 5) c:\oracle\product\11.2.0\dbhome_2\bin\emctl.bat start dbconsole

 To secure Em Key, run c:\oracle\product\11.2.0\dbhome_2\bin\emctl.bat config emkey -remove_from_repos -sysman_pwd < Password
 for SYSMAN user >

C:\Users\teb>
When this is finished, on local disk (under %ORACLE_HOME%) the same directories would be created as in initial creation. After that stop DB console (do not test WEB address because it probably will not work) and repeat step 5 to create 2 links to same shared directories as it was shown before.
Start DB console as explained on this node. Now check connectivity through WEB-it should be OK with same url value!

Step 8-Add DB console to Cluster group

Now that you have verified running DB console on both nodes on shared directories, it is time to add this resource to DB_Group (Oracle cluster group). So add "OracleDBConsoleTRA" (generic name should be OracleDBConsole) as generic service to existing DB_Group.
For dependencies, place all other group resources to ensure that DB is already fully opened when "OracleDBConsoleTRA" will start.
Check (for the last time) WEB url and try the same after fail-over to other node. All should works like a charm.

The End

Regardless this is not verified configuration, I ty nice results with it. If you add that for "supported" Oracle solution you need EM Grid and two agent licences, cost savings is obvious.
Hope that this doesn't violate any Oracle licence and that will help someone.

Cheers!

Sunday, January 8, 2012

Oracle 11g architecture Interactive Quick Reference

People say that picture shows more then 1.000 words. I agree 1,000%.

Here is Oracle 11g architecture in one image, taken from official Oracle site.

  • Processes shown are started by default
  • Processes in red are mandatory
  • Processes ARC and RVWR are not started by default but are recommended for productions
  • Processes marked with * are new in 11g

For those who want full information, here is magnificent flash that explain many thing with details ...

Hope will help someone as it is to for quick reference.

This information are taken from official Oracle site and may be downloaded from original Oracle location.

Cheers!

Tuesday, January 3, 2012

Dangerous careless trigger declaration




Just few days before going to Christmas holidays, I have run into one pretty interesting and dangerous situation which was caused because careless trigger definition. The beauty of this example is that it is very hard to see real problem until you run in it. I had a luck that, regardless all happened on production database, involved table were part of source data for warehouse part, which is updated once a day in a batch job, so nothing luckily went wrong.

The problem arise when I have imported (impdp) one table from original schema to other one (just to see some structure and data). Database version as well as OS is totally unimportant for this issue-so I may say this danger is present in any database version across all platforms. Just to mention that original schema is TAB and table was imported to TAB_DWH schema. It is also important to mention that in original schema (TAB) the same table (OSO_ATOL) exists with a lot of data.

Here is log from that import:
F:\>imp '/ as sysdba' FILE='hcpro_4_cet.dmp' fromuser=tab touser=tab_dwh log=imp_oso_atol.log tables=('OSO_ATOL') RECORDLENGTH=65535 BUFFER=50000000

Import: Release 10.2.0.4.0 - Production on Pet Pro 16 15:57:03 2011

Copyright (c) 1982, 2007, Oracle.  All rights reserved.


Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

Export file created by EXPORT:V10.02.01 via direct path
import done in EE8MSWIN1250 character set and AL16UTF16 NCHAR character set
. importing TAB's objects into TAB_DWH
. . importing table                     "OSO_ATOL"          0 rows imported
IMP-00041: Warning: object created with compilation warnings
 "CREATE TRIGGER "TAB_DWH".TGBI_OSO_ATOL_KRE"
 "BEFORE INSERT"
 "ON TAB.OSO_ATOL "
 "REFERENCING NEW AS NEW OLD AS OLD"
 "FOR EACH ROW"
 "begin"
 "  IF :NEW.UNIID IS NULL THEN "
 "    FOR c1 IN (SELECT OSO_ATOL_SEQ.NEXTVAL next_val FROM dual) LOOP"
 "      :NEW.UNIID :=  c1.next_val;"
 "      IF :new.korkre is NULL THEN"
 "        :new.korkre:=user;"
 "        :new.datkre:=sysdate;"
 "      END IF;"
 "    END LOOP;       "
 "  END IF;"
 "end;"
About to enable constraints...
Import terminated successfully with warnings.
As you see table had no rows and import was done with sys user and finished with some warnings in trigger creation. Let us focus on that warned trigger and it's code in original declaration:
CREATE OR REPLACE TRIGGER TGBI_OSO_ATOL_KRE
BEFORE INSERT
ON TAB.OSO_ATOL
REFERENCING NEW AS NEW OLD AS OLD
BEGIN
  IF :NEW.UNIID IS NULL THEN
    FOR c1 IN (SELECT OSO_ATOL_SEQ.NEXTVAL next_val FROM dual) LOOP
      :NEW.UNIID :=  c1.next_val;
      IF :new.korkre is NULL THEN
        :new.korkre:=user;
        :new.datkre:=sysdate;
      END IF;
    END LOOP;
  END IF;
END;
This trigger generate ID (PK) value with one sequence and some information when record is created ... what is in this context irrelevant, but just want to explain that there is nothing rocket science in it.

When I look in TAB_DWH.OSO_ATOL table definition, which were just imported, I saw that there is no trigger there!?? In first moment I was surprised, but returning mine focus on mentioned import log, I saw interesting part:
...
BEFORE INSERT
ON TAB.OSO_ATOL
REFERENCING NEW AS NEW OLD AS OLD
...
So this looks like trigger should be created in TAB schema ... original OSO_ATOL table ... but I was importing OSO_ATOL in TAB_DWH schema-WTH ... ?!?

After checking TAB.OSO_ATOL table here is what I saw (picture taken from Toad):
As picture shows, beside original three triggers (original TAB owner) additional three triggers were created, two with no errors and one which was invalid!!
So in this moment:
  • OSO_ATOL was unable to get any new data because TGBI_OSO_ATOL_KRE trigger was invalid
  • If update occur, the same thing will fire twice-once for TAB and once for TAB_DWH triggers, which were the same
Pretty nasty and unpleasant situation indeed!

The Solution

The solution is very easy-define triggers without table owner. Simple as that. Here is correct code of same trigger:
CREATE OR REPLACE TRIGGER TGBI_OSO_ATOL_KRE
BEFORE INSERT
ON OSO_ATOL
REFERENCING NEW AS NEW OLD AS OLD
BEGIN
  IF :NEW.UNIID IS NULL THEN
    FOR c1 IN (SELECT OSO_ATOL_SEQ.NEXTVAL next_val FROM dual) LOOP
      :NEW.UNIID :=  c1.next_val;
      IF :new.korkre is NULL THEN
        :new.korkre:=user;
        :new.datkre:=sysdate;
      END IF;
    END LOOP;
  END IF;
END;
As you see table owner is avoided!

The End

First let me be honest and tell that if no warnings in import were shown error will be raised when it will be too late. But I had "luck" because trigger was invalid because sequence has no declared PUBLIC synonym, so in TAB_DWH schema OSO_ATOL_SEQ was unknown object.

Regardless this problem may be avoided if impdp was done with TAB_DWH user (which doesn't have enough privileges to create anything in TAB schema), ask yourself which user you use for ordinary impdp-sys or system!?

I think that there are many developers and fewer DBAs who are not familiar with mentioned problem, so introducing people around you is something that might help in protecting production database.

Hope this helps someone.

Cheers!

P.S.

Happy New 2012 again. Hope it would not be worse then 2011!
:-)