\ /
ZDRLA  Backup  Database  Oracle  Engineered Systems 

Zero Data Loss Recovery Appliance

Zero Data Loss Recovery Appliance

In the last weeks I had the fortune to follow the implementation and the first configuration of the latest Engineered system born in Oracle: ZDLRA (Zero Data Loss Recovery Appliance). It is an Engineered system dedicated to Oracle database backups and through a specific library allows the integration with RMAN and inside it has several automatic features that make it very interesting in the management of Oracle databases backups.

Physically it is like an Exadata and the basic configuration includes 2 db servers (Oracle Linux) and 3 storage cells, the cells can be scale up to a maximum of 18. Using the new X7 servers, with internal 10TB disks, you can have a minimum space of 76TB up to a maximum space of about 472TB using diskgroup in High redundancy mode. In the 2 db servers, during configuration through OEDA, a RAC database is created where the recovery appliance (RA) and RMAN catalogs will be contained, while on the storage cells two different ASM diskgroups, CATALOG and DELTA, will be created. The first will contain the RAC database and the second will be the destination of RMAN backups.

*The main features of the ZDLRA are:

  •  Automatic creation of Virtual full backup based on the first full backup and the subsequents incremental backups. The full backup will be only the first backup, while the remaining backups will always be incremental.
  •  Automatic management of deletion of backups within the retention policy, it will no longer be necessary to perform maintenance/cancellation of backups using RMAN commands (delete obsolete)
  •  Real Time Redo Transport, databases backed up with ZDLRA can be configured to send every single redo information (like a Dataguard) to the appliance, so you are sure to back up every committed transaction on the ZDLRA.
  •  Automatic data compression and deduplication for backups, through this function the space for backups is better managed.

The ZDLRA is monitored via Enterpise Manager Cloud Control (EMCC) by installing the EMCC agent and the specific plugins, once discovered and added in the EMCC you can configure the databases backups, they must be already discovered in EMCC.

For the configuration of the databases to be protected, the following activities are necessary. With the EMCC its will be automatically done, but can also be done manually:

1. Copy the OS library

In the $ORACLE_HOME/lib path on a RAC database nodes in the ZDLRA there are the zip files of the library divided for the various platforms

[oracle@zd01ingest01 lib]$ ls -lrt ra_*zip
-rw-r--r-- 1 oracle oinstall 27268290 Jan 26  2017 ra_linux64.zip
-rw-r--r-- 1 oracle oinstall 26158265 Jan 26  2017 ra_aix_ppc64.zip
-rw-r--r-- 1 oracle oinstall 22273315 Jan 26  2017 ra_solaris_x64.zip
-rw-r--r-- 1 oracle oinstall 33011832 Jan 26  2017 ra_solaris_sparc64.zip
-rw-r--r-- 1 oracle oinstall 75146708 Jan 26  2017 ra_hpux_ia64.zip
-rw-r--r-- 1 oracle oinstall 27302947 Jan 26  2017 ra_zlinux64.zip
-rw-r--r-- 1 oracle oinstall 20225757 Jan 26  2017 ra_windows64.zip

the library will must be copied in the $ORACLE_HOME/lib path on the database server to protect.

2. Create catalog user

Using the scripts provided with the recovery appliance software, the user, that will be used to connect to the ZDLRA and will be the owner of the RMAN catalog, will be created

[root@zd01ingest01 bin]$ pwd
/opt/oracle.RecoveryAppliance/bin
[root@zd01ingest01 ~]# racli add vpc_user --user_name=vpctest
[vpctest] New Password:                                                 ----- <Password>
Tue Jun 12 16:22:26 2018: Start: Add vpc user vpctest.
Tue Jun 12 16:22:26 2018:        Add vpc user vpctest successfully.
Tue Jun 12 16:22:26 2018: End:   Add vpc user vpctest.
[root@zd01ingest01 ~]#

3. Wallet creation

To store the ZDLRA credential it's necessary to create a wallet. In the wallet the credential are inserted with Easy Connect method.

NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)
SQLNET.WALLET_OVERRIDE = true
WALLET_LOCATION = 
(SOURCE = 
    (METHOD = FILE)
    (METHOD_DATA =
    (DIRECTORY = /app/oracle/product/11.2.0/dbs/ra_wallet)
)
)

mkstore -wrl $ORACLE_HOME/dbs/ra_wallet -createALO -createCredential zd01ingest-scan:1521/zdlra:dedicated vpctest <Password>
IMPORTANT: Easy Connect wallet credentials with "dedicated/shared" mode specified, works only with RDBMS version 11.2 and above

4. Enroll database to ZDLRA

It's necessary to register (ENROLL) the database to be protect inside the RA, it will be done assigning protection policy, reserved space and associating the catalog owner To do this you need to connect to the ZDLRA database with RASYS user (owner RA software) via sqlplus

sqlplus rasys
exec DBMS_RA.ADD_DB(db_unique_name => 'CNM_PRO',protection_policy_name => 'GOLD',reserved_space => '100G');
exec DBMS_RA.GRANT_DB_ACCESS(db_unique_name => 'CNM_PRO',username => 'vpctest' );

5. Configure RMAN channels

Set the default RMAN channels to using ZDLRA

rman target /
CONFIGURE DEFAULT DEVICE TYPE TO 'SBT_TAPE';
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%U' PARMS  "SBT_LIBRARY=/app/oracle/product/11.2.0/lib/libra.so, ENV=(RA_WALLET='location=file:/app/oracle/product/11.2.0/dbs/zdlra credential_alias=zd01ingest-scan1-v817:1521/zdlra_817:dedicated')";

6. Register database in the RMAN catalog

The "classic" command to register the database inside the recovery catalog on ZDLRA

rman target / catalog /@zd01ingest-scan1-v817:1521/zdlra_817:dedicated
register database;

7. First full backup level 0

rman target / catalog /@zd01ingest-scan1-v817:1521/zdlra_817:dedicated
backup cumulative incremental level 0 filesperset 1 section size 64g database plus archivelog filesperset 32 ;

It is important to specified the "fileperset 1"* during the backup for enable the deduplication feature of the ZDLRA

Check the backup*

RMAN> list backup of datafile 1;

List of Backup Sets
===================

BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ --------------------
458     Incr 0  413.12M    SBT_TAPE    00:00:32     12-jun-2018 17:05:44
        BP Key: 459   Status: AVAILABLE  Compressed: YES  Tag: TAG20180612T170435
        Handle: VB$_4116808279_447I   Media:
List of Datafiles in backup set 458
File LV Type Ckp SCN    Ckp Time             Name
---- -- ---- ---------- -------------------- ----
1    0  Incr 56758646   12-jun-2018 17:05:12 /ora01_CNM/CNM_PRO/data/system01.dbf

8. Incremental backup level 1

rman target / catalog /@zd01ingest-scan1-v817:1521/zdlra_817:dedicated
backup cumulative incremental level 1 filesperset 1 section size 64g database;

Check all the backups

RMAN> list backup of datafile 1;

List of Backup Sets
===================

BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ --------------------
458     Incr 0  413.12M    SBT_TAPE    00:00:32     12-jun-2018 17:05:44
        BP Key: 459   Status: AVAILABLE  Compressed: YES  Tag: TAG20180612T170435
        Handle: VB$_4116808279_447I   Media:
List of Datafiles in backup set 458
File LV Type Ckp SCN    Ckp Time             Name
---- -- ---- ---------- -------------------- ----
1    0  Incr 56758646   12-jun-2018 17:05:12 /ora01_CNM/CNM_PRO/data/system01.dbf

BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ --------------------
578     Incr 1  32.00K     SBT_TAPE    00:00:03     12-jun-2018 17:07:42
        BP Key: 579   Status: AVAILABLE  Compressed: YES  Tag: TAG20180612T170739
        Handle: VB$_4116808279_577I   Media:
List of Datafiles in backup set 578
File LV Type Ckp SCN    Ckp Time             Name
---- -- ---- ---------- -------------------- ----
1    1  Incr 56758828   12-jun-2018 17:07:39 /ora01_CNM/CNM_PRO/data/system01.dbf

BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ --------------------
582     Incr 0  411.66M    SBT_TAPE    00:00:03     12-jun-2018 17:07:42
        BP Key: 583   Status: AVAILABLE  Compressed: YES  Tag: TAG20180612T170739
        Handle: VB$_4116808279_577_1   Media:
List of Datafiles in backup set 582
File LV Type Ckp SCN    Ckp Time             Name
---- -- ---- ---------- -------------------- ----
1    0  Incr 56758828   12-jun-2018 17:07:39 /ora01_CNM/CNM_PRO/data/system01.dbf

The last backup is a Virtual full backup created with the last incremental backup. You can see that the level is 0 and the "Ckp SCN" is the same of the last incremental backup.

To perform a restore you will proceed as usual, once connected to RMAN, in the target database and in the catalog, just run the restore command followed by the command of recover.

Real Time Redo Transport

From 11.2.0.3 database version and above, it's possible to configure the RTRT by settings this parameters:

alter system set log_archive_config='dg_config=(zdlra,CNM_PRO)';
alter system set log_archive_dest_2='service="zd01ingest-scan1-v817:1521/zdlra_817:dedicated"','valid_for=(all_logfiles,all_roles) ASYNC DB_UNIQUE_NAME=zdlra';
alter system set log_archive_dest_state_2=enable;

Basically are the same parameters used in the Dataguard configuration, in this case the ZDLRA would be the standby database even if it is not.

Start/Stop ZDLRA database

A particular note for the restart phase of the ZDLRA database. Sometimes is necessary to restart it (patching, maintenance, etc...), it is necessary to proceed in steps, before turning off the database it is necessary to turn off the RA software, with the following commands, after connected to the database:

sqlplus rasys
exec dbms_ra.shutdown;

For the subsequent restart, after turning on the database it is necessary to turn on the RA software:

sqlplus rasys
exec dbms_ra.startup;

To check the RA status query the view RASYS.RA_SERVER There is a scripts provided with the recovery appliance software that shows the status of the primary components of the RA, Cluster (crs), Database (db) and Recovery Appliance Server (ra_server)

[root@zd01ingest01 bin]$ pwd
/opt/oracle.RecoveryAppliance/bin
[root@zd01ingest01 bin]# ./racli status appliance
zd01ingest02 crs        state:  [ONLINE]
zd01ingest02 ra_server  state:  [ONLINE]
zd01ingest02 db         state:  [ONLINE]
zd01ingest01 db         state:  [ONLINE]
zd01ingest01 ra_server  state:  [ONLINE]
zd01ingest01 crs        state:  [ONLINE]

In case the ra_server is DOWN, during the backup/restore phase appear the following error:

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set
including current control file in backup set
channel ORA_SBT_TAPE_1: starting piece 1 at 12-JUN-18
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_SBT_TAPE_1 channel at 06/12/2018 16:54:27
ORA-19506: failed to create sequential file, name="CNM_PRO_olt59am1_1_1", parms=""
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
     KBHS-01404: See trace file /app/oracle/diag/rdbms/cnm_pro/CNM_PRO/trace/sbtio_19660_1.log for details
KBHS-00719: Error 'recovery appliance Error'; ORA-45164: The Recovery Appliance is not running.
KBHS-00700: HTTP response error 'Recovery Appliance Error'

RMAN>

viewing the related log file you show the error message:

...
Status => 500
Reason => Recovery Appliance Error
<?xml version="1.0"?>
<Error>
    <Code>recovery appliance Error</Code>
    <CodeDescription></CodeDescription>
    <Message>ORA-45164: The Recovery Appliance is not running.
</Message>
    <Retry>false</Retry>
    <Busy>false</Busy>
</Error>19660_1; Tue, 12 Jun 2018 16:54:26
KBHS-01404: See trace file /app/oracle/diag/rdbms/cnm_pro/CNM_PRO/trace/sbtio_19660_1.log for details
KBHS-00719: Error 'recovery appliance Error'; ORA-45164: The Recovery Appliance is not running.
KBHS-00700: HTTP response error 'Recovery Appliance Error'
...

I hope that this post has been interesting, following the link to official ZDLRA Data Sheet

Bye Glauco

comments powered by Disqus