Disaster Recovery - Documentation for BMC AMI Storage FDR 6.1 (2023)

One of the purposes of data backup with a product such as FDR is recovery of the data necessary to keep your business running at a disaster recovery site after a disaster of some sort disables your home data center.

Recovery procedures

This section details several procedures for recovering data from ABR backups after a disaster. These procedures may be used to recover all or a critical subset of your data at a disaster recovery site, and may also be used when you move back to your restored home site. These procedures may also be useful when you are simply moving to a new data center location.

• Stand Alone Restore (SAR) may be required to restore full-volume backups of the volume necessary to IPL your operating system. This may include the IPL volume (SYSRES), spool volumes, paging volumes, catalog volumes and others. If your disaster recovery site provides a “starter” system, you can execute FDR or ABR on that system to do the same restores.
• ABR full-volume recovery recreates entire DASD volumes on DASD at the new site. In each case, the recovered volume looks exactly like it did at the time of the most recent incremental backup. This is the most efficient way of rebuilding your data. However, it does require that the new site have a DASD volume of the proper type (for example, 3390) and size (or larger) for each DASD volume you plan to recover.
• ABR data set restore is used to restore the latest copy of individual data sets from the full and incremental ABR backups. This procedure allows you to restore selected data sets to new volumes, even if the new volumes are smaller or larger than the original volumes (as long as individual data sets fit on the selected volumes).
• ABR Application restore is used to restore the latest copy of individual data sets that were backed up by ABR Application Backup (FDRAPPL - Application Backup ). Since Application Backups are run as part of an application job stream, the data sets are restored to a point-in-time that is most useful to that application.

FDRDRP

FDRDRP is an optional, extra-cost feature of ABR that simplifies many of the procedures described in this section.

FDRDRP does full-volume recovery from ABR full-volume and incremental Volume Backups, just like ABR does (see above). While ABR does one DASD volume at a time in a given ABR job and may mount the required tapes many times, FDRDRP does many DASD volumes in parallel and mounts the tapes a minimum number of times, restoring data belonging to many DASD volumes in one pass of the tape. This can greatly reduce the time required to do full-volume restores.

FDRDRP is described in FDRDRP.

Choosing a recovery procedure

You may need to use several or perhaps all of these recovery procedures to rebuild your Data Center at the disaster site:

1.If your disaster site does not have a starter z/OS system, you must use SAR to restore the full-volume backups of the minimum volumes required to IPL your system. If you do have a starter system available, you can do FDR full-volume restores of those same volumes, and then IPL your system to complete the recovery (you could also restore all your ABR data on the starter system).

2.If your disaster site has sufficient DASD volumes of the proper type and size, you probably want to do ABR or FDRDRP full-volume recoveries for each DASD volume. This is the most efficient method of restoring those volumes to their exact contents as of the time of their last incremental backup. If the disaster site has DASD of different types or smaller DASD or a fewer number of larger DASD, you may not be able to use full-volume recovery for your entire installation.

3.If your disaster site has less capacity than your home site, or a smaller number of larger DASD, you need to do ABR data set recovery so that the restored data sets can be allocated on those DASD volumes. You can also selectively restore critical data sets if the disaster site does not have room for all of your data.

4.For applications that have implemented Application Backup, you probably want to use Application restore, since it restores the data sets to the point-in-time designated by the application (data sets restored with ABR full-volume or data set restore are restored to the time of the last incremental backup, chosen by the Data Center).

Required resources

The disaster recovery procedures assume that several resources were stored in some off-site storage location and are available at the disaster recovery site. Please review this list carefully to be sure that you are not missing some required item:

• Copies of all of your ABR Volume Backups (full-volume and incremental backups). These are usually the COPY2 backups produced by ABR using the TAPExxDD statements, but they may also be copies (COPY2 through COPY9) produced by the FDRTCOPY or FDRTSEL utilities (seeFDR and ABR Backup Maintenance). At least the most recent full-volume backup of each required volume must be in the off-site storage, plus as many of the subsequent incremental backups of those volumes as possible.
• A backup of a system residence volume (SYSRES) that is configured to run on the hardware of the disaster recovery site. You must generate an I/O configuration, with HCD or IOCP/MVSCP, which matches the configuration of the site. It is possible to put multiple configurations on the same SYSRES volume, so this might actually be your normal SYSRES volume. In some cases, multiple volumes are necessary to contain the configurations.
• A separate backup of the ABR catalog produced by FDRDSF or by ABR DUMP TYPE=DSF, after all daily ABR runs are complete (for safety, send two (2) copies of this backup off-site). Do not depend on the backup of the ABR catalog made by normal ABR incremental backups, since this is taken before all ABR backups have been recorded.
• A hard copy of the ABR catalog, produced by program FDRABRP, using the command PRINT CATLG (seeFDRABRP - Standard Reporting Facility). This serves as the ultimate backup if the ABR catalog backups are unusable, and can help you pre-stage the required tapes. If you must do SAR restores, it identifies the volumes containing the latest full-volume backups.
• A tape containing SAR (the Stand-Alone-Restore program of FDR). It is easiest to IPL SAR if the SAR tape is an unlabeled tape, but a labeled tape can also be used. A copy of the FDR distribution tape from BMC Corporation contains SAR as file 1. See SAR - Stand Alone Restorefor instructions on generating a SAR IPL tape.
• A tape containing an IEBCOPY unloaded copy of the ABR program library may be required if you are going to use an operating system provided by the disaster recovery site to do the restores, since they may not have a current copy of FDR or ABR available.
• The tape containing the daily FDREPORT extract file, if you are using the data set recovery procedure described later in this section.
• Separate backups of other special-purpose data, such as tape management system databases, using special utilities, may be required to successfully restore data with ABR; however, we suggest that you do not activate tape management or security systems until the ABR restores are done.
• ABR Application Backup tapes for all applications using ABR Application Backup.

The more frequently these resources are sent off-site, the more current the off-site information is, and so the more current your system is after restoring at the disaster recovery site. The expense and effort of sending data into off-site storage must be evaluated by each installation versus the impact of having back-level data after disaster recovery.

Full volume recovery

The following procedure may be used to recreate entire DASD volumes from the ABR Volume Backups. You can use this to recover your entire installation, to recover a critical subset of your volumes, or just to recover the volumes required to operate your system (the latter assumes that you recover critical data by other means, such as ABR data set restore or ABR Application Restore).

Volume recovery step 1

This creates an IPLable version of your operating system. You can skip this step if the disaster site provides you with a starter z/OS system adequate for ABR restores.

Do a Stand Alone Restore (SAR) of the system residence volume, any other volumes that you need in order to IPL (such as paging and spool volumes, and volumes containing LPA and link list (LNKLST) libraries) and the volume containing the ABR program library. The input to each Stand Alone Restore (SAR) is the most recent full-volume backup (TYPE=FDR) for that volume. This step does not necessarily restore the volumes to the most current level, but at this stage you are just trying to get a system you can IPL. Remember that the system residence volume you restore must be one that supports an I/O configuration that runs on the disaster recovery site hardware.

Volume recovery step 2

IPL the operating system (either the system you just restored, or the starter system provided by the site).

If the system you have IPL’d does not contain a current ABR program library, create that program library. You can reinstall the ABR library from the ABR distribution tape (following the instructions inInvoking the ABR Install ISPF Dialog) if you have an off-site copy of that tape, or you can restore an IEBCOPY unloaded copy of that library if you have sent that off-site. The ABR library must be APF authorized.

Volume recovery step 3

Next, the ABR catalog must be restored and made available to the system where you do the restores. Werecommend that you restore the ABR catalog to a spare volume, not the one where it normally resides, so that it is not overlaid by the full-volume restores inVolume recovery step 4.

If the master catalog of that system already contains the FDRABR alias or a catalog by the name of your catalog, it may be necessary to delete it with IDCAMS with this input:

DELETE (FDRABR,#) ALIASEXPORTcatalognameDISCONNECT

Now the separate FDRDSF or ABR backup of the ABR catalog can be restored, to some DASD volume that does not currently contain such a catalog. DSF allocates the catalog on that volume and does an IMPORT CONNECT for it in the master catalog of the running system. The aliases associated with the ABR catalog are automatically

defined. The JCL to do this restore looks something like this example:

//RESTCAT EXEC PGM=FDRDSF,REGION=0M//SYSPRINT DD SYSOUT=*//SYSUDUMP DD SYSOUT=*//TAPE1 DD DSN=catalog.backup.name,// VOL=SER=vol,UNIT=TAPE,DISP=OLD//SYSIN DD * RESTORE TYPE=DSF SELECT DSN=catn,NVOL=nvol /

Volume recovery step 4

Use the ABR full-volume restore function to reconstruct all of the required volumes at the most current level, using job steps similar to:

//RESTVOL EXEC PGM=FDRABR,REGION=0M//SYSPRINT DD SYSOUT=*//SYSUDUMP DD SYSOUT=*//SYSIN DD * RESTORE TYPE=FDR,CPYVOLID=YES,CONFMESS=NO, SMSPROT=NONE,DYNTAPE,ONLINE,MAXDD=500 SELECT VOL=PROD01,COPY=2,NVOL=SC3A54 SELECT VOL=PROD02,COPY=2,NVOL=SC3A55 SELECT VOL=PROD03,COPY=2,NVOL=SC3A56/*

The ABR full-volume restore function automatically recreates each volume exactly as it was at the time of the most recent incremental backup. It reads that most recent incremental backup first, restoring the VTOC and all other data on that backup, then reads backwards through the earlier incremental dumps until it reaches the full-volume backup that started the current generation, restoring only the data required from each backup. Unlike some competing products, in ABR, there is no need to separately “apply” incremental dumps; the entire restore is done by the above step.

The output volumes do NOT have to be preinitialized to the volume serials of your original volumes, nor do they have to be initialized for ABR. NVOL=nnnnnnspecifies the current volume serial of the DASD volume that is overlaid by the restore of your volume “vol”. CPYVOLID=YES causes that volume to be renamed to your volume serial after the restore completes. Many disaster recovery sites guarantee that their DASD are initialized to a certain set of volume serials before you arrive; you can then plan that each one of your volumes “vol” is to be restored to a selected one of their serials “nnnnnn”. The restore JCL must include a DISKxxxxDD statement specifying the output DASD volume serial number unless the ONLINE operand is specified on the RESTORE statement. Additionally, specify MAXDD= with a value sufficient for the number of volumes being restored if restoring more than the default of 256 volumes.

Change COPY=2 if COPY2 is not the copy you keep in your off-site storage.

The ABR Model DSCB on each volume to be restored is not available (since the DASD volumes have not yet been restored), so ABR locates the most recent incremental backup of the DASD volume in the ABR catalog that you have just restored. ABR then reads that incremental backup, followed by each preceding incremental backup, and finally the full-volume backup that begins the current generation. Once this process is completed, the volume looks exactly as it did at the time that the latest incremental backup was taken. If there are certain volumes for which you want to restore only the most recent full-volume backup, add “GEN=CURRENT,CYCLE=00” onto the SELECT statement.

If the backup tapes required for a given DASD volume are in use by another restore job, ABR waits for them to become available as long as you have theWTVOLoption enabled in the ABR option table (seeABR Options, panel A.I.4.4). Use of DYNTAPE is recommended, since without it abends may occur if the same tape is required by two concurrent restore jobs.

You can restore more than one volume in an ABR step (by specifying more than one SELECT statement), but those volumes are restored one at a time, so it is easier to simply restore one volume per step as shown above. Those steps can be organized one per job, or several steps per job. In order to be able to restore a large number of volumes in minimum time, you need to have multiple ABR restore jobs executing at once. Any number of concurrent ABR restores can be executing, limited only by the number of available tape drives. Performance is reduced if the capacity of the tape or DASD data paths is exceeded.

However, since incremental backups almost always put data from several DASD volumes on one tape, concurrent restore jobs often contend for the same tape, causing one or more to wait. One way to avoid this is to divide the restores into groups of DASD volumes, creating restore jobs that contain multiple ABR steps, each restoring one DASD volume. The DASD volumes in each restore group should be selected so that they are usually not requesting the same backup tapes required by another restore group; this can be done in several ways:

• The best way is to setup your normal ABR backup jobs to select DASD volumes by the same groups. In other words, there is a backup job that processes a given set of DASD volumes, and a restore job that restores those same volumes, one at a time. Since the backup tapes used for one volume group never overlap those used for another group, there is never tape contention.
• Another option is to simply divide the volumes to be restored into groups based on the order in which they are selected for processing by your normal ABR backup jobs. For example, the first 10 volumes that ABR normally selects might be restored by one restore job, the next 10 by the next restore, and the rest. This minimizes contention but cannot totally eliminate it.

FDRDRP can also be used to do these same full-volume restores. FDRDRP restores many DASD volumes concurrently and mounts the input tapes far fewer times than ABR. Although the operation for a single DASD volume is the same under FDRDRP and ABR, FDRDRP attempts to mount each tape only once and restore the required data from all backups on the same tape. With FDRDRP, most of the preceding considerations do not apply. However, it may still be useful to do your backups in groups of volumes and do the FDRDRP restores in those same groups, to minimize tape contention.In most cases, FDRDRP restores the same DASD volumes in far less time than FDRABR. Here is an FDRDRP example:

//RESTVOL EXEC PGM=FDRDRP,REGION=0M//SYSPRINT DD SYSOUT=*//SYSUDUMP DD SYSOUT=*//SYSIN DD * RESTORE TYPE=DRP,CPYVOLID=YES SELECT VOL=PROD01,COPY=2,NVOL=SC3A54 SELECT VOL=PROD02,COPY=2,NVOL=SC3A55 SELECT VOL=PROD03,COPY=2,NVOL=SC3A56/

Volume recovery step 5

If you usedVolume Recovery Step 1of this procedure, then you used SAR to restore the IPL volumes from ONLY the full-volume backup tapes, so some data sets on those volumes might be back-level. If this is a problem, then you must restore those volumes again with ABR. Since you are currently running your system from those volumes, the ABR restores must output to a different set of volumes; after the restores are complete, be sure that you place the original SAR-restored volumes offline and IPL using the new versions of those volumes (the IPL procedure allows you to choose between two identically labeled volumes).

However, if your IPL volumes contain only data that does not change frequently, it may be unnecessary to do these repeat restores.

Volume recovery step 6

If you did not restore the ABR catalog to a temporary volume as recommended inVolume recovery step 3, and are using it on its normal volume serial number, you must insure that the volume containing the ABR catalog is restored last. This restore cannot be run concurrently with the other restores because it may temporarily down-level the catalog.

A down-level copy of the ABR catalog was restored by the full-volume restores inVolume recovery step 4. When the full-volume restores are finished, repeat the restore of the ABR catalog that was done inVolume recovery step 3, to return the ABR catalog to the most current level. However, this time it should be restored to the volume where it normally resides. It is not necessary to delete the ABR catalog; the restore JCL inVolume recovery step 3simply overlays the existing catalog.

You have now completed the full-volume recovery procedure. If you are recovering all your data with full-volume recovery, you are done. If not, keep reading.

Data set recovery

The following procedure may be used to restore individual data sets or groups of data sets from ABR full and incremental backup tapes. It can be used when you do not want to restore all of the data sets on those backups, or when you must restore data sets to volumes other than their original volumes because:

• The recovery site has volumes of different sizes, such as 3390-3’s instead of your 3390-9’s.
• The recovery site has fewer volumes, requiring selective restores.

Data set restore at a recovery site requires that some additional information be saved as part of your daily backups. Normally, information about the backups of individual data sets is stored in the DSCB of the data set itself on DASD. The ABR catalog contains the locations of the backup data sets created by ABR, but does not record information about the individual data sets in those backups. For DASD volumes that have not been restored by the full-volume recovery process just described, ABR does not know where to find an individual data set.

However, if you tell ABR where to find a backup, specifying the DASD volume, generation and cycle, for example,

SELECT DSN=dsname,VOL=volser,GEN=generation,CYCLE=cycle

ABR can locate the backup and restore the data set.

FDREPORT, the ABR generalized report writer described inFDREPORT - Generalized Report Writer, can be invoked under FDRABR when you run your volume backups, to build a “database” that contains the location of the latest backup of every data set in your system (or any subset of them). At the recovery site, you can then run FDREPORT against that database to build SELECT statements as shown above to instruct ABR to restore selected data sets.

Build the Database

To create this database, modify your ABR volume backup steps by adding the operand “PRINT=RPT” to your ABR DUMP statements, and adding a $RPTSUT2 DD statement to define the database. As ABR finishes processing each DASD volume, it invokes FDREPORT internally to add to the database relevant information about the data sets that were backed up, including the ABR gen/cycle information necessary to restore them. Since you need this database at the disaster site, we recommend that its name begin with FDRABR (or whatever you use for the ABR index) so that it is in the ABR catalog. The database can be a GDG if you like and can be on tape to be sent off-site, or you can put it on DASD and back it up with the ABR catalogs after the backups are complete.

//BACKUP EXEC PGM=FDRABR,REGION=0M//SYSPRINT DD SYSOUT=*//SYSUDUMP DD SYSOUT=* .. other DD statements required by ABR ..//$RPTSUT2 DD DSN=FDRABR.OFFSITE.DATABASE,DISP=(,CATLG),// UNIT=TAPE,DCB=(BLKSIZE=32760,BUFNO=10)//SYSIN DD * DUMP TYPE=ABR,PRINT=RPT,…/*

Naturally, the database tape or backup should be sent to your off-site storage with your other backups.

Restoring data sets

At the recovery site, FDREPORT is used to read the database and generate SELECT statements for any data sets you wish to restore. These SELECT statements are passed to an ABR restore step that allocates the proper backup tapes and restore the data sets. The target volumes for the restores can be specified by NVOL= operands on the SELECT statements (as shown in the example below) or by the ABR RESTORE ALLOCATE LIST (seeDefine the ABR Protect Lists and Restore Allocation List). FDREPORT XSELECT statements can be used to select a subset of the data sets on the database so that multiple restore jobs can be run concurrently.

The user catalogs in which these data sets are cataloged must be restored before this job stream is run. The RECAT and VRECAT operands update those catalogs to point to the volumes to which the data sets were restored.

//SELREST EXEC PGM=FDREPORT,REGION=0M//SYSPRINT DD SYSOUT=*//SYSUDUMP DD SYSOUT=*//SYSUT2 DD DSN=FDRABR.OFFSITE.DATABASE,DISP=SHR//SYSPUNCH DD DSN=&&ABR,DISP=(,PASS),// UNIT=SYSALLDA,SPACE=(TRK,(5,2))//SYSIN DD * XSELECT XDSN=PAYROLL.** ** select all PAYROLL data sets PUNCH FDRLIB=MASK ** punch mask for ABR statements PRINT DATATYPE=EXTRACT,RPTYPE=SELPCH/*//MASK DD *)PREFIX RESTORE TYPE=ABR,DYNTAPE,RECAT,VRECAT,COPY=2,MAXCARDS=2000)ENDPREFIX SELECT DSN=<NAME>, VOL=<VOL>,GEN=<BKGEN>,CYCLE=<BKCYCLE>,NVOL=PROD*/*//ABRREST EXEC PGM=FDRABR,REGION=0M//SYSPRINT DD SYSOUT=*//SYSUDUMP DD SYSOUT=*//SYSIN DD DSN=&&ABR,DISP=(OLD,DELETE)

Even though many data sets may be requested from many different backup tapes, ABR sorts the requests so that each backup tape is mounted only once (if possible) and multiple data sets are restored from a given backup file in one pass of that file, even if the outputs are going to multiple DASD volumes.

Application restore

If applications that run at your home data center have implemented Application Backup (FDRAPPL - Application Backup) and a copy of the backup is off-site, then a restore those backups is desired. Application Backup allows the programmers to choose the point in their daily processing when to do the backups; usually this is a point where it is easy to restart the application. ABR Volume Backups are taken at a time chosen by the Data Center and may not meet the recovery requirements of some applications.

FDRAPPL - Application Backup has details on restoring from Application Backups, the details vary depending on options chosen by the application programmers.

If the same data sets have already been recovered by ABR full-volume recovery or ABR data set restore, the Application Restore simply overlays those data sets with the contents from the Application Backup tape.