Eloquence MPE migration utilities --------------------------------- Revision: B.07.10 (2008-04-23) This archive provides an updated version of the Eloquence MPE data migration tools. As of the Eloquence B.07.10 release, dbexport supports creating binary export files in addition to the text based export files. To create bexp files dbexport needs to run in privileged mode and must be copied to a privileged group after compiling. Also the binary capabilities must be changed with linkedit. Limitations: - No support for Business Basic decimals - Export files are limited in size to 2GB (workaround available) - Binary export mode includes the IMAGE record number and chain pointers along with a binary representation of the record. Any broken chains will show up in Eloquence. - Binary export mode depends on the data set entry count. Data base inconsistencies may result in failures during export or may miss extra records. Changes to version 2006-12-07: - fixed a problem with text mode export and standalone details when using option -S - fixed info 406 interpretation. ROOT file version C4 supports record numbers - fixed a corner case problem converting record labels to record numbers - fixed problem with the -x option when db opened in write mode Changes to version 2005-03-30: - dbexport did not allow for more than 199 data sets - changed output format to support 9 digits for capacity/entries Changes to beta version 2005-03-29: - Fixes a compile error with hp3kinfo.c - Changed Makefile.MPE to disable warnings until we figure out how to disable them selectivly - Changed bimport.sh example script to concatenate bexp file fragements Changes to beta version 2005-03-22: - Document workaround to overcome the 2 GB file size limitation with text mode export - Document workaround to impove import performance for sorted chains with text mode export - Document workaround for binary export mode with data sets beyond 2 GB - Added option -S to prefer sort item over primary path in chained export mode - Added enhancement to support specifying the search item in chained export mode (option -c or -S specified) How to compile -------------- Transfer the files to MPE, start a POSIX shell and run make. As a result the file DBEXPORT and DBINFO are built. How to install -------------- If the new bexp format is not required the executables may be installed in an arbitrary location. To use the new bexp format, the files should be installed in a privilged group and the DBEXPORT capbilities need to be changed as shown below: :linkedit altprog DBEXPORT; cap = ia,ba,pm exit This adds the PM capability to DBEXPORT. How to use the binary export mode --------------------------------- The binary export mode is used when the -B (upper case B) command line option is specified. The new binary export mode creates a separate file for each dataset (unless it has zero entries). DBEXPORT uses the naming convention DBNAME.999.exb for each binary export file. The binary export file includes a header to identify the data in the file and for each record the original record number and the chain pointers (besides the sysnonym chain information for a master set). This information may then be used by the Eloquence bimport to efficiently import the database into Eloquence, retaining the chain order and record number for any detail set. How to use bimport on Eloquence ------------------------------- The new bimport function allows to efficiently import data from TurboIMAGE while retaining the chain information. For a detail set the record number is also retained. While undocumented, the bimport function is included for testing purposes in the most recent Eloquence B.07.00 eloqdb patches (PE70-0501070 or newer) for the HP-UX platform. Currently, bimport is not a convential utility program but builtin the server process and run with the dbctl utility. dbctl bimport [options] database set bexp_file This first erases the specified data set and then processes the specified import file. As the bimnary import files are directly accessed by the eloqdb6 server process the files must be readable by the user that is configured for the eloqdb6 and must reside on the same system. The following options are supported with bimport: /v - Display additional information. Specifying the option /v twice (/v /v) causes the import progress to be output to stderr for every 10,000 records. /a - In case more than one bexp file is needed to import a data set this flag is specified with any subsequent bexp file. /n - This file must be specified if additional bexp files exist for this data set. As the current MPE dbexport only creates a single binary export file for each data set, the options /a and /n should not be used. Please note that PE70-0501070 has a small bug bug that causes a hang when the bimport /v option is used and EnableIPC=0 is configured. An example script (bimport.sh) is provided that may be used to automatically import a single database. DBEXPORT enhancements --------------------- New -S option ------------- The new option -S (upper case S) may be used to specify chained export mode and that a sort item has preference over the primrary path. The option -S implies options -c (chained export mode). Specifying the search item in chained export mode ------------------------------------------------- When using chained export mode (option -c or -S present) the search item for a detail set may be specified to override the default of using the primary path. The (detail) search item number may be specified with the dataset, seperated by a dot. The examples below all specifiy to export the data set PO-RECEIPTS (set number 39) of the data base IOSDB, using chained export and the search item PART-NUMBER (item number 159). ./DBEXPORT -c IOSDB.QEDDB 39 ./DBEXPORT -c IOSDB.QEDDB 39.PART-NUMBER ./DBEXPORT -c IOSDB.QEDDB 39.129 ./DBEXPORT -c IOSDB.QEDDB PO-RECEIPTS.159 ./DBEXPORT -c IOSDB.QEDDB PO-RECEIPTS.PART-NUMBER Binary export mode was enhanced to split bexp files --------------------------------------------------- When using binary export mode (-B option present), a new export file is created for each GB of data. In this case no .exb file but multiple partial files are created that need to be combined to run bimport. This allows binary exporting data sets up to a size of 100 GB with little extra effort by concatenating the files. See below (workaround #3) for an example. Please note that this is different from the bexport option to use multiple input files files (as specified with the options /v and /a). For bimport to accept multiple files for a data set each file must provide a valid BEXP header which is not supported by DBEXPORT at this time. DBEXPORT workarounds -------------------- Workaround #1: To overcome the filesize 2GB limitation with text mode export files you may pipe the output to split. DBEXPORT -s - database | split -b 1000m - exp. This will split the export file into 1000 MB chunks named exp.aa, exp.ab ... On the target system you may combine the files with a command like below: cat exp.* > database.exp You may also pass the result to dbimport though stdin: cat exp.* | dbimport -s - database Workaround #2: Sorted chains wre a performance problem in text mode. It might help to export the database in sorted order. The -S options specifies to use chained export mode giving preference to any sort items. dbexport -S database This attempts to retain the chain order for the first sort item which should make the chain rebuild on the target platform less painful. Using the -x option (exclude) you may also export the database in two steps. In the first step, all "normal" data sets are processed, exclusing any sets that would benefit from different options. In the second step any data set is exported specifying specific options. dbexport -c -x database 5 This exports the entire database except set #5 dbexport -c database 5:CUSTNO This uses the chain associated with the item CUSTNO to export set #5. Workaround #3: Binary export files are splitted into multiple segments for each GB of data. In this case no .exb file but multiple partial files are created that need to be combined on the target platform to run bimport. For example, exporting set #5 could create the following partial binary export files: SAMPLE.005.b00 SAMPLE.005.b01 SAMPLE.005.b02 ... The cat command may be used like below to combine these partial binary export files into a complete export file. cat SAMPLE.005.b* > SAMPLE.005.exp While bimport has the option to process partial files (each with a complete bexp header) this approach is currently not supported by DBEXPORT.