Skip to main content

Pentaho+ documentation has moved!

The new product documentation portal is here. Check it out now at docs.hitachivantara.com

 

Hitachi Vantara Lumada and Pentaho Documentation

Copybook Input

Parent article

The Copybook Input step reads binary data files that are mapped by a fixed-length COBOL copybook definition file. COBOL definition and binary files are used in IT scenarios that include data stored on mainframes. You can extract the binary data files and the definition files from the mainframe for data transformation and analysis, and avoid using mainframe cycles for complex data analysis tasks.

NoteThe Copybook Input step performs self-contained static extraction of the data in the binary format. If you only need to perform ETL metadata injection, use the Read Metadata from Copybook step. You are not required to use both copybook steps in the same transformation. For more information see Copybooks in PDI.

Before you begin

Review these prerequisites before using the Copybook Input step.
  • For PDI to process binary data files, you must first download both the copybook definition file and the binary data files from the mainframe environment. For example, you can use FTP or an SFTP server to download the files to a staging area accessible from PDI. You can also use a SFTP VFS path to connect to and read data directly from the mainframe at runtime.
  • The binary data file must remain in binary format when used as input to this step. If you are using FTP to download the files, ensure that the data file is not converted to ASCII.
  • This step works with Fixed Length COBOL records only. Variable record types such as VB, VBS, OCCURS DEPENDING ON are not supported.
  • Your mainframe administrator can provide more details about the environment-specific copybook file definitions and structures this step requires for reading binary data.

General

Enter the following information in the transformation step field.
  • Step name: Specifies the unique name of the Copybook Input step on the canvas. You can customize the name or leave it as the default.

Options

The Copybook Input step includes settings for locating the files to read, a table to define data fields on the PDI output stream, and additional options.

Input tab

Input tab, Copybook Input step

The Input tab has the following sections.

Source

These options specify the location of the binary data

.
OptionDescription
Predefined fileSelect this option to specify a path to a binary data file that contains the data you want to read into the PDI stream. You can type any VFS path directly into the File field, including any variables, or you can click the Browse button to locate the binary data file.
File defined in a fieldSelect this option to read the names of the binary files from a field name in the previous step. Select the name of the field from the drop-down list.
Data already loaded in a binary fieldSelect this option if the binary data is passed into the step from a binary field on the PDI stream. Select the step generating the binary field from the drop-down list. You can use this option to prepare the output of records by another Copybook Input step. Using this method, you can selectively process fields and avoid conversion errors in definition files that include REDEFINES. See the Store record as a binary field option in the Options tab.
Schema

These options define the location of the copybook definition file and include mapping options for the binary data files.

OptionDescription
COBOL Copybook file pathSpecify the file path to the copybook definition file. You can enter any VFS or SFTP file path or click Browse to open the system file browser. After selecting a file, click Validate to verify that the definition file can be accessed and parsed.
COBOL Copybook line structureSpecify the line structure of the definition file.
  • Standard columns (6 to 72)

    Select this option when the definition file contains line numbers. The first 6 columns of text from each line are ignored. Any data beyond column 72 is ignored.

  • Full line

    Select this option when the definition file does not contain line numbers.

Binary Format

Use these options to describe the binary format of the selected file.

OptionDescription
Source architectureSelect the machine architecture of the binary data source files. The values are:
  • Big endian (mainframe)

    The most significant byte first and the least significant byte last.

  • Little endian

    The least significant byte first and the most significant byte last.

Source charset nameSelect the character encoding set of the binary data file.

Mainframe EBCDIC is typically encoded using IBM037 or cp1047 character sets.

For more information about character sets and their aliases, see Supported Encodings in the Oracle® documentation.

Packed decimal (COMP-3) sign conventionSelect how COMP-3 Packed decimals are parsed from the binary data as it relates to sign convention. For a given field, if validation occurs and fails, a conversion error will occur at runtime. See Use Error Handling for details.
  • Strict

    Must follow the IBM S370FPD specification to avoid validation errors. Validation is performed to verify that all nibbles (half-bytes), except the sign nibble, are decimal digits (0-9). This is the default value.

    • For signed packed decimals, the sign nibble must be C (positive) or D (negative).
    • For unsigned packed decimals, the sign nibble must be F.
  • Lenient

    Validation is performed to verify that all nibbles contain decimal digits and the sign nibble contains a hexadecimal value of A-F. The sign nibble is only used to interpret a negative number if the value is D.

  • Lenient - unchecked

    No validation is performed on the source bytes. The sign nibble may contain any hexadecimal value 0-F, and the last nibble is not included in the result. The sign nibble is only used to interpret a negative number if the value is D.

Output tab

The table in the Output tab provides details of the fields that are read from the binary data and how those fields are placed in the PDI stream as output from the step. Output tab, Copybook Input step

You can populate the table using either the Get Fields or Get Fields with Parent Groups commands. These commands extract directly from the copybook definition file selected in the Input tab. Use the Get Fields with Parent Groups command if you want to include the higher-level organizational data from the copybook definition file that omits the PICTURE clauses.

To pass the input fields to the PDI output stream, select the Pass through input fields check box. Clear this check box to omit the input fields from the PDI output stream.

NoteThis option is only available if you chose either the File defined in a field option or the Data already loaded in a binary field option, located in the Source section of this step's Input tab.

The output table contains the following columns.

ColumnDescription
NameThe name of the field in the PDI output stream. You can revise or update this field name as necessary.
PathThe fully qualified path to the binary data column in the copybook definition file.
NoteThis field cannot be edited. It is controlled by the copybook definition file.
Dest TypeThe PDI data type of the column mapped from the column definition.
NoteThis field cannot be edited. It is controlled by the copybook definition file.
  • X becomes String
  • COMP, COMP-3, and DISPLAY become BigNumber
  • COMP-1 and COMP-2 become Number
  • Parent groups become Binary
ConversionThis data type is read from the source data for a given field.
  • Dest Type

    (Default) The field are cast to the PDI type in the Dest Type column.

  • String

    The column is converted to the Dest Type, then cast into a string.

  • Hex String

    The bytes of the underlying data are converted to a hex string. The output field is a string data type.

  • Binary

    The bytes of the underlying data of the column are not converted, but are instead extracted and placed on the stream as a binary type.

Options tab

Options tab, Copybook Input step

Use this tab to define PDI output stream options.

Output record information

Use this section to specify details about the records for output.

OptionDescription
Store record as a binary fieldSpecify an additional output field to contain the binary bytes that make up the record currently being processed. You can use the stored binary field as the input for the data fields downstream from the Copybook Input step.
Create field with record numberSpecify an additional output field to contain the record number within the file. For fixed-length record definitions, multiplying this number by the fixed record size yields the offset of the record within the input file. This field will reset to zero when a new data file is read. Also, the counter is specific to a copy of the step, so changes to the Change Number of Copies to Start option may cause unexpected results.
Create field with record checksumSpecify an additional output field to contain a hex string representation of the sha1 checksum of the source record byte data.
NoteThis option is useful for debugging conversion errors, but it could be resource intensive.
Conversion errors

Use this section to specify how to handle errors during conversion.

  • Ignore conversion errors

    Select this check box to log multiple conversion error messages such as malformed records, bad enclosure strings, wrong number of fields, and premature line ends. The errors are logged in JSON object format in a single PDI row. See Use Error Handling for details about the format.

    Clear this check box if you want conversion errors in the source binary files to stop the transformation.

Use Error Handling

The JSON object is placed in the error description field within the error row. Error handling must be enabled on the step to capture the error columns and descriptions. On the canvas, right-click the Copybook Input step and select Error Handling to open the Step error handling settings window and configure the error output column names. See the Use the Transformation menu for details.

The following table details the JSON object format for the output error stream.

KeyTypeExampleDescription
recordInteger0The record number originating from the Copybook Input step.
converterStringBigNumberColumnConverterThe converter class that originated the error.
exceptionStringRecordExceptionThe exception class of the error.
messageStringInvalid sign in field: OPEN-YEAR The error message text, if it exists.
fieldNameStringOPEN-YEARThe name of field that has the error.
positionInteger22The position of the field.
lengthInteger3The length of the field.
valueString404040The value read as a hexadecimal string.
recordHashString240c992c3aaebccf6dc0e99a4ed1a447e4811bedThe record checksum originating from the Copybook Input step.

Metadata injection support

All fields of this step support metadata injection. You can use this step with ETL metadata injection to pass metadata to your transformation at runtime.

Use the Read metadata from Copybook step to read copybook definition files and obtain the required mapping information to inject fields. In addition to the Name, Path, Dest Type, and Conversion, a decimal precision must be provided.