Transcoder configuration

You can configure the Mainframe Connector transcoder by adding the required configuration in a JSON file. This file is referred to as the transcoder configuration file. You must define the configuration as specified in section Configuration. The qsam encode and qsam decode commands use the transcoder configuration file to perform data transcoding.

This page describes the various ways in which you can configure the Mainframe Connector transcoder.

Configuration

The Configuration object is the root of the transcoder configuration. It contains all the configuration options for the transcoder.

JSON representation 
{
    "defaults": object (DefaultsSection),
    "field_suffixes": object (FieldSuffix),
    "field_overrides": object (FieldOverride),
    "transformations": object (Transformation),
    "schema_validation_mode": enum (SchemaValidationMode)
}
Fields
defaults

object (DefaultsSection)

Specify default field modifiers for Cobol archetypes.
field_suffixes

object (FieldSuffix)

Specify field suffixes.
field_overrides

object (FieldOverride)

Specify field overrides.
transformations

object (Transformation)

Specify field transformations.
schema_validation_mode

enum (SchemaValidationMode)

Specify the schema validation mode.

DefaultsSection

The DefaultsSection object can be used to specify default modifications by cobol types. These are applied before any suffix or override modifications.

JSON representation 
{
    "alpha_numeric_display": object (FieldModifier),
    "numeric_display": object (FieldModifier),
    "binary": object (FieldModifier),
    "packed_decimal": object (FieldModifier),
    "national": object (FieldModifier),
    "utf8": object (FieldModifier),
    "dbcs": object (FieldModifier),
    "hexadecimal_floating_point": object (FieldModifier)
}
Fields
alpha_numeric_display

object (FieldModifier)

Specify defaults for alphanumeric (PIC X) fields.
numeric_display

object (FieldModifier)

Specify defaults for numeric display (zoned decimal) fields.
binary

object (FieldModifier)

Specify defaults for binary number (COMP) fields.
packed_decimal

object (FieldModifier)

Specify defaults for packed decimal (COMP-3) fields.
national

object (FieldModifier)

Specify defaults for national (PIC N) fields.
utf8

object (FieldModifier)

Specify defaults for UTF-8 (PIC U) fields.
dbcs

object (FieldModifier)

Default for dbcs (DISPLAY-1) fields.
hexadecimal_floating_point

object (FieldModifier)

Default for hexadecimal floating point (COMP-1, COMP-2) fields.

FieldSuffix

Field suffixes apply to all fields that have a suffix.

Fields are matched if they end with a hyphen (-) or underscore (_) followed by the suffix.

Suffixes are case-insensitive.

The FieldSuffix modifier is applied after the FieldOverride modifier.

For example, the modifier defined for the suffix NID will be applied to the field named FLD-NID but not for the field FUNID.

JSON representation 
{
    "suffix": string,
    "is_inverse": boolean,
    "modifier": object (FieldModifier)
}
Fields
suffix

string

The field with this suffix will have the modifier applied to it.
is_inverse

boolean

Specify whether the modifier is an inverse field modifier or not. An inverse field modifier applies the modifier on another field that has the same name as the field with the modifier without the modifier. For example, if both FLD-NID and FLD fields exist in the same record, the modifier will be applied to FLD.

When using an inverse field modifier, the special identifier $self can be used whenever a field name can traditionally be used to refer to the field with the suffix.

For example, to create a null-indicator field, you can use the null_if field modifier with is_inverse set to true. Refer to NullIf for more information.
modifier

object (FieldModifier)

Specify the modifier to apply to matching fields.

FieldOverride

Override or modify the decode and encode chain for the specified field.

JSON representation 
{
    "field": string,
    "modifier": object (FieldModifier)
}
Fields
field

string

Specify the name of the field to apply the modifier to.
modifier

object (FieldModifier)

Specify the modifier to apply to matching field.

Transformation

View transformations are used to modify the relationship between the table and the QSAM file. Transformations are always phrased from the point of view of the data. The concept is similar to view tables in BigQuery.

JSON representation 
{
    "exclude": object (Exclude),
    "unnest": object (Unnest),
    "move": object (Move),
    "rename": object (Rename)
}
Fields
exclude

object (Exclude)
unnest

object (Unnest)
move

object (Move)
rename

object (Rename)

FieldModifier

A field modifier lets you modify the encoding or decoding of a specific field. Note that not all modifiers can be applied to all fields. See the documentation for the specific modifiers for more information.

JSON representation 
{
    "filler": object (Filler),
    "null_if": object (NullIf),
    "format_date": object (FormatDate),
    "chain": object (ModifierChain),
    "zoned_decimal": object (ZonedDecimal),
    "binary": object (Binary),
    "packed_decimal": object (PackedDecimal),
    "null_if_invalid": object (NullIfInvalid),
    "bytes": object (Bytes),
    "varlen": object (VarLen),
    "string": object (String),
    "null_if_empty": object (NullIfEmpty),
    "format_timestamp": object (FormatTimestamp),
    "hfp": object (HFP)
}
Fields
filler

object (Filler)

Ignore and exclude the field.
null_if

object (NullIf)

Set the value of the underlying field to null on a condition.
format_date

object (FormatDate)

Format a string field as date.
chain

object (ModifierChain)

Chain together multiple modifiers.
zoned_decimal

object (ZonedDecimal)

Override zoned field decimal configuration.
binary

object (Binary)

Override binary field configuration.
packed_decimal

object (PackedDecimal)

Override packed decimal field configuration.
null_if_invalid

object (NullIfInvalid)

Set the field to null instead of spilling the line on error.
bytes

object (Bytes)

Override bytes field.
varlen

object (VarLen)

Set the record as a variable length field.
string

object (String)

Override string field configuration.
null_if_empty

object (NullIfEmpty)

Set the value of the underlying field to null if the field is empty.
format_timestamp

object (FormatTimestamp)

Format a string field as timestamp.
hfp

object (HFP)

Set this field to a Hexadecimal Floating-Point.

Exclude

Exclude a field from the resulting table, but still undergo decoding or encoding. This is useful when the field doesn't need to be transferred to the table, but is required for transcoding. For example, null indicator or length fields can be omitted from the table.

To bypass transcoding altogether, apply the filler modifier.

JSON representation 
{
    "field": string
}
Fields
field

string

Specify the field to exclude.

Unnest

Unnest the field.

JSON representation 
{
    "field": string,
    "format": string
}
Fields
field

string

Specify the field to unnest
format

string

Specify the new field format.

The ${parent} will be released with the name of the field being unnested.

For unnested structs, ${field} is replaced with the name of the structs field.

For unnested arrays and lists, ${index} is replaced with the indexes of the array.

Move

Move a field in the record.

JSON representation 
{
    "field": string,
    "offset": int
}
Fields
field

string

Specify the field to move.
offset

int

Specify the number of places, forward or backwards, the field needs to be moved to.

Rename

Rename one or more fields based on a regular expression match.

For example, to replace all hyphens with underscores, use the following JSON format: {"find": "\\-", "replace":"_"}.

JSON representation 
{
    "find": string,
    "replace": string
}
Fields
find

string

Specifies a Java regular expression pattern to identify the field(s) to rename.

The pattern is matched against the full field name. If the pattern matches any part of the field name, the field is considered a match.

Examples:

  • "\\-" (matches any field containing a hyphen)
  • "^field_name$" (matches fields with the exact name field_name)
  • "^field_(.*)$" (matches any fields starting with field_ and captures the rest)
  • "part_of_name" (matches any field containing part_of_name)
replace

string

Specifies the new name for the matched field(s).

Capture groups from the find regular expression can be used in the replace string using backreferences like $1, $2. This allows for more complex transformations based on parts of the original field name.

Examples:

  • "new_field_name" (replaces the field with a fixed name)
  • "new_$1" (uses the first capture group from find)
  • "${1}_new" (alternative syntax for capture groups)
  • "prefix_$1_suffix" (uses a capture group and adds prefixes/suffixes)

Filler

Specifies that a field won't be decoded or encoded. Additionally, it will also be excluded from the resulting table during the decoding process. You can apply this modifier to any field with a known size.

Provide an empty JSON object as follows:

JSON representation 
{
}

NullIf

Set a field to null if a condition is satisfied. You must specify either null_value or non_null_value or both.

To create a null-indicator field, you can use a FieldSuffix with a null_if field modifier, and set is_inverse to true as shown in the following examples:

Example: Null-indicator

To create a null-indicator field, we can use the null_if field modifier like so. 

 {
  "field_suffixes": [
   {
     "suffix": "NID",
     "is_inverse": true,
     "modifier": {
     "null_if": {
       "null_value": "?",
       "target_field": "$self"
     }
    }
   }
  ]
 }

This allows for all fields with the suffix NID to effectively be null indicators as shown in the following copybook snippet: 

 01 REC.
   02 FIELD     PIC X(10).
   02 FIELD-NID PIC X(1).

Example: Binary null-indicator

To create a binary null-indicator field, we can use the binary and null_if field modifiers like so. 

 {
  "field_suffixes": [
   {
     "suffix": "NID",
     "modifier": {
       "binary": {}
     }
   },
   {
     "suffix": "NID",
     "is_inverse": true,
     "modifier": {
     "null_if": {
       "null_value": "15",
       "target_field": "$self"
     }
    }
   }
  ]
 }

This allows for all fields with the suffix NID to effectively be binary null indicators using the same copybook from the earlier example.

Example: Bytes null-indicator

To create a bytes null-indicator field, we can use the bytes and null_if field modifiers like so. The values for null and not-null are expressed as HEX. 

 {
  "field_suffixes": [
   {
     "suffix": "NID",
     "modifier": {
       "bytes": {}
     }
   },
   {
     "suffix": "NID",
     "is_inverse": true,
     "modifier": {
     "null_if": {
       "null_value": "FF",
       "target_field": "$self"
     }
    }
   }
  ]
 }

This allows for all fields with the suffix NID to effectively be a bytes null indicator using the same copybook from the earlier example.

JSON representation 
{
    "target_field": string,
    "null_value": string,
    "non_null_value": string
}
Fields
target_field

string

Specify the field whose value you want to check. The field must be in scope.
null_value

string

When specified, if target_field is equal to this value, the field is not decoded or encoded and the value is set to null.
non_null_value

string

When specified, if target_field is not equal to this value, the field is not decoded or encoded and the value is set to null.

FormatDate

Format a string to a date using one of the supported formats. You can only apply this modifier to sized fields. During the decoding process, the formats are tested in order until one of the formats matches the string. During the encoding process, the first format is used and the rest are ignored.

JSON representation 
{
    "formats": object (DateTimeFormat)
}
Fields
formats

object (DateTimeFormat)

List of date formats.

ModifierChain

Specify a modifier chain to apply multiple modifiers in series. The modifiers are applied in the order they are specified.

JSON representation 
{
    "modifiers": object (FieldModifier)
}
Fields
modifiers

object (FieldModifier)

Specify the list of modifiers to apply.

ZonedDecimal

Sets various options related to the encoding and decoding of zoned decimals. You can only be apply this modifier on a decimal field.

JSON representation 
{
    "logical_type": enum (DecimalLogicalType),
    "encoding": enum (ZonedDecimalEncoding)
}
Fields
logical_type

enum (DecimalLogicalType)

Specify the logical type to use when decoding or encoding the field.
encoding

enum (ZonedDecimalEncoding)

The encoding with which the field is encoded.

Binary

Ignore any previous modifiers and treat this field as a binary number.

JSON representation 
{
    "signedness": enum (BinarySignedness)
}
Fields
signedness

enum (BinarySignedness)

The signedness of the number.

PackedDecimal

Set this field to a PackedDecimal.

JSON representation 
{
    "logical_type": enum (DecimalLogicalType)
}
Fields
logical_type

enum (DecimalLogicalType)

Override the logical type. By default, Mainframe Connector uses the optimal logical type based on the precision and scale.

NullIfInvalid

Treat the value as null if transcoding fails. You can only apply this modifier to sized fields. The error is ignored and is not logged in the spillover dataset. During the decoding process, the value of this field will be null for this record. During the encoding process, if the data can't be written, the entire field with be filled with null bytes.

Provide an empty JSON object as follows:

JSON representation 
{
}

Bytes

Ignore the modifier chain and treat the data as raw bytes. You can be apply this modifier to any sized field.

Provide an empty JSON object as follows:

JSON representation 
{
}

VarLen

Represents a variable-length field.

A variable-length field contains three parts:

  1. A group item that contains two subfields.
  2. A field within the group item that contains the length of the transaction data.
  3. A field within the group item that contains the data.

The name of the variable-length field will be the group name.

Provide an empty JSON object as follows:

JSON representation 
{
}

String

Sets the various options related to string decoding and encoding. Can only be applied on a string field.

JSON representation 
{
    "encoding": string,
    "trim_suffix": boolean,
    "pad_char": string
}
Fields
encoding

string

The encoding with which the field is encoded.
trim_suffix

boolean

When set to true, any whitespace at the end of the string will be trimmed. trim_suffix affects only decoding, encoding ignores trim_suffix. Note that strings that consist only of whitespaces will become empty strings.
pad_char

string

When set padding export strings with pad_char. If set, the length of pad_char must be 1. pad_char affects only encoding, decoding ignores pad_char.

NullIfEmpty

Field should be set to null if all the bytes in that field are 0.

Provide an empty JSON object as follows:

JSON representation 
{
}

FormatTimestamp

Format a string to a timestamp using one of the provided formats. This can only be applied to sized fields. During decode, the formats are tested in order until one of the formats matches the string. During encode, the first format will be used and the rest will be ignored.

JSON representation 
{
    "formats": object (DateTimeFormat)
}
Fields
formats

object (DateTimeFormat)

List of timestamp formats.

HFP

Set this field as Hexadecimal Floating-Point.

Provide an empty JSON object as follows:

JSON representation 
{
}

DateTimeFormat

Size and pattern to use when converting the field to a date.

JSON representation 
{
    "size": int,
    "pattern": string
}
Fields
size

int

Specify the size of the field this pattern applies to.
pattern

string

Date formatter pattern. For more information on valid formatter patterns, see Class DateTimeFormatter.

SchemaValidationMode

The schema validation mode during copybook compilation. This mode ensures compatibility with a specific target data format.

Enums
DEFAULT Default schema validation mode. This mode ensures unique field names within the copybook.
BIG_QUERY Schema validation mode for BigQuery compatibility. This mode extends the default validation to ensure the copybook's schema is compatible with BigQuery's data types.

ZonedDecimalEncoding

Specify the encoding to use when decoding or encoding a zoned decimal field.

Enums
UNSPECIFIED Keep the encoding that was specified in the modifier chain. If no modifier was specified, EBCDIC is used.
EBCDIC Use EBCDIC encoding.
ASCII Use ASCII encoding.

DecimalLogicalType

Logical type to use for a decimal field.

Enums
AUTO Use the most optimal type based on the scale and precision.
LONG Use 64-bits to store the value. This modifier only works for numbers whose precision is less than or equal to 18, and the scale is 0.
DECIMAL64 Use 64-bits to store the value. This modifier only works for numbers whose precision is less than or equal to 18.
BIG_DECIMAL Store the value as an unbounded decimal value. This is the slowest option but supports any decimal of any precision at any scale.
BIG_INTEGER Store the value as an unbounded integer value. This is the slowest option but supports any integer of any precision.

BinarySignedness

Logical type to use for a decimal field.

Enums
UNSPECIFIED Use the most optimal type based on the scale and precision.
SIGNED Use 64-bits to store the value. This modifier only works for numbers whose precision is less than or equal to 18, and the scale is 0.
UNSIGNED Use 64-bits to store the value. This modifier only works for numbers whose precision is less than or equal to 18.