Wrangler command-line directives

This page describes the directives you can use in Wrangler.

In Wrangler Power mode (the Wrangler CLI), you can add directives and functions to a recipe. For more information, see Wrangler CLI Directives and Wrangler Functions.

Wrangler supports the following directives:

Column operations

Directive Description
Change Column Case Changes column names to either lowercase or uppercase.
Changing Case Changes the case of column values.
Cleanse Column Names Sanitizes column names, following specific rules.
Columns Replace Alters column names in bulk.
Copy Copies values from a source column into a destination column.
Drop Column Drops a column in a record.
Fill Null or Empty If the value is null or empty, enters a fixed column value.
Keep Keeps specified columns from the record.
Merge Merges two columns by inserting a third column.
Rename Renames an existing column in the record.
Set Headers Sets the names of columns, in the order they're specified.
Split to Columns Splits a column based on a separator into multiple columns.
Swap Swaps column names of two columns.
Set Type Converts the data type of a column.

Date transformations

Directive Description
Diff Date Calculates the difference between two dates.
Format Date Custom patterns for date-time formatting.
Format UNIX Timestamp Formats a UNIX timestamp as a date.

Datetime transformations

Directive Description
Current Datetime Generates current datetime with a given time zone.
Datetime to Timestamp Creates a timestamp from datetime and given time zone.
Format Datetime Formats datetime to a string of a given format.
Timestamp to Datetime Converts a timestamp to datetime.

Encoders and decoders

Directive Description
Decode Decodes a column value as one of base32, base64, or hex.
Encode Encodes a column value as one of base32, base64, or hex.

Hashing and masking

Directive Description
Hash Generates a message digest.
Mask Number Applies substitution masking on the column values.
Mask Shuffle Applies shuffle masking on the column values.

Lookups

Directive Description
Catalog Lookup Static catalog lookup of ICD-9, ICD-10-2016, and ICD-10-2017 codes.
Table Lookup Performs lookups into Table datasets.

Natural language processing

Directive Description
Stemming Tokenized Words Applies the Porter stemming algorithm for English words.

Output Formatters

Directive Description
Write as CSV Converts a record into CSV format.
Write as JSON Map Converts the record into a JSON map.
Write JSON Object Composes a JSON object based on the fields specified.
Format as Currency Formats a number as currency as specified by locale.

Parsers

Directive Description
JSON Path Uses a DSL (a JSON path expression) for parsing JSON records.
Parse as AVRO File Parses an AVRO data file.
Parse as CSV Parses an input record as comma-separated values
Parse as Currency Parses a currency value that is a string representation of local currency into a number.
Parse as Datetime Parses string as datetime data type with a given format.
Parse as Excel Parses into a Microsoft Excel file.
Parse as Fixed Length Parses as a fixed length record with specified widths.
Parse as HL7 Parses Health Level 7 Version 2 (HL7 V2) messages.
Parse as JSON Parses a JSON object.
Parse as Log Parses access log files, such as Apache httpd and NGINX servers.
Parse as Simple Date Parses date strings.
Parse as Timestamp Parses column values representing unix timestamp as date.
Parse XML To JSON Parses an XML document into a JSON structure.

Row operations

Directive Description
Filter Rows Filters records based on a condition.
Filter Row if Matched Filters rows that match a pattern for a column.
Flatten Separates the elements in a repeated field.
Fail on condition Fails processing when the condition is evaluated to true.
Send to Error Filters records to an error collector.
Send to Error and Continue Processing Filters records to an error collector and continues processing.
Split to Rows Splits based on a separator into multiple records.

Transformations

Directive Description
Changing Case Changes the case of column values to uppercase or lowercase characters.
Create Record Creates a record column with nested values by copying values from source columns into a destination column.
Cut Character Selects parts of a string value.
Set Column Sets the column value to the result of an expression execution.
Find and Replace Transforms string column values using a "sed"-like expression.
Quantization Applies quantization to column values.
Extract Regex Groups Extracts the data from a group of regular expressions into its own column.
Set Charset Sets the encoding and then converts the data to a UTF-8 String.
Set Record Delim Sets the record delimiter.
Split Email Splits an email ID into an account and its domain.
Split URL Splits a URL into its constituents.
Text Distance (Fuzzy String Match) Measures the difference between two sequences of characters.
Text Metric (Fuzzy String Match) Measures the difference between two sequences of characters.
URL Decode Decodes from the application/x-www-form-urlencoded MIME format.
URL Encode Encodes to the application/x-www-form-urlencoded MIME format.
Trimming Spaces Functions for trimming white spaces around string data.

Transient aggregators and setters

Directive Description
Increment Variable Increments a transient variable with a record of processing.
Set Variable Sets a transient variable with a record of processing.

Unique ID

Directive Description
Generate UUID Generates a universally unique identifier (UUID).

What's next