Whistle basics

This page describes the basics of writing a mapping in Whistle to transform data from one schema to another.

For a list of built-in Whistle functions, see the Whistle built-in function reference, and for a list of MDE-specific functions, see the Whistle MDE function reference.

Simple mapping

The following section outlines some array outputs.

Add array outputs

If you use the following Whistle code sample:

Planet[0]: "Earth";
Planet[1]: "Mars";
Planet[2]: "Jupiter";
Moon[0]: "Luna";

The output would be the following:

{
  "Moon": [
    "Luna"
  ],
  "Planet": [
    "Earth",
    "Mars",
    "Jupiter"
  ]
}

To add another moon, Io, before Luna, edit your Whistle code as follows:

Planet[0]: "Earth";
Planet[1]: "Mars";
Planet[2]: "Jupiter";
Moon[0]: "Io";
Moon[1]: "Luna";

With the previous input, the output will be the following:

{
  "Moon": [
    "Io",
    "Luna"
  ],
  "Planet": [
    "Earth",
    "Mars",
    "Jupiter"
  ]
}

Define and call functions

This section describes call functions.

Defining functions

A function in Whistle is a set of mappings that produce a JSON object. The function maps a set of inputs to a set of fields in the resulting JSON object. The following examples use a function to add structure to the planets from the previous section.

Planet[0]: PlanetName_PlanetInfo("Earth");
Planet[1]: PlanetName_PlanetInfo("Mars");
Planet[2]: PlanetName_PlanetInfo("Jupiter");
Moon[0]: MoonName_MoonInfo("Luna");

def PlanetName_PlanetInfo(planetName) {
  name: planetName;
  type: "Planet"
}

def MoonName_MoonInfo(moonName) {
  name: moonName;
  type: "Moon"
}

With the previous input, the output would be the following:

{
  "Moon": [
    {
      "name": "Luna",
      "type": "Moon"
    }
  ],
  "Planet": [
    {
      "name": "Earth",
      "type": "Planet"
    },
    {
      "name": "Mars",
      "type": "Planet"
    },
    {
      "name": "Jupiter",
      "type": "Planet"
    }
  ]
}

Calling functions

The following sample demonstrates calling functions, which is similar to C or Python.

A function call looks like FunctionName(a, b, c).

You can chain function calls by passing the result of one function to the next one. For example, SingleParameterFunctionName(FunctionName(a, b, c)).

Similarly, you can pass the results of a function into a function with multiple parameters. For example, MultipleParamFunctionName(FunctionName(a, b, c), d).

The following sample function generalizes the function from the previous section by making the celestial body's type an input.

Planet[0]: BodyName_BodyType_BodyInfo("Earth", "Planet");
Planet[1]: BodyName_BodyType_BodyInfo("Mars", "Planet");
Planet[2]: BodyName_BodyType_BodyInfo("Jupiter", "Planet");
Moon[0]: BodyName_BodyType_BodyInfo("Luna", "Moon");

def BodyName_BodyType_BodyInfo(bodyName, bodyType) {
  name: bodyName
  type: bodyType
}

With the previous input, the output would be the following:

{
  "Moon": [
    {
      "name": "Luna",
      "type": "Moon"
    }
  ],
  "Planet": [
    {
      "name": "Earth",
      "type": "Planet"
    },
    {
      "name": "Mars",
      "type": "Planet"
    },
    {
      "name": "Jupiter",
      "type": "Planet"
    }
  ]
}

Functions returning a primitive

Functions in Whistle can return Arrays, Objects and Primitives.

The following sample function sets the Primitive field to the number 20:

Primitive: Num_DoubleNum(10);

def Num_DoubleNum(num) 2 * num

With the previous input, the output is the following:

{
  "Primitive": 20
}

Merge semantics

The following steps show how data objects are merged using Whistle.

Given the following mapping:

    Merged: Colour_Colour_MergedColours("red", "blue")

    def Colour_Colour_MergedColours(col1, col2) {
      field1: "hello";

      Colour_Col1(col1);

      Colour_Col2(col2);
    }

    def Colour_Col1(col) {
      colour.first: col;
      colours[1]: col;
    }

    def Colour_Col2(col) {
      colour.second: col;
      colours[1]: col;
    }

The output would be the following:

    {
      "Merged": {
          "colour": {
            "first": "red",
            "second": "blue"
          },
          "colours": [
            "blue"
          ]
          "field1": "hello"
      },
    }

Data objects are merged as follows:

Arrays are concatenated.
Hardcoded array indexes are preserved, so in the previous example, blue overwrites red.
New fields in objects are added.
Existing fields in objects are merged recursively.
Primitives are overwritten.
null and empty values don't overwrite existing values.

Mapping from input data

Given the following source message:

{
  "Planets": [
    {
      "name": "Earth"
    },
    {
      "name": "Mars"
    },
    {
      "name": "Jupiter"
    }
  ],
  "Moons": [
    {
      "name": "Luna"
    }
  ]
}

This data is loaded into an input called $root. Data loading into an input to the mapping engine is always in this $root input. You can use $root inside functions, but you should avoid doing so because it can signal non-modular mappings.

Planet[0]: BodyName_BodyType_BodyInfo($root.Planets[0], "Planet");
Planet[1]: BodyName_BodyType_BodyInfo($root.Planets[1], "Planet");
Planet[2]: BodyName_BodyType_BodyInfo($root.Planets[2], "Planet");
Moon[0]: BodyName_BodyType_BodyInfo($root.Moons[0], "Moon");

def BodyName_BodyType_BodyInfo(body, bodyType) {
  name: body.name
  type: bodyType
}

The output would be the following:

{
  "Planet": [
    {
      "name": "Earth",
      "type": "Planet"
    },
    {
      "name": "Mars",
      "type": "Planet"
    },
    {
      "name": "Jupiter",
      "type": "Planet"
    }
  ],
  "Moon": [
    {
      "name": "Luna",
      "type": "Moon"
    }
  ]
}

Arrays

This sections describes the different operations with arrays.

Iteration

The syntax for iterating over an array is to add a [] suffix. You can use this syntax in the following ways:

Function(a[]) means "pass each element of a (one at a time) to Function".
Function(a[], b) means "pass each element of a (one at a time), along with b to Function". If b is an array of the same length as a, you can iterate over them together.
Function(a[], b[]) means "pass each element of a (one at a time), along with each element of b (at the same index) to Function".
[] is also allowed after function calls: Function2(Function(a)[]) means "pass each element from the result of Function(a) (one at a time) to Function2".
The result of an iterating function call is also an array.

See the following example to iterative over the Planets and Moons arrays:

Planet: BodyName_BodyType_BodyInfo($root.Planets[], "Planet");
Moon: BodyName_BodyType_BodyInfo($root.Moons[], "Moon");

def BodyName_BodyType_BodyInfo(body, bodyType) {
  name: body.name
  type: bodyType
}

With the previous input, the output would be the following:

{
  "Planet": [
    {
      "name": "Earth",
      "type": "Planet"
    },
    {
      "name": "Mars",
      "type": "Planet"
    },
    {
      "name": "Jupiter",
      "type": "Planet"
    }
  ],
  "Moon": [
    {
      "name": "Luna",
      "type": "Moon"
    }
  ]
}

Appending to arrays

You can append to an array using [] in the middle of the path. For example, types[].typeName: ... is valid and creates types: [{"typeName": ... }]. You can also use hardcoded indexes, such as types[0]: .... "Out of bounds" indexes, such as types[153]: ..., generate missing elements as null.

Example using index numbers:

Planet[0]: "Earth"
Planet[1]: "Mars"
Planet[2]: "Jupiter"
Moon[0]: "Luna"

Example using appending:

Planet[]: "Earth"
Planet[]: "Mars"
Planet[]: "Jupiter"
Moon[]: "Luna"

Wildcards

The [*] syntax works the same as specifying an index, except that it returns an array of values. Multiple arrays mapped through with [*], for example a[*].b.c[*].d, result in one long, non-nested array of the values of d with the same item order. Hardcoded array indexes are preserved.

Null values are included through jagged traversal. For example, a[*].b.c[*].d. If some instance of a does not contain b.c, then a single null value is returned for that instance.

For example:

PlanetNames: $root.Planets[*].name;

With the previous example, the output is the following:

{
  "PlanetNames": ["Earth", "Mars", "Jupiter"]
}

Prepend the words "Celestial Body" to the names in PlanetNames using what you learned about iterating over arrays:

PlanetNames: AddPrefix("Celestial Body ", $root.Planets[]);

def AddPrefix(prefix, planet) {
  prefix + planet.name
}

With the previous example, the output would be the following:

{
  "PlanetNames": [
    "Celestial Body Earth",
    "Celestial Body Mars",
    "Celestial Body Jupiter"
  ]
}

Writing to array fields

The following steps show how to write to array fields:

PlanetNames: ($root.Planets[*].name[]);

Planet: BodyName_BodyType_BodyInfo($root.Planets[], "Planet");
Moon: BodyName_BodyType_BodyInfo($root.Moons[], "Moon");

def BodyName_BodyType_BodyInfo(body, bodyType) {
  name: body.name
  types[]: bodyType
  types[]: "Body"
}

With the previous input, the output would be the following:

{
  "Moon": [
    {
      "name": "Luna",
      "types": ["Moon", "Body"]
    }
  ],
  "Planet": [
    {
      "name": "Earth",
      "types": ["Planet", "Body"]
    },
    {
      "name": "Mars",
      "types": ["Planet", "Body"]
    },
    {
      "name": "Jupiter",
      "types": ["Planet", "Body"]
    }
  ],
  "PlanetNames": ["Earth", "Mars", "Jupiter"]
}

Variables

Variables allow mapped data to be reused without re-executing it. The var keyword indicates that the target field is a variable. Variables have identical semantics to fields. You can write to or iterate over variables in the same way as any input, however, variables don't show up in the mapping output.

The following mapping is equivalent to the mapping in Writing to array fields.

PlanetNames: $root.Planets[*].name;

Planet: BodyName_BodyType_BodyInfo($root.Planets[], "Planet")
Moon: BodyName_BodyType_BodyInfo($root.Moons[], "Moon")

def BodyName_BodyType_BodyInfo(body, bodyType) {
  var tempName: body.name;
  name: tempName;
  types[]: bodyType;
  types[]: "Body";
}

Whistle containers

A Whistle container is an object used to map a set of fields to values. Containers are equivalent to a Python map or a JSON object containing keys and fields.

The following snippet shows how to declare a container variable:

var PlanetContainer: {
  Planet: "Earth"
  Moon: "Luna"
  Size: "Medium"
  Neighbors: ["Mars", "Jupiter"]
}

Conditions

Conditions are values that are only evaluated if a condition is met. Conditions in Whistle are expressed as ternary expressions.

Preparation

In this section, you will use planetary data taken from the NASA factsheets.

{
  "Planets": [
    {
      "name": "Earth",
      "semiMajorAxis": 149.6
    },
    {
      "name": "Mars",
      "semiMajorAxis": 227.92
    },
    {
      "name": "Jupiter",
      "semiMajorAxis": 778.57
    }
  ],
  "Moons": [
    {
      "name": "Luna",
      "semiMajorAxis": 0.3844
    }
  ]
}

Transforming the data into Astronomical Units (AU) involves converting from the input in millions of km, using the conversion factor 149.598 million km = 1 AU. To get the distance in AU, use the $Div predefined function to divide the distance in millions of km by the conversion constant.

PlanetNames: $root.Planets[*].name;

Planet: BodyName_BodyType_BodyInfo($root.Planets[], "Planet");
Moon: BodyName_BodyType_BodyInfo($root.Moons[], "Moon");

def BodyName_BodyType_BodyInfo(body, bodyType) {
  name: body.name;
  types[]: bodyType;
  types[]: "Body";

  semiMajorAxisAU: body.semiMajorAxis / 149.598;
}

Conditional mappings

The following sample demonstrates conditional mappings, which are only evaluated if a condition is met.

Add a condition so that the semiMajorAxisAU field is output on planets and not moons:

Use the == (equal) operator for comparison.
Use the if ... then ... else ... statement to conditionally execute the mapping.
- The expression in the if statement is evaluated and the value after then is evaluated and returned if and only if the conditions are true. Otherwise, the value after else is evaluated and returned.
- The else ... is optional and defaults to else {}, meaning that it returns a null value.

PlanetNames: $root.Planets[*].name[];

Planet: BodyName_BodyType_BodyInfo($root.Planets[], "Planet")
Moon: BodyName_BodyType_BodyInfo($root.Moons[], "Moon")

def BodyName_BodyType_BodyInfo(body, bodyType) {
  var bigName: body.name
  name: bigName
  types[]: bodyType
  types[]: "Body"

  semiMajorAxisAU: if bodyType == "Planet" then body.semiMajorAxis / 149.598
}

In this example, the / operator divides the million km distance by the conversion constant to produce the distance in AU.

The output would be the following:

{
  "Moon": [
    {
      "name": "Luna",
      "types": ["Moon", "Body"]
    }
  ],
  "Planet": [
    {
      "name": "Earth",
      "semiMajorAxisAU": 1.0000142656266688,
      "types": ["Planet", "Body"]
    },
    {
      "name": "Mars",
      "semiMajorAxisAU": 1.5235511458665132,
      "types": ["Planet", "Body"]
    },
    {
      "name": "Jupiter",
      "semiMajorAxisAU": 5.2044191630277785,
      "types": ["Planet", "Body"]
    }
  ],
  "PlanetNames": ["Jupiter"]
}

Conditional blocks

The following sample demonstrates conditional blocks, which wrap a set of mappings within a condition. Set the semiMajorAxis.unit to AU if the bodyType is a Planet:

PlanetNames: $root.Planets[*].name[];

Planet: BodyName_BodyType_BodyInfo($root.Planets[], "Planet")
Moon: BodyName_BodyType_BodyInfo($root.Moons[], "Moon")

def BodyName_BodyType_BodyInfo(body, bodyType) {
  var bigName: body.name
  name: bigName
  types[]: bodyType
  types[]: "Body"
  if (bodyType == "Planet") then {
    semiMajorAxis.value: body.semiMajorAxis / 149.598
    semiMajorAxis.unit: "AU"
  } else {
    semiMajorAxis.value: body.semiMajorAxis * 1000000
    semiMajorAxis.unit: "KM"
  }
}

With the previous example, the output would be the following:

{
  "Moon": [
    {
      "name": "Luna",
      "semiMajorAxis": {
        "unit": "KM",
        "value": 384400
      },
      "types": ["Moon", "Body"]
    }
  ],
  "Planet": [
    {
      "name": "Earth",
      "semiMajorAxis": {
        "unit": "AU",
        "value": 1.0000142656266688
      },
      "types": ["Planet", "Body"]
    },
    {
      "name": "Mars",
      "semiMajorAxis": {
        "unit": "AU",
        "value": 1.5235511458665132
      },
      "types": ["Planet", "Body"]
    },
    {
      "name": "Jupiter",
      "semiMajorAxis": {
        "unit": "AU",
        "value": 5.2044191630277785
      },
      "types": ["Planet", "Body"]
    }
  ],
  "PlanetNames": ["Jupiter"]
}

Operators

Similar to Python or C, there are operators available for common arithmetic and logical operations. This section shows some of the following operators, where num is a number input, bool is a boolean input, and any is any type of input:

Operator	Description
num + num	Addition
num - num	Subtraction
num * num	Multiplication
num / num	Division
str + any	Concatenation
any + str	Concatenation
bool and bool	Logical AND
bool or bool	Logical OR
!bool	Logical NOT
any == any	Deep equals - all elements in an array or values in an object must be the same to return true. `x = y = z` is a valid expression and is equivalent to `(x = y) = z`. If `x = y` is true, this checks `true = z`.
any != any	Not Equal
any?	Value exists - is defined, isn't literal `null`, and isn't empty. An empty array has 0 elements (`null`s count as elements). An empty object has 0 keys.
!any?	Value Does Not Exist

Filters

Filters have the following properties:

Filters allow narrowing an array to items that match a condition.
The where keyword indicates a filter, similar to if indicating a condition.
Each item from the array is loaded one at a time into an input named $ in the filter.
The filter produces a new array. To iterate over the results, use the [] operator.
Filters can only be the last element in a path. For example, a.b[where $.color = "red"].c is invalid.

The following steps show how to use filters with the planets example:

Planet: BodyName_BodyType_BodyInfo($root.Planets[where $.semiMajorAxis > 200][], "Planet");

With the previous example, the output would be the following:

{
  "Moon": [
    {
      "name": "Luna",
      "semiMajorAxis": {
        "unit": "KM",
        "value": 384400
      },
      "types": ["Moon", "Body"]
    }
  ],
  "Planet": [
    {
      "name": "Mars",
      "semiMajorAxis": {
        "unit": "AU",
        "value": 1.5235511458665132
      },
      "types": ["Planet", "Body"]
    },
    {
      "name": "Jupiter",
      "semiMajorAxis": {
        "unit": "AU",
        "value": 5.2044191630277785
      },
      "types": ["Planet", "Body"]
    }
  ],
  "PlanetNames": ["Jupiter"]
}

Caveats

This section describes the null propagation.

Nulls and null propagation

By default, the mapping engine handles and ignores null and missing values or fields, according to the following rules:

If a field is written with a null or empty value, it's ignored (thus null, {}, and [] can never show up in the mapping output).
If a non-existent field is accessed, it returns null.
If a null value is passed to a function, the function is still executed.

The following sample input and mapping demonstrate nulls and null propagation:

Input

{
  "Red": {
    "Blue": 1
  }
}

Mapping

def Root_Example(rt) {
  // This field does not appear in the output
  excluded: rt.Abcdefghijklmnop

  // This array will only contain the existing items
  included[]: rt.Red.Blue
  included[]: rt.Abcd[123].efghi[*].jk[*].lmnop
  included[]: rt.Red.Blue

  // nested_1 will appear with just the constant, nested_2 won't appear
  nested_1: Nested_Example(rt.Abcdefghijklmnop, "Constant")
  nested_2: Nested_Example(rt.Abcdefghijklmnop, rt.Abcdefghijklmnop)
}

def Nested_Example(one, two) {
  one: one
  two: two
}

The output would be similar to the following sample:

{
  "Example": [
      "included": [
        1,
        1
      ],
      "nested_1": {
        "two": "Constant"
      }
  ]
}

Using `root` in a function

The following sample demonstrates using the root keyword inside a function to send data to the root of the output:

Red[]: "Blue"
Complex: Hello_World_HelloWorldObject("Hi", "Planet")

def Hello_World_HelloWorldObject(hello, world) {
    hello: hello
    world: world
    root Red[]: world
    root Complex.boo: "boo!"
}

With the previous example, the output would be the following:

{
  "Complex": [
    {
      "boo": "boo!",
      "hello": "Hi",
      "world": "Planet"
    }
  ],
  "Red": ["Blue", "Planet"]
}

Whistle basics

Simple mapping

Add array outputs

Define and call functions

Defining functions

Calling functions

Functions returning a primitive

Merge semantics

Mapping from input data

Arrays

Iteration

Appending to arrays

Wildcards

Writing to array fields

Variables

Whistle containers

Conditions

Preparation

Conditional mappings

Conditional blocks

Operators

Filters

Caveats

Nulls and null propagation

Using root in a function

Using `root` in a function