Nondeterministic Expression Contract

Nondeterministic is a contract for Catalyst expressions that are non-deterministic and non-foldable.

Nondeterministic expressions require explicit initialization (with the current partition index) before evaluating a value.

package org.apache.spark.sql.catalyst.expressions

trait Nondeterministic extends Expression {
  // only required methods that have no implementation
  protected def initializeInternal(partitionIndex: Int): Unit
  protected def evalInternal(input: InternalRow): Any
}
Table 1. Nondeterministic Contract
Method Description

initializeInternal

Initializing the Nondeterministic expression

Used exclusively when Nondeterministic expression is requested to initialize

evalInternal

Evaluating the Nondeterministic expression

Used exclusively when Nondeterministic expression is requested to evaluate a value
Note
 Nondeterministic expressions are the target of PullOutNondeterministic logical plan rule.
Table 2. Nondeterministic Expressions
Expression Description

CurrentBatchTimestamp

InputFileBlockLength

InputFileBlockStart

InputFileName

MonotonicallyIncreasingID

NondeterministicExpression

Rand

Randn

RDG

SparkPartitionID

Table 3. Nondeterministic’s Internal Properties (e.g. Registries, Counters and Flags)
Name Description

deterministic

Always turned off (i.e. false)

foldable

Always turned off (i.e. false)

initialized

Controls whether a Nondeterministic expression has been initialized before evaluation.

Turned off by default.

Initializing Expression — initialize Method

initialize(partitionIndex: Int): Unit

Internally, initialize initializes itself (with the input partition index) and turns the internal initialized flag on.

Note
 initialize is used exclusively when InterpretedProjection and InterpretedMutableProjection are requested to initialize themselves.

Evaluating Expression — eval Method

eval(input: InternalRow): Any
Note
 eval is part of Expression Contract for the interpreted (non-code-generated) expression evaluation, i.e. evaluating a Catalyst expression to a JVM object for a given internal binary row.

eval is just a wrapper of evalInternal that makes sure that initialize has already been executed (and so the expression is initialized).

Internally, eval makes sure that the expression was initialized and calls evalInternal.

eval reports a IllegalArgumentException exception when the internal initialized flag is off, i.e. initialize has not yet been executed.

requirement failed: Nondeterministic expression [name] should be initialized before eval.

results matching ""

    No results matching ""