UserDefinedFunction represents a user-defined function.

UserDefinedFunction is created when:

import org.apache.spark.sql.functions.udf
val lengthUDF = udf { s: String => s.length }

scala> :type lengthUDF

val r = lengthUDF($"name")

scala> :type r

UserDefinedFunction can have an optional name.

val namedLengthUDF = lengthUDF.withName("lengthUDF")
scala> namedLengthUDF($"name")
res2: org.apache.spark.sql.Column = UDF:lengthUDF(name)

UserDefinedFunction is nullable by default, but can be changed as non-nullable.

val nonNullableLengthUDF = lengthUDF.asNonNullable
assert(nonNullableLengthUDF.nullable == false)

UserDefinedFunction is deterministic by default, i.e. produces the same result for the same input. UserDefinedFunction can be changed to be non-deterministic.

val ndUDF = lengthUDF.asNondeterministic
assert(ndUDF.deterministic == false)
Table 1. UserDefinedFunction’s Internal Properties (e.g. Registries, Counters and Flags)
Name Description


Flag that controls whether the function is deterministic (true) or not (false).

Default: true

Used when UserDefinedFunction is requested to execute

Executing UserDefinedFunction (Creating Column with ScalaUDF Expression) — apply Method

apply(exprs: Column*): Column

apply creates a Column with ScalaUDF expression.

import org.apache.spark.sql.functions.udf
scala> val lengthUDF = udf { s: String => s.length }
lengthUDF: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(<function1>,IntegerType,Some(List(StringType)))

scala> lengthUDF($"name")
res1: org.apache.spark.sql.Column = UDF(name)
apply is used when…​FIXME

Marking UserDefinedFunction as NonNullable — asNonNullable Method

asNonNullable(): UserDefinedFunction


asNonNullable is used when…​FIXME

Naming UserDefinedFunction — withName Method

withName(name: String): UserDefinedFunction


withName is used when…​FIXME

Creating UserDefinedFunction Instance

UserDefinedFunction takes the following when created:

UserDefinedFunction initializes the internal registries and counters.

results matching ""

    No results matching ""