UserDefinedFunction

UserDefinedFunction represents a user-defined function.

UserDefinedFunction is created when:

  1. udf function is executed

  2. UDFRegistration is requested to register a Scala function as a user-defined function (in FunctionRegistry)

import org.apache.spark.sql.functions.udf
scala> val lengthUDF = udf { s: String => s.length }
lengthUDF: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(<function1>,IntegerType,Some(List(StringType)))

scala> lengthUDF($"name")
res1: org.apache.spark.sql.Column = UDF(name)

(new in 2.3.0) UserDefinedFunction can also have a name.

// New in 2.3.0
val namedLengthUDF = lengthUDF.withName("lengthUDF")
scala> namedLengthUDF($"name")
res2: org.apache.spark.sql.Column = UDF:lengthUDF(name)

(new in 2.3.0) UserDefinedFunction is nullable by default, but can be changed as non-nullable.

// New in 2.3.0
val nonNullableLengthUDF = lengthUDF.asNonNullable
scala> nonNullableLengthUDF.nullable
res1: Boolean = false

Executing UserDefinedFunction (Creating Column with ScalaUDF Expression) — apply Method

apply(exprs: Column*): Column

apply creates a Column with ScalaUDF expression.

import org.apache.spark.sql.functions.udf
scala> val lengthUDF = udf { s: String => s.length }
lengthUDF: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(<function1>,IntegerType,Some(List(StringType)))

scala> lengthUDF($"name")
res1: org.apache.spark.sql.Column = UDF(name)
Note
apply is used when…​FIXME

Marking UserDefinedFunction as NonNullable — asNonNullable Method

asNonNullable(): UserDefinedFunction

asNonNullable…​FIXME

Note
asNonNullable is used when…​FIXME

Naming UserDefinedFunction — withName Method

withName(name: String): UserDefinedFunction

withName…​FIXME

Note
withName is used when…​FIXME

Creating UserDefinedFunction Instance

UserDefinedFunction takes the following when created:

UserDefinedFunction initializes the internal registries and counters.

results matching ""

    No results matching ""