UserDefinedFunction

UserDefinedFunction represents a user-defined function.

UserDefinedFunction is created when:

import org.apache.spark.sql.functions.udf
scala> val lengthUDF = udf { s: String => s.length }
lengthUDF: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(<function1>,IntegerType,Some(List(StringType)))

scala> lengthUDF($"name")
res1: org.apache.spark.sql.Column = UDF(name)

UserDefinedFunction can also have a name.

val namedLengthUDF = lengthUDF.withName("lengthUDF")
scala> namedLengthUDF($"name")
res2: org.apache.spark.sql.Column = UDF:lengthUDF(name)

UserDefinedFunction is nullable by default, but can be changed as non-nullable.

val nonNullableLengthUDF = lengthUDF.asNonNullable
scala> nonNullableLengthUDF.nullable
res1: Boolean = false

Executing UserDefinedFunction (Creating Column with ScalaUDF Expression) — apply Method

apply(exprs: Column*): Column

apply creates a Column with ScalaUDF expression.

import org.apache.spark.sql.functions.udf
scala> val lengthUDF = udf { s: String => s.length }
lengthUDF: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(<function1>,IntegerType,Some(List(StringType)))

scala> lengthUDF($"name")
res1: org.apache.spark.sql.Column = UDF(name)
Note
apply is used when…​FIXME

Marking UserDefinedFunction as NonNullable — asNonNullable Method

asNonNullable(): UserDefinedFunction

asNonNullable…​FIXME

Note
asNonNullable is used when…​FIXME

Naming UserDefinedFunction — withName Method

withName(name: String): UserDefinedFunction

withName…​FIXME

Note
withName is used when…​FIXME

Creating UserDefinedFunction Instance

UserDefinedFunction takes the following when created:

UserDefinedFunction initializes the internal registries and counters.

results matching ""

    No results matching ""