FunctionRegistry — Contract for Function Registries (Catalogs)

FunctionRegistry is the contract of function registries (catalogs) of native and user-defined functions.

package org.apache.spark.sql.catalyst.analysis

trait FunctionRegistry {
  // only required properties (vals and methods) that have no implementation
  // the others follow
  def clear(): Unit
  def dropFunction(name: FunctionIdentifier): Boolean
  def listFunction(): Seq[FunctionIdentifier]
  def lookupFunction(name: FunctionIdentifier): Option[ExpressionInfo]
  def lookupFunction(name: FunctionIdentifier, children: Seq[Expression]): Expression
  def lookupFunctionBuilder(name: FunctionIdentifier): Option[FunctionBuilder]
  def registerFunction(
    name: FunctionIdentifier,
    info: ExpressionInfo,
    builder: FunctionBuilder): Unit
}
Table 1. FunctionRegistry Contract
Property Description

clear

Used exclusively when SessionCatalog is requested to reset

dropFunction

Used when…​FIXME

listFunction

Used when…​FIXME

lookupFunction

Used when:

lookupFunctionBuilder

Used when…​FIXME

registerFunction

Used when:

Note
The one and only FunctionRegistry available in Spark SQL is SimpleFunctionRegistry.

FunctionRegistry is available through functionRegistry property of a SessionState (that is available as sessionState property of a SparkSession).

scala> :type spark
org.apache.spark.sql.SparkSession

scala> :type spark.sessionState.functionRegistry
org.apache.spark.sql.catalyst.analysis.FunctionRegistry
Note
You can register a new user-defined function using UDFRegistration.
Table 2. FunctionRegistry’s Attributes
Name Description

builtin

SimpleFunctionRegistry with the built-in functions registered.

FunctionRegistry manages function expression registry of Catalyst expressions and the corresponding built-in/native SQL functions (that can be used in SQL statements).

Table 3. (Subset of) FunctionRegistry’s Catalyst Expression to SQL Function Mapping
Catalyst Expression SQL Function

CumeDist

cume_dist

IfNull

ifnull

Left

left

MonotonicallyIncreasingID

monotonically_increasing_id

NullIf

nullif

Nvl

nvl

Nvl2

nvl2

ParseToDate

to_date

ParseToTimestamp

to_timestamp

Right

right

CreateNamedStruct

struct

expression Internal Method

expression[T <: Expression](name: String)
  (implicit tag: ClassTag[T]): (String, (ExpressionInfo, FunctionBuilder))

expression…​FIXME

Note
expression is used when…​FIXME

SimpleFunctionRegistry

SimpleFunctionRegistry is the default FunctionRegistry that is backed by a hash map (with optional case sensitivity).

createOrReplaceTempFunction Final Method

createOrReplaceTempFunction(name: String, builder: FunctionBuilder): Unit

createOrReplaceTempFunction…​FIXME

Note
createOrReplaceTempFunction is used exclusively when UDFRegistration is requested to register an user-defined function, user-defined aggregate function, user-defined function (as UserDefinedFunction) or registerPython.

functionExists Method

functionExists(name: FunctionIdentifier): Boolean

functionExists…​FIXME

Note
functionExists is used when…​FIXME

results matching ""

    No results matching ""