ExtractPythonUDFs Physical Query Optimization

ExtractPythonUDFs is a physical query optimization (aka physical query preparation rule or simply preparation rule) that QueryExecution uses to optimize the physical plan of a structured query by extracting Python UDFs from a physical query plan (excluding FlatMapGroupsInPandasExec operators that it simply skips over).

Technically, ExtractPythonUDFs is just a Catalyst rule for transforming physical query plans, i.e. Rule[SparkPlan].

ExtractPythonUDFs is part of preparations batch of physical query plan rules and is executed when QueryExecution is requested for the optimized physical query plan (i.e. in executedPlan phase of a query execution).

Extracting Python UDFs from Physical Query Plan — extract Internal Method

extract(plan: SparkPlan): SparkPlan


extract is used exclusively when ExtractPythonUDFs is requested to optimize a physical query plan.

trySplitFilter Internal Method

trySplitFilter(plan: SparkPlan): SparkPlan


trySplitFilter is used exclusively when ExtractPythonUDFs is requested to extract.

results matching ""

    No results matching ""