LocalRelation Leaf Logical Operator

LocalRelation is a leaf logical operator that allow functions like collect or take to be executed locally, i.e. without using Spark executors.

LocalRelation is created when…​FIXME

When Dataset operators can be executed locally, the Dataset is considered local.

LocalRelation represents Datasets that were created from local collections using SparkSession.emptyDataset or SparkSession.createDataset methods and their derivatives like toDF.

val dataset = Seq(1).toDF
scala> dataset.explain(true)
== Parsed Logical Plan ==
LocalRelation [value#216]

== Analyzed Logical Plan ==
value: int
LocalRelation [value#216]

== Optimized Logical Plan ==
LocalRelation [value#216]

== Physical Plan ==
LocalTableScan [value#216]

It can only be constructed with the output attributes being all resolved.

The size of the objects (in statistics) is the sum of the default size of the attributes multiplied by the number of records.

When executed, LocalRelation is translated to LocalTableScanExec physical operator.

Creating LocalRelation Instance

LocalRelation takes the following when created:

LocalRelation initializes the internal registries and counters.

results matching ""

    No results matching ""