Data Source Filter Predicate (For Filter Pushdown)

Filter is the contract for filter predicates that can be pushed down to a relation (aka data source).

Filter is used when:

package org.apache.spark.sql.sources

abstract class Filter {
  // only required methods that have no implementation
  // the others follow
  def references: Array[String]
}
Table 1. Filter Contract
Method Description

references

Column references, i.e. list of column names that are referenced by a filter

Used when:

Table 2. Filters
Filter Description

And

EqualNullSafe

EqualTo

GreaterThan

GreaterThanOrEqual

In

IsNotNull

IsNull

LessThan

LessThanOrEqual

Not

Or

StringContains

StringEndsWith

StringStartsWith

Finding Column References in Any Value — findReferences Method

findReferences(value: Any): Array[String]

findReferences takes the references from the value filter is it is one or returns an empty array.

Note
findReferences is used when EqualTo, EqualNullSafe, GreaterThan, GreaterThanOrEqual, LessThan, LessThanOrEqual and In filters are requested for their column references.

results matching ""

    No results matching ""