SupportsPushDownFilters Contract — Data Source Readers with Filter Pushdown Optimization Support

SupportsPushDownFilters is the extension of the DataSourceReader contract for data source readers in Data Source API V2 that support filter pushdown performance optimization (and hence reduce the size of the data to be read).

Table 1. SupportsPushDownFilters Contract
Method Description

pushedFilters

 
Filter[] pushedFilters()

Data source filters that were pushed down to the data source (in pushFilters)

Used exclusively when DataSourceV2Strategy execution planning strategy is executed (on a DataSourceV2Relation logical operator with a SupportsPushDownFilters reader)

pushFilters

 
Filter[] pushFilters(Filter[] filters)

Data source filters that need to be evaluated again after scanning (so Spark can plan an extra filter operator)

Used exclusively when DataSourceV2Strategy execution planning strategy is executed (on a DataSourceV2Relation logical operator with a SupportsPushDownFilters reader)
Note

SupportsPushDownFilters is an Evolving contract that is evolving towards becoming a stable API, but is not a stable API yet and can change from one feature release to another release.

In other words, using the contract is as treading on thin ice.

results matching ""

    No results matching ""