Column API — Column Operators

Column API is a set of operators to work with values in a column (of a Dataset).

Table 1. Column Operators
Operator Description

asc

 
asc: Column

asc_nulls_first

 
asc_nulls_first: Column

asc_nulls_last

 
asc_nulls_last: Column

desc

 
desc: Column

desc_nulls_first

 
desc_nulls_first: Column

desc_nulls_last

 
desc_nulls_last: Column

isin

 
isin(list: Any*): Column

isInCollection

 
isInCollection(values: scala.collection.Iterable[_]): Column

(New in 2.4.0) An expression operator that is true if the value of the column is in the given values collection

isInCollection is simply a synonym of isin operator.

isin Operator

isin(list: Any*): Column

Internally, isin creates a Column with In predicate expression.

val ids = Seq((1, 2, 2), (2, 3, 1)).toDF("x", "y", "id")
scala> ids.show
+---+---+---+
|  x|  y| id|
+---+---+---+
|  1|  2|  2|
|  2|  3|  1|
+---+---+---+

val c = $"id" isin ($"x", $"y")
val q = ids.filter(c)
scala> q.show
+---+---+---+
|  x|  y| id|
+---+---+---+
|  1|  2|  2|
+---+---+---+

// Note that isin accepts non-Column values
val c = $"id" isin ("x", "y")
val q = ids.filter(c)
scala> q.show
+---+---+---+
|  x|  y| id|
+---+---+---+
+---+---+---+

results matching ""

    No results matching ""