SessionCatalog

SessionCatalog is a proxy between SparkSession and the underlying metastore, e.g. HiveSessionCatalog.

SessionCatalog is created when SessionState sets catalog (lazily).

You can access the SessionCatalog for a SparkSession through SessionState.

sparkSession.sessionState.catalog
Table 1. SessionCatalog’s Internal Registries and Counters (in alphabetical order)
Name Description

tempTables

FIXME

Used when…​FIXME

currentDb

FIXME

Used when…​FIXME

functionResourceLoader

FIXME

Used when…​FIXME

tableRelationCache

A cache of fully-qualified table names to table relation plans (i.e. LogicalPlan).

Used when SessionCatalog refreshes a table

refreshTable Method

Caution
FIXME

createTempFunction Method

Caution
FIXME

loadFunctionResources Method

Caution
FIXME

functionExists Method

Caution
FIXME

alterTempViewDefinition Method

alterTempViewDefinition(name: TableIdentifier, viewDefinition: LogicalPlan): Boolean

alterTempViewDefinition alters the temporary view by updating an in-memory temporary table (when a database is not specified and the table has already been registered) or a global temporary table (when a database is specified and it is for global temporary tables).

Note
"Temporary table" and "temporary view" are synonyms.

alterTempViewDefinition returns true when an update could be executed and finished successfully.

createTempView Method

Caution
FIXME

createGlobalTempView Method

Caution
FIXME

createTable Method

Caution
FIXME

alterTable Method

Caution
FIXME

Creating SessionCatalog Instance

SessionCatalog takes the following when created:

SessionCatalog initializes the internal registries and counters.

Finding Function by Name (Using FunctionRegistry) — lookupFunction Method

lookupFunction(
  name: FunctionIdentifier,
  children: Seq[Expression]): Expression

lookupFunction finds a function by name.

For a function with no database defined that exists in FunctionRegistry, lookupFunction requests FunctionRegistry to find the function (by its unqualified name, i.e. with no database).

If the name function has the database defined or does not exist in FunctionRegistry, lookupFunction uses the fully-qualified function name to check if the function exists in FunctionRegistry (by its fully-qualified name, i.e. with a database).

For other cases, lookupFunction requests ExternalCatalog to find the function and loads its resources. It then creates a corresponding temporary function and looks up the function again.

Note
lookupFunction is used exclusively when Analyzer resolves functions.

results matching ""

    No results matching ""