HiveUtils

HiveUtils is used to create a HiveClientImpl that HiveExternalCatalog uses to interact with a Hive metastore.

Tip

Enable INFO logging level for org.apache.spark.sql.hive.HiveUtils logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.sql.hive.HiveUtils=INFO

Refer to Logging.

Creating HiveClientImpl — newClientForMetadata Method

newClientForMetadata(
  conf: SparkConf,
  hadoopConf: Configuration): HiveClient  (1)
newClientForMetadata(
  conf: SparkConf,
  hadoopConf: Configuration,
  configurations: Map[String, String]): HiveClient
  1. Executes the other newClientForMetadata with time configurations formatted

Internally, newClientForMetadata creates a new SQLConf with spark.sql properties only (from the input SparkConf).

newClientForMetadata then creates a IsolatedClientLoader per the input parameters and the following configuration properties:

You should see one of the following INFO messages in the logs:

Initializing HiveMetastoreConnection version [hiveMetastoreVersion] using Spark classes.
Initializing HiveMetastoreConnection version [hiveMetastoreVersion] using maven.
Initializing HiveMetastoreConnection version [hiveMetastoreVersion] using [jars]

In the end, newClientForMetadata requests the IsolatedClientLoader to create a HiveClientImpl.

Note
newClientForMetadata is used exclusively when HiveExternalCatalog is requested for a HiveClient.

inferSchema Method

inferSchema(table: CatalogTable): CatalogTable

inferSchema…​FIXME

Note
inferSchema is used when…​FIXME

results matching ""

    No results matching ""