IsolatedClientLoader Utility

IsolatedClientLoader is created for HiveUtils utility for newClientForExecution and newClientForMetadata.

Tip

Enable ALL logging level for org.apache.spark.sql.hive.client.IsolatedClientLoader logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.sql.hive.client.IsolatedClientLoader=ALL

Refer to Logging.

Creating IsolatedClientLoader Instance

IsolatedClientLoader takes the following to be created:

  • HiveVersion

  • SparkConf

  • Hadoop Configuration

  • Execution JARs (default: empty)

  • Configuration (default: empty)

  • isolationOn flag (default: true)

  • sharesHadoopClasses flag (default: true)

  • Root ClassLoader (default: ClassLoader.getSystemClassLoader.getParent.getParent)

  • Base class ClassLoader (default: Thread.currentThread().getContextClassLoader)

  • Shared prefixes (default: empty)

  • Barrier prefixes (default: empty)

IsolatedClientLoader initializes the internal properties.

Hive Metastore Version — hiveVersion Utility

hiveVersion(
  version: String): HiveVersion

hiveVersion creates a HiveVersion based on the input version.

Acceptable versions and synonyms:

  • 12, 0.12, 0.12.0

  • 13, 0.13, 0.13.0, 0.13.1

  • 14, 0.14, 0.14.0

  • 1.0, 1.0.0

  • 1.1, 1.1.0

  • 1.2, 1.2.0, 1.2.1, 1.2.2

  • 2.0, 2.0.0, 2.0.1

  • 2.1, 2.1.0, 2.1.1

  • 2.2, 2.2.0

  • 2.3, 2.3.0, 2.3.1, 2.3.2, 2.3.3

Note

hiveVersion is used when:

Creating IsolatedClientLoader for Given Hive Metastore Version — forVersion Utility

forVersion(
  hiveMetastoreVersion: String,
  hadoopVersion: String,
  sparkConf: SparkConf,
  hadoopConf: Configuration,
  config: Map[String, String] = Map.empty,
  ivyPath: Option[String] = None,
  sharedPrefixes: Seq[String] = Seq.empty,
  barrierPrefixes: Seq[String] = Seq.empty,
  sharesHadoopClasses: Boolean = true): IsolatedClientLoader

forVersion…​FIXME

Note
 forVersion is used when HiveUtils utility is used to newClientForMetadata.

Creating HiveClient — createClient Method

createClient(): HiveClient

createClient…​FIXME

Note

createClient is used when:

results matching ""

    No results matching ""