Hive Variables - Spectrum_Routing_for_Big_Data - 5.1

Spectrum Routing Installation: Cloudera

Product type
Software
Portfolio
Locate
Product family
Spectrum
Product
Spatial Big Data > Routing for Big Data
Version
5.1
Language
English
Product name
Spectrum Routing for Big Data
Title
Spectrum Routing Installation: Cloudera
Copyright
2024
First publish date
2017
Last updated
2024-10-18
Published on
2024-10-18T09:54:26.541418

The Routing UDFs use Hive variables to set various properties, which take precedence over the system properties. They can also be set as part of an option in a UDF query, where they will take precedence over the Hive variables as well as the system properties.

Variable Required Example
pb.routing.config.location

Location of dbList.json containing the routing configuration.

Yes hdfs:///precisely/routing/software/resources/config/dbList.json;
pb.routing.engine.timeout

Specifies the duration (in milliseconds) after which a GRA engine shuts down if it does not receive any request and is idle. The default value for this property is 900000 (15 minutes).

No 600000
pb.routing.routeTimeout

Specifies the amount of time (in milliseconds) allowed for a point-to-point routing job to be processed and completed. If the job is not processed or completed within the specified duration, the SQL returns null to the console and exception to logs for each record processed.

This property can be used for the PointToPointRoute UDF.

No 10000
pb.routing.isoTimeout

Specifies the amount of time (in milliseconds) allowed for a boundary routing job to be processed and completed. If the job is not processed or completed within the specified duration, the SQL returns null to the console and exception to logs for each record processed.

This property can be used for the IsoDistance and IsoChrone UDFs

No 17000
pb.routing.allowFallback

This Boolean property controls whether the engine can use major roads in case other roads cannot be used.

If set to true, the engine can use major roads.

No false
pb.download.location
Note: Use only if reference data was distributed remotely via HDFS or S3.

Location of the directory where reference data will be downloaded to. This path must exist on every data node and the HiveServer2 node.

No /precisely/downloads
pb.download.group
Note: Use only if reference data was distributed remotely via HDFS or S3.

This is an optional property and only specific to POSIX-compliant platforms like Linux. It specifies the operating system group which should be applied to the downloaded data on a local file system, so that each Hadoop service can update the data when required. This group should be present on all nodes in the cluster and the operating system user executing the Hadoop service should be a part of this group.

For more information, see Download Permissions.

No dm_users
pb.routing.error.limit

The number of routing errors to allow before failing a task for "too many errors". This prevents a task (and thus the job) from continuing in the case of a likely configuration error. The default value is 10.

No 1
pb.routing.error.limit.disabled

Disables the error limit. All errors will be logged but will not cause the task or job to fail.

No true

Setting Variables

Since these are Hive configuration variables, you can set them permanently by editing the hiveserver2-site.xml file in your cluster.

Example

pb.routing.config.location=hdfs:///precisely/routing/software/resources/config/dbList.json
pb.download.location = /precisely/downloads
pb.download.group= dm_users

Alternatively you can set them temporarily in each Hive session that you open.

For example, if you wish to perform routing in Beeline you can set the variables in the following way:

Example
set pb.routing.config.location=hdfs:///precisely/routing/software/resources/config/dbList.json;
set pb.routing.allowFallback=true;
set pb.download.location = /precisely/downloads;
set pb.download.group= dm_users;

Variables set in the local session will override any hive-site variables. This means you can have default values set in the hiveserver2-site file and override them in Beeline when necessary.

System Variables (Deprecated)

Hive variables used to exist in the system namespace. This is deprecated as of version 3.1 and no longer the recommended method.