Hadoop Cluster - Connect_ETL - 9.13

Run the self-extracting shrink-wrap executables for the software package and license packages from the directory where it is located. For the software executable this is:

./dmexpress-<Connect version>-<OS>.parcel.bin

For the license executable, this is:

./dmexpresslicense_<license site ID>-<date>-<OS>.parcel.bin

Install the Connect for Big Data (dmexpress) parcel and the Connect for Big Data license (dmexpresslicense-XXXXX) parcel on all nodes in the cluster as follows:

1. Depending on whether you are using a local parcel repository or a remote parcel repository, do one of the following:
   • Local parcel repository – With root/sudo privileges, copy the extracted .parcel and .sha files for software and license to the Cloudera Manager local parcel repository. The default location is /opt/cloudera/parcel-repo/.
   • Remote parcel repository – With root/sudo privileges, copy the extracted .parcel and manifest.json files for software and license to your remote parcel repository. Ensure that the files have read and execute permissions for all users. As outlined on Cloudera’s Creating and Using a Parcel Repository page, follow the steps to Configure the Cloudera Manager Server to Use the Parcel URL.
2. Logged in to Cloudera Manager as an Administrator user, click on the parcel indicator button in the Cloudera Manager Admin console navigation bar to bring up the Parcels tab of the Hosts page.
3. If not already detected, click on the Check for New Parcels button. Consider the following:
   • If you are using a local parcel repository, you can see the “downloaded” parcels on this page, for example, dmexpress 9.11.1 and/or dmexpresslicense_12345 20180928.
   • If you are using a remote parcel repository, click on the Download button to download the dmexpress and/or dmexpresslicense-XXXXX parcel from the remote repository.

     Click on the Distribute button to distribute the dmexpress and/or dmexpresslicense-XXXXX parcel to the nodes in the cluster. By default, the files are written to /opt/cloudera/parcels/parcel_name/ on each node.

4. Upon completion of the distribution, either or both parcels can be activated by clicking on its Activate button. If there was a previously activated distribution of Connect for Big Data, be sure that no Connect for Big Data jobs are running, because Cloudera Manager automatically deactivates the old parcel upon activation of the new parcel, and any running jobs fail.
5. Upon activation, the symbolic link /usr/dmexpress is created/updated to point to the activated Connect installation.

   See the Cloudera Manager Enterprise Edition User Guide for details on Managing Parcels.

Execute the following steps on the machine where the Ambari server resides:

1. Run the self-extracting shrink-wrap executable for the software package from the directory where it is located. For the software executable this is:

   ./dmexpress-<Connect version>-<OS>.parcel.bin

   For the license executable, this is:

   ./dmexpresslicense-<license site ID>-<date>-<arch>.ambari-service.bin

   e.g. dmexpresslicense-12345-20180928-any.ambari-service.bin

2. Read and accept the Software License Agreement.
3. Enter a target directory in which to extract the Connect for Big Data or Connect for Big Data license service folder, or press Enter to accept the default, which is the current directory. If a folder with the same name already exists, you are prompted to overwrite; enter yes to overwrite, or no to exit the extracting process.
4. Enter a target directory in which to copy the Connect for Big Data or Connect for Big Data license service package where it can be found by the Ambari server, or press Enter to accept the default, which is the root path of the latest stack.
5. Enter yes to restart the Ambari server for the new package to be picked up, or no to restart later.
6. If the Connect for Big Data or Connect for Big Data license, respectively, service definition already exists in the repository, you are prompted to upgrade; enter yes to upgrade, or no to exit the process without updating the existing service definition package.
7. Enter the Ambari server's hostname, username, and password, and the cluster name, as prompted, to complete the upgrade.
   1. If the credentials entered fail, you can re-run this step manually by executing the following script, where

      <Ambari service extracted package path>

      is the directory you specified in step 3:

      <Ambari service extracted package path>/services/DMXh
       /package/scripts/prepare_dmxh_upgrade.sh

   2. If the credentials entered fail for the license package, execute this script:

      <Ambari service extracted package path>/services/ 
      DMXhLicense/package/scripts/prepare_dmxh_license_upgrade.sh

8. If there is no license installed, repeat steps 1-7 for the license .bin file.

Install the Connect for Big Data and/or Connect for Big Data License service on all nodes in the cluster as follows:

1. Log in to the Ambari dashboard and select Actions->Add Service.
2. On the Add Service Wizard page, select Connect for Big Data and/or Connect for Big Data License and click Next.
3. On the Assign Slaves and Clients page, check Client for all nodes, and click Next.
4. On the Configure Services page, click Next to continue with the default options (recommended). Alternatively, if you wish to change the default installation directory, expand the “Advanced” section and make changes to the Connect for Big Data Base Directory setting, ensuring that the same directory is specified for both the Connect for Big Data and Connect for Big Data License tabs, and then click Next.
5. On the Review page, verify the configuration and click Deploy to deploy Connect for Big Data and/or Connect for Big Data License, or click Back to make modifications.
6. On the Install, Start and Test page, wait for the Connect for Big Data and/or Connect for Big Data License service to be successfully installed on each node. If an error occurs, select the "Failures encountered" text to display an error log and identify the problem.

   See http://docs.hortonworks.com/ for details on Apache Ambari.

Execute the following steps starting with one node in or with access to the Hadoop cluster:

1. Run the self-extracting shrink-wrap executable for the software and license packages from the directories where they are located. For the software RPM, this is:

   ./dmexpress-<Connect version>-1.x86_64.bin

   For the license RPM, this is:

   ./dmexpresslicense-<license site ID>-<date>-<revision>.<arch>.bin

   e.g. dmexpresslicense-12345-20180927-1.x86_64.bin

2. Read and accept the Software License Agreement.
3. Enter a target directory in which to put the extracted RPM file (the default is the current folder).

rpm -i dmexpress-<Connect version>-1.x86_64.rpm

rpm -i dmexpresslicense-<license site ID>-<date>-<revision>.<arch>.rpm

rpm -i --prefix /some/other/directory dmexpress-<Connect version>-1.x86_64.rpm 

yum install dmexpress-<version>-1.x86_64.rpm

yum install dmexpresslicense-<license site ID>-<date>-<revision>.<arch>.rpm

rpm -U <package>.rpm

yum upgrade <package>.rpm

apt install ./dmexpress-<Connect version>-1.deb

apt install ./dmexpresslicense-<license site ID>-<date>-<revision>.deb

The following example shows how to run the install manually on each node. You copy the .deb file from the location it was extracted to your local system, then run the Debian install scripts:

#!/bin/bash
debPackageDBFSPath=/dbfs/mnt/azuregen2/dir1 
connectDEBPackageName=dmexpress_9.11.33-1.deb 
connectLicenseDEBPackageName=dmexpresslicense-123456_77910427-1.deb
cp -f $ debPackageDBFSPath/$connectDEBPackageName
    $connectDEBPackageName 
	chmod a+x $connectDEBPackageName
cp -f $ debPackageDBFSPath/$connectLicenseDEBPackageName 
    $connectLicenseDEBPackageName 
chmod a+x $connectLicenseDEBPackageName
sudo apt install ./$connectDEBPackageName 
rm $connectDEBPackageName
sudo apt install ./$connectLicenseDEBPackageName
rm $connectLicenseDEBPackageName

apt -y remove dmexpress

The following steps are required prior to running the manual installation:

1. Create a shared directory, hereafter referred to as <shared_directory>, that can be accessed by all nodes in the cluster for sharing the following files/folders (otherwise, they would need to be copied to the same location on each node in the cluster):
   • The DMExpressLicense.txt file obtained from the download.
   • The dmexpress sub-directory created upon the dmexpress tar file extraction.
   • The response file for the Connect silent installations (generated upon install on the first node).
2. Extract the Connect Software.
   1. Copy DMExpressLicense.txt and the dmexpress tar file to the <shared_directory>.
   2. Extract the contents of the dmexpress tar file in the <shared_directory> on your UNIX system:

      tar xvof dmexpress_<Connect version>-1_<language>_linux_2-6_x86-64_64bit.tar 

      This creates a dmexpress/ directory under the current directory, hereafter referred to as the <connect_download_directory>.