Set up the Adaptive Execution Layer (AEL)

You must meet the following requirements for using the AEL daemon and operating an alternative engine for transformations:

• Pentaho 8.3 or later installation. See Pentaho installation.
• Cloudera 5.13 or later or Hortonworks 2.6 or later distribution of Hadoop.
• Spark Client 2.2 or later.
• Pentaho Spark application.
• If you are configuring AEL for use with Cloudera, Hortonworks, MapR, or Amazon EMR, review Vendor-Supplied Clients.

Parent Topic

• Set up the Adaptive Execution Layer (AEL)

Child Topics

• Pentaho installation
• Spark client
• Pentaho Spark application

Procedure

1. Navigate to the data-integration/adaptive-execution/config directory and open the application.properties file.

2. Set the following properties for your environment:

   1. Set the sparkHome property to the Spark 2 filepath on your local machine.

   2. Set the sparkApp property to the data-integration directory.

   3. Set the hadoopConfDir property to the directory containing the *site.xml files.

3. Save and close the file.

4. Navigate to the data-integration/adaptive-execution folder and run the daemon.sh command from the command line interface.

Parent Topic

• Set up the Adaptive Execution Layer (AEL)

Procedure

1. Navigate to the data-integration/adaptive-execution/config directory and open the application.properties file.

2. Set the following properties for your environment:

   Property	Value
   websocketURL	The fully-qualified domain name of the node where the AEL daemon is installed. For example, websocketURL=ws://localhost:${ael.unencrypted.port}
   sparkHome	The path to the Spark client folder on your cluster
   sparkApp	The data-integration directory
   hadoopConfDir	The directory containing the *site.xml files. This property value tells Spark which Hadoop/YARN cluster to use. You can download the directory containing the *site.xml files using the cluster management tool, or you can set the hadoopConfDir property to the location in the cluster.
   hadoopUser	The user ID the Spark application will use, if you are not using security.
   sparkMaster	yarn
   assemblyZip	hdfs:$HDFS_SPARK_EXECUTOR_LOCATION

3. Save and close the file.

4. Copy the pdi-spark-executor.zip file to your HDFS cluster, as in the example below.

   $ hdfs dfs put pdi-spark-executor.zip /opt/pentaho/pdi-spark-executor.zip

5. Run the pdi-daemon startup script, daemon.sh, from the command line interface.

6. (Optional) Perform the following steps to manually start the AEL daemon.

   You can manually start the AEL daemon by running the daemon.sh script. By default, this startup script is installed in the data-integration/adaptive-execution folder, which is referred to as the variable PDI_AEL_DAEMON_HOME.

   1. Navigate to the data-integration/adaptive-execution directory.

   2. Run the daemon.sh script.

      The daemon.sh script supports the following commands:

      Command	Action
      daemon.sh	Starts the daemon as a foreground process.
      daemon.sh start	Starts the daemon as a background process. Logs are written to the PDI_AEL_DAEMON_HOME/daemon.log file.
      daemon.sh stop	Stops the daemon.
      daemon.sh status	Reports the status of the daemon.

Parent Topic

• Set up the Adaptive Execution Layer (AEL)

Procedure

1. Navigate to the data-integration/adaptive-execution/config directory and open the application.properties file.

2. Set the sparkEventLogEnabled property to true.

   If this field is missing or set to false, Spark does not log events.

3. Set the sparkEventLogDir property to a directory where you want to store the log.

   This can either be a file system directory (for example, file:///users/home/spark-events), or an HDFS directory (for example, hdfs:/usrs/home/spark-events).

4. Set the spark.history.fs.logDirectory property to point to the same event log directory you configured in the previous step.

Results

Next steps

Parent Topic

• Set up the Adaptive Execution Layer (AEL)

Additional configuration steps may be required when using AEL with a vendor’s version of the Spark client.

Parent Topic

• Set up the Adaptive Execution Layer (AEL)

Child Topics

• Cloudera
• Hortonworks
• MapR
• Amazon EMR

export SPARK_DIST_CLASSPATH=$(hadoop classpath)

export HDP_VERSION=${HDP_VERSION:-2.6.0.3-8}

Parent Topic

• Set up the Adaptive Execution Layer (AEL)

Parent Topic

• Set up the Adaptive Execution Layer (AEL)