Pentaho uses the Adaptive Execution Layer (AEL) for running transformations in different engines. AEL adapts steps from the transformation you develop in PDI to native operators in the engine you select for your environment.For example, AEL currently supports a Spark engine, which is better suited for running big data transformations in a Hadoop cluster. Additional engines will be added to AEL in future releases.
Set Up AEL
AEL must be configured before using a supported optional engine in your transformation's run configuration.
Refer your Pentaho or IT administrator to Setting Up the Adaptive Execution Layer for more details.
Once configured, you can select the AEL optional engine for the transformation. See Run Configurations for more details.
As a general rule, most PDI transformation steps function as documented while running under AEL. Some PDI steps are optimized for a specific engine, while other steps have known exceptions when you run a transformation under AEL.
The following topics help to extend your knowledge of the Adaptive Execution Layer beyond basic setup and use:
- Specify Additional Spark
You can define additional Spark properties within the application.properties file or as run modification parameters within a transformation.
- Configuring AEL with Spark in a
If your AEL daemon server and your cluster machines are in a secure environment like a data center, you may only want to configure a secure connection between the PDI client and the AEL daemon server.