Adaptive Execution Layer
Pentaho uses the Adaptive Execution Layer (AEL) for running transformations in different engines. AEL adapts steps from the transformation you develop in PDI to native operators in the engine you select for your environment.For example, AEL currently supports a Spark engine, which is better suited for running big data transformations in a Hadoop cluster. Additional engines will be added to AEL in future releases.
Set Up AEL
AEL must be configured before using a supported optional engine in your transformation's run configuration.
Refer your Pentaho or IT administrator to Setting Up the Adaptive Execution Layer for more details.
Use AEL
Once configured, you can select the AEL optional engine for the transformation. See Run Configurations for more details.
Feature Parity
As a general rule, most PDI transformation steps function as documented while running under AEL. Some PDI steps are optimized for a specific engine, while other steps have known exceptions when you run a transformation under AEL.
See the following articles for more information:
- PDI steps with AEL considerations
A list of the PDI steps that function differently under AEL, depending upon the complexity of your transformation.
- PDI steps optimized for AEL-Spark
A list of the PDI steps that are optimized for use with AEL-Spark.
Advanced topics
The following topics help to extend your knowledge of the Adaptive Execution Layer beyond basic setup and use:
- Specify Additional Spark
Properties
You can define additional Spark properties within the application.properties file or as run modification parameters within a transformation.
- Configuring AEL with Spark in a
Secure Cluster
If your AEL daemon server and your cluster machines are in a secure environment like a data center, you may only want to configure a secure connection between the PDI client and the AEL daemon server.
Troubleshooting
See our list of common problems and resolutions.