Skip to main content

Pentaho+ documentation has moved!

The new product documentation portal is here. Check it out now at docs.hitachivantara.com. 

 
Version Data Integration and Analytics
Hitachi Vantara Lumada and Pentaho Documentation

Specify additional Spark properties

  1. Last updated
  2. Save as PDF

Parent article

You can define additional Spark properties in two ways:

  • Using the application.properties file
  • Using run modification parameters within a transformation

Any property that starts with spark. is passed directly to the Spark configuration.

Parent Topic
Child Topics

Use the properties file

Within the application.properties file, you may add any number of Spark configuration properties, as detailed at: https://spark.apache.org/docs/latest/configuration.html

Parent Topic

Use run modification parameters

You may also define any additional Spark configuration properties as run modification parameters within a transformation.

Learn more

Parent Topic

Order of processing

Because Spark properties can be set in multiple ways, it is important to understand the following order of processing:

  1. Spark properties set on the Pentaho Server apply to all users and all transformations.
  2. Spark properties specified within a KTR apply to the specific user who runs the transformation. For example, if a user wants to change the memory size, they can embed the appropriate Spark parameter in the KTR so that it executes when they run the transformation.
NoteIf an identical property is also set on the Pentaho Server, the user's KTR takes precedence.
Parent Topic