Skip to main content

Pentaho+ documentation has moved!

The new product documentation portal is here. Check it out now at docs.hitachivantara.com. 

 
Version Data Integration and Analytics
Hitachi Vantara Lumada and Pentaho Documentation

Scalability and High Availability (HA) on Pentaho Worker Nodes

  1. Last updated
  2. Save as PDF

Parent article

Here are some points to remember when working with scalability and High Availability (HA) on Pentaho Worker Nodes.

  • For async execution mode, the PDI-Job scales automatically to handle load.
  • For sync execution mode, PDI-Service can be scaled manually to handle large number of request for execution.
  • Additional nodes can be added to the PentahoWorkerNode cluster to handle the load.

For HA, scaling CER to more than one node provides redundancy and avoids single point of failure. In this setup, it is recommended to configure a external load balancer (like HaProxy) to distribute the request between multiple instances of CER.

CER is capable of routing the request to PDI-Service running on any node. Similarly, based on resources, it can create an instance of PDI-Job on any node in the cluster. GUID-A715A860-806F-47F8-9130-318C56663AA1-low.png

Parent Topic
Child Topics

Execution Flow

The following process illustrate the execution flow.

  1. Client uses the REST API to send the request to PentahoWorkerNode.
  2. The request is routed via the load balancer to one of the ContentExecutionRouter(CER).
  3. Based on the request url, the request is routed to either PDI-Service or PDI-Job.
    1. In case of async execution mode, CER will create an instance of PDI-job that can be scheduled on any node in the cluster depending on the resource availability.
    2. In case of sync execution mode, CER will use the service discovery to find the instance of PDI_Service and route the request
  4. The PDI-service/PDI-Job will use the mounted volume to access the KTR/KJB and execute it.
Parent Topic

Failure Scenarios
Node Failure

Any single node can fail temporarily and the cluster will continue working. If the node returns, it will be handled automatically. If the failed node is a master, it can be replaced with another master.

A cluster with 3 master node will need minimum of 2 masters to function properly.

Any jobs running on the node that goes down will be marked as failed. They will have to resubmitted.

Parent Topic

CER Scaling and Loadbalancing

Once your multi-node cluster has been setup, scale out the CER service to every master node. Once CER service has been scaled out, place a load balancer in front of the cluster that points to the IPs of the master nodes. Once the load balancer has been setup, all requests should be sent to the IP of the loadbalancer. Thus, if you wish to use PS, reconfigure<pentaho-server-install-dir>/pentaho-server/pentaho-solutions/system/karaf/etc/pentaho.worker.nodes.cfg to have the wn-hostname set to the IP of the loadbalancer.

Parent Topic