Skip to main content
Pentaho Documentation

Use Worker Nodes

The Pentaho Server can be configured for worker nodes, after completing the installation and setup process, so that transformations and jobs can be run as work items. You can take advantage of worker nodes, for example when using Kafka, to scale out across clusters when you have workload increases. Results from the transformations and jobs occur normally within the Pentaho platform, while Hitachi Content Intelligence (HCI) provides monitoring for the health and performance of the worker node instances and services.

Run Transformations and Jobs on Worker Nodes

You can run transformation and job files on worker nodes using the PDI Client or the User Console. Files can be run at regular intervals, on certain dates and times, or with different parameters using scheduling.

To monitor the worker nodes, use Chronos to view queued jobs and transformations and use Mesos to view the completed work items.

Using the PDI Client 

Run jobs or transformations on worker nodes using the PDI Client as described here.

Using the User Console 

Perform the following actions to run jobs or transformations on worker nodes using PUC:

  1. Connect to the Pentaho Repository and then save your transformation or job file to a folder in the Pentaho Repository.  
  2. Login to the User Console, click on the Browse Files button.
  3. In the Folders pane, click on the folder containing the file. 
  4. In the File pane, click on the file.
  5. In File Actions pane, select Run in background… or to schedule the file to run select Schedule…
  6. Enter a Schedule Name for the generated content and enter the Generated Content Location for the generated file location.
  7. If Scheduling, enter the RecurrenceStart Time, and Start Date.
  8. Click OK. The file will be processed on the worker nodes and then output as specified. 

Monitor the Worker Nodes 

The monitoring reporting functions for worker instances and operations of worker nodes are performed by HCI. Master instances run an essential set of services, which includes the administration application, cluster-coordination, synchronization, and service deployment. Worker instances can run any services besides those listed above. More information on a work-item by work-item basis is available on Mesos and Chronos, which are also services available on HCI.

Access the HCI Administration Application 

You will need your username and password to access the HCI Administration Application.

Perform the following actions to open the administration app:

  1. Open a web browser to go to the HCI Administration Application:

https://<HCI-instance-IP-address>:8000

  1. In the Security Realm field, select the location where your user account is defined. To log in using the local admin account, select Local.
  2. Enter your username and password then click on the Log In button. The Home page appears.

WN_HCI_Home.png

Monitor Overview 

You can view the status of the HCI system, resource allocations, instances, and services.  

Perform the following actions to monitor the system and services:

  1. Open the HCI Administration Application and log in.
  2. Click Monitoring > Dashboard > Overview.  Pentaho Worker Nodes Services and Instances are displayed, including the state and status of the service and its resource allocations.

WN_HCI_MonitoringDashboardOverview.png

Monitor Instances 

You can view all Master and Worker instances, which are identified by IP address. Single-instance systems have one master instance while multi-instance systems have either one or three master instances.

Perform the following actions to monitor instances:

  1. Open the HCI Administration Application and log in.
  2. Click Monitoring > Dashboard > Instances. The Master and Worker instances are displayed, including their state, service units, and resource allocations.

WN_HCI_MonitoringDashboardInstances.png

Monitor Services 

You can access details, status, and results about the files running on worker nodes . Services perform functions essential to the health or functionality of the system. 

Perform the following actions to monitor services:

  1. Open the HCI Administration Application and log in.
  2. Click Monitoring > Dashboard > Services. Pentaho Worker Nodes Services are displayed, including the number of instances, state, and resource allocations.

WN_HCI_MonitoringDashboardServices.png

Worker Node Queue 

The jobs and transformations run on the worker nodes as work items and can be viewed using Chronos.  

Perform the following actions to view the worker node queue for jobs and transformations:

  1. Open a web browser to go to the Chronos Administration Application:

https://<HCI-instance-IP-address>:8081

When using secure mode, a VNC viewer is needed to view the internal IP.

  1. Select a work item (WI) name in the queue to Edit, Run, Copy, or Delete that item.

WN_Chronos_Work_Item.png

Worker Node Logs 

The logging function for worker nodes is provided by Mesos.  

Perform the following actions to view logs for jobs or transformations:

  1. Open a web browser to go to the Mesos Administration Application:

https://<HCI-instance-IP-address>:5050 

  1. Locate the name of the work item that you want to view and then click the individual Sandbox link in the ActiveCompleted, or Orphan Tasks section to navigate to the page that contains the logs for that job or transformation.

WN_Mesos_Task_Page.png

  1. Click stdout to view the logs for that job or transformation. The logs contain the task progress, or failure and the reason. 

WN_Mesos_Log_Access.png

  • You can retrieve the stored Pentaho log files from 

opt/hci/log/com.pentaho.foundry.workernode

WN_Mesos_LogFiles.png