Skip to main content
Pentaho Documentation

Enable Logging

The logging functionality in Data Integration enables you to more easily troubleshoot complex errors and failures, and measure performance. To turn on logging in Data Integration, follow the below procedure.

  1. Create a database or table space called pdi_logging.
  2. Start Spoon, and open a transformation or job for which you want to enable logging.
  3. Go to the Edit menu and select Settings... The Settings dialog appears.
  4. Select the Logging tab.
  5. In the list on the left, select the function you want to log.
  6. Click the New button next to the Log Connection field. The Database Connection dialogue appears.
  7. Enter your database connection details, then click Test to ensure that they are correct. Click OK when you are done.
  8. Look through the list of fields to log, and ensure that the correct fields are selected.
    Warning: Monitoring the LOG_FIELD field can negatively impact BA Server or DI Server performance. However, if you don't select all fields, including LOG_FIELD, when configuring transformation logging, you will not see information about this transformation in the Operations Mart logging.
Logging is enabled for the job or transformation.
When you run a job or transformation that has logging enabled, you have the option of choosing the log verbosity level in the execution dialogue:
  • Nothing Do not record any output
  • Error Only show errors
  • Minimal Only use minimal logging
  • Basic This is the default level
  • Detailed Give detailed logging output
  • Debug For debugging purposes, very detailed output
  • Row level Logging at a row level. This will generate a lot of log data

If the Enable time option is enabled, all lines in the logging will be preceded by the time of day.

Log Rotation

This procedure assumes that you do not have or do not want to use an operating system-level log rotation service. If you are using such a service on your Pentaho server, connect to the BA Server and Data Integration Server and use that instead of implementing this solution.

The Business Analysis and Data Integration servers use the Apache log4j Java logging framework to store server feedback. The default settings in the log4j.xml configuration file may be too verbose and grow too large for some production environments. Follow these instructions to modify the settings so that Pentaho server log files are rotated and compressed.

  1. Stop all relevant Pentaho servers.
  2. Download a .zip archive of the Apache Extras Companion for log4j package: Apache Logging Services.
  3. Unpack the apache-log4j-extras JAR file from the zip archive, and copy it to the following locations:
    • Business Analytics Server: /tomcat/webapps/pentaho/WEB-INF/lib/
    • Data Integration Server: /tomcat/webapps/pentaho-di/WEB-INF/lib/
  4. Edit the log4j.xml settings file for each server that you are configuring. The files are in the following locations:
    • BA Server: /tomcat/webapps/pentaho/WEB-INF/classes/
    • DI Server: /tomcat/webapps/pentaho-di/WEB-INF/classes/
  5. Remove all PENTAHOCONSOLE appenders from the configuration.
  6. Modify the PENTAHOFILE appenders to match the log rotation conditions that you prefer. You may need to consult the log4j documentation to learn more about configuration options. Two examples that many Pentaho customers find useful are listed:

    Daily (date-based) log rotation with compression:

    <appender name="PENTAHOFILE" class="org.apache.log4j.rolling.RollingFileAppender">
        <!-- The active file to log to; this example is for BA/DI Server.-->
        <param name="File" value="../logs/pentaho.log" />
        <param name="Append" value="false" />
        <rollingPolicy class="org.apache.log4j.rolling.TimeBasedRollingPolicy">
            <!-- See javadoc for TimeBasedRollingPolicy -->
            <param name="FileNamePattern" value="../logs/pentaho.%d.log.gz" />
        </rollingPolicy>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%d %-5p [%c] %m%n"/>
        </layout>
    </appender>

    Size-based log rotation with compression:

    <appender name="PENTAHOFILE" class="org.apache.log4j.rolling.RollingFileAppender">
        <!-- The active file to log to; this example is for BA/DI Server.-->
        <param name="File" value="../logs/pentaho.log" />
        <param name="Append" value="false" />
        <rollingPolicy class="org.apache.log4j.rolling.FixedWindowRollingPolicy">
            <param name="FileNamePattern" value="../logs/pentaho.%i.log.gz" />
            <param name="maxIndex" value="10" />
            <param name="minIndex" value="1" />
        </rollingPolicy>
        <triggeringPolicy class="org.apache.log4j.rolling.SizeBasedTriggeringPolicy">
            <!-- size in bytes -->
            <param name="MaxFileSize" value="10000000" />
        </triggeringPolicy>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%d %-5p [%c] %m%n" />
        </layout>
    </appender>
  7. Save and close the file, then start all affected servers to test the configuration.
You have an independent log rotation system in place for all modified Pentaho servers.