Sometimes when you’re developing a data pipeline in Matillion ETL, you may find that a component that you’re working on just isn’t behaving as you expect. It might be producing unexpected output, or it might just be behaving in an odd way but with no error message to help you work out what’s going on.

I recently had just such an issue when I was configuring a Google Sheets component within Matillion ETL. Having set up the connection, I was expecting it to list some available data sources as input options. Instead, it presented a blank dialogue and was unable to connect to the data source I specified. There were no error messages. It just didn’t work.

I needed more data to work out what’s going on. That’s where activating component level logging really helped. With many Matillion components, we can add logging as a connection option.

Start by selecting the component and then click the three dots next to Connection Options in the properties window:

Add a new parameter Logfile with a value that specifies a location for a new logfile (/tmp/log2.txt in this example). If the file doesn’t exist, it will be created.

Add a new parameter Verbosity with a value of 5:

Having specified the location of the logfile, add a bash script component to the canvas:

Connect the script to the rest of the flow so you can run it:

Edit the bash script component to cat (display) the log file:

From this point, anything you do with the component will be logged in the file. This will include any changes you attempt to make to component properties.

Now recreate the issue with the problematic component.

Right-click the bash script and Run Component.

Expand the Tasks section and find the section where the bash component was run. Expand that and click the right arrow to the top right:

Expand the job and click the three dots:

You’ll now get a window with the log that can expand or copy/paste elsewhere:

Often the log file can give you specific error messages, and even suggest a solution.

Once you’ve finished, don’t forget to turn off logging and delete the logfile you created (rm /tmp/log2.txt in this case).

At InterWorks, we have helped clients of all sizes get the most from Matillion. We can offer support for your data and analytics infrastructure including Matillion with our KeepWatch service. Click here to find out more.