Retrieving Logs from a Matillion Virtual Machine via SSH

Data

Retrieving Logs from a Matillion Virtual Machine via SSH

Often when working with Matillion Support or investigating other Matillion issues, you will need to retrieve the logs from the Matillion virtual machine. This could be for several reasons, including:

  • The logs available through the UI may be trimmed whilst the virtual machine itself stores the unabridged logs and archives.
  • You may be unable to access the UI and need to download the logs directly from the virtual machine to identify the issue.

How to Download Logs from the UI

If the reasons above do not apply to you, you can download the current log through the Admin tab of the Matillion UI, assuming you have the necessary privileges for your Matillion login. Otherwise, continue reading to find out how to download the logs directly from the virtual machine.

Prerequisites

  • Administrator credentials to your Linux virtual machine. This could be either:
    • A username and password
    • A username and private SSH key
  • The OpenSSH Client optional service must enabled on your machine, and OpenSSH must be added to your PATH environment variable. You can read how to do that here.

Copying the Matillion Logs to a Temp Directory on the Virtual Machine

Before we download the log file, we first move it to a temporary directory and modify the security of the file, so it can be downloaded. For our example, we will download the main log file (catalina.out); however, you could follow this process for any of the archived files, too (such as catalina.out-2021-08-22.gz).

First, log into your Matillion virtual machine via SSH as demonstrated in this guide. Leverage the following command to make a new temporary directory that will store our copied log file:

sudo mkdir /tmp_<date>

For our example, we use the following command:

sudo mkdir /tmp_2021-09-01

We can then navigate to the folder to verify its existence with the cd command, and use the pwd command to verify which directly we are currently in:

Navigate to the folder on the Matillion VM that stores the logs, using the following command:

cd /var/log/tomcat8

Use the list command ls to verify that the file is within the desired folder. The log file is called catalina.out:

Copy the catalina.out log file into the temporary directory that was created earlier using the following command:

sudo cp catalina.out /tmp_<date>

For our example, this command is:

sudo cp catalina.out /tmp_2021-09-01

Navigate to the temporary directory using the cd command. For our example, this command is:

cd /tmp_2021-09-01

To be safe, use the ls command again to verify that the catalina.out file exists in your temporary directory:

Prepare the file for download by using the chmod command to change the privileges on the file to 755:

sudo chmod 755 catalina.out

Downloading the Log File Using SCP

With a single command, we can leverage SCP (Secure Copy Protocol) to copy a file from a Linux virtual machine to our local machine. SCP is downloaded as part of the standard OpenSSH Client optional service discussed in the prerequisites.

Downloading Files with SCP with SSH Password Authentication

If authenticating via an SSH with a password, this can be achieved by entering your password after using a command such as:

scp user@host:path/to/remote/file path/to/local/destination/filename

For example, my command could be:

scp MatillionAdmin@xx.xx.xx.xx:/tmp_2021-09-01/catalina.out "C:\Users\chris.hastie\Downloads\catalina.out"

The file will now be sitting safely in your local directory, where you can either read it yourself or zip it and send it through to Matillion support.

Downloading Files with SCP with SSH Key Authentication

If authenticating via an SSH private key, this can be achieved with a command such as:

scp -i path/to/ssh/key user@host:path/to/remote/file path/to/local/destination/filename

For example, my command could be:

scp -i "C:\...\matillion-ssh-key.pem" centos@xx.xx.xx.xx:/tmp_2021-09-01/catalina.out "C:\Users\chastie\Downloads\catalina.out"

The file will now be sitting safely in your local directory, where you can either read it yourself or zip it and send it through to Matillion support.

More About the Author

Chris Hastie

Data Engineer
How to Clean up Temporary Matillion Stages in Snowflake When using Matillion to load data into Snowflake, temporary stages are often created in Snowflake to store the data before it lands in ...
How to Configure Remote Git Integration for Matillion with SSH Keys Remote git integration is a way of backing up a local git repository to an online git management provider, such as GitHub, GitLab or ...

See more from this author →

Subscribe to our newsletter

  • I understand that InterWorks will use the data provided for the purpose of communication and the administration my request. InterWorks will never disclose or sell any personal data except where required to do so by law. Finally, I understand that future communications related topics and events may be sent from InterWorks, but I can opt-out at any time.
  • This field is for validation purposes and should be left unchanged.

InterWorks uses cookies to allow us to better understand how the site is used. By continuing to use this site, you consent to this policy. Review Policy OK

×

Interworks GmbH
Ratinger Straße 9
40213 Düsseldorf
Germany
Geschäftsführer: Mel Stephenson

Kontaktaufnahme: markus@interworks.eu
Telefon: +49 (0)211 5408 5301

Amtsgericht Düsseldorf HRB 79752
UstldNr: DE 313 353 072