Remote git integration is a way of backing up a local git repository to an online git management provider, such as GitHub, GitLab or Azure DevOps. If a customer has multiple Matillion instances for any reason, this also allows them to make changes on one instance and sync those changes to others.

SSH keys are a method of authentication that can be used with git to improve security and remove any reliance on user names or passwords that may change.

Each project in Matillion leverages its own local git repository and thus must connect to a unique remote repository. Unless the two projects are supposed to contain the same content, such as in a blue/green deployment or a dev/test/prod setup, sharing remote repositories between projects does not make much sense. However, many git management providers allow you to group multiple remote repositories into a folder, so you should still be able to keep your remote repositories organised.

Why?

When setting up git with Matillion, it is useful to align your local repository with a remote one. The main benefit here is the ability to safely have your repository backed up outside the Matillion environment.

Requirements

• Enterprise edition of Matillion
• A git management provider – We will use GitLab for this article; however, the process will be similar for other git management providers.

Best Practices

• Create private SSH keys from within the Matillion VM to ensure they are backed up in an accessible location for any Matillion admins.
• If possible, configure project-specific deploy keys in your git provider that are shared amongst the various repositories. If your git management provider does not allow for project-specific deploy keys, the SSH key should be set up for a specific user account; ideally, a service user specifically for Matillion/git integration.

Important Gotchas

• Matillion only supports RSA SSH keys in the PEM format. If you are leveraging another key, you can convert it to PEM format with the following command:

  ssh-keygen -p -f path\to\ssh\key -m pem

  For example, we could convert an existing private key called id_rsa as follows:

  ssh-keygen -p -f ~\.ssh\id_rsa -m pem

• Matillion only stores a single private key at a time and shares this amongst ALL projects on the Matillion instance on a PER USER basis. If you wish to set up git integration for multiple projects against multiple remote git repositories, they all must share the same SSH key pair. Otherwise, you will find that entering a new private SSH key to configure the remote git integration for one project will overwrite the stored key for other projects and break their remote git integration. If you wish for multiple users to be able to interact with the remote repository, each user will need to configure their git SSH key pair or authentication separately.

Let’s get into the practical steps involved in the process.

Initialise Local Git Repository in Matillion

From within the project, select Project > Git:

Select Init local repository to create a new local repository. If you already have a remote repository storing a Matillion project you want to clone, you can clone using standard user name and password credentials as a one-off to get started and skip the rest of this section:

Populate your Matillion user name and email address then hit OK. It is okay to leave the commit information as Initial commit:

Access the Git Integration menu by selecting the git icon next to the Project dropdown to confirm that you have a local repository active:

Initialise Remote Git Repository in GitLab

Within InterWorks, we already have a group in GitLab for our Matillion projects. The group is called Matillion and sits within the DataManagement group. This group contains several subgroups depending on the work stream:

From within your desired subgroup, select New project. For this example, we will use the Sandboxes subgroup. If you do not have access to the New project option, reach out to the administrator for your git provider, which is often your IT Support team:

Create a blank project:

Give your project a name and project slug. The project slug is the text at the end of the URL to access this particular repository. Add a description, too, if you like. Then select Create project, leaving Initialize remote with a README unselected. InterWorks projects are left as Internal so that other members of the team can access them if needed:

You now have an empty remote git repository in GitLab that is ready to integrate with Matillion:

Setting up an SSH Key Pair

Using a command line tool such as Bash or PowerShell, you should be able to follow these steps to create a local SSH key pair.

Above: Click the text box to learn how to enable SSH commands in Windows.

Create a new 2048 bit RSA PEM key, giving it a useful comment:

ssh-keygen -t rsa -b 2048 -C "Git-Matillion-Demo" -m pem

By default, the file will be stored in ~/.ssh in bash or your local user’s SSH repository in Windows. You can also add a password if you like. You will then be shown the key’s randomart image to confirm creation:

Copy the contents of the public key to your clipboard. You can read this public key with the following command:

cat path\to\ssh\key.pub

For example, our code is likely:

cat ~\.ssh\id_rsa.pub

A sample is shown here. You can then select this and copy it:

Within GitLab, navigate to Settings > Repository > Deploy keys, and enter your public key, ensuring that you grant it write permissions:

Retrieve the SSH address of your remote repository by clicking the Clone dropdown on the repository’s homepage and copying the URL:

Within the Matillion Git Integration menu, enter the SSH path of your remote repository under the Configure Remote button:

Copy the contents of the private key to your clipboard. You can read this private key with the following command:

cat path\to\ssh\key

For example, our code is likely:

cat ~\.ssh\id_rsa

A sample is shown here. You can then select this and copy it:

Within the Matillion Git Integration menu, enter the private key and passphrase under the Configure Default Credentials button. Set the encryption type to Encoded unless you are leveraging additional keyvault security not covered in this article:

Test the connection by pushing to your remote repository:

Confirm the successful push:

Congratulations! You have now set up a local git repository in Matillion that can sync with a remote repository using SSH key authentication.