This is a walkthrough for adding a task that automatically starts the Matillion instance at a certain time and another task that shuts down (deallocates) the resources of the Matillion instance via an API endpoint. These two tasks are intended to be deployed in Azure environments as a cost-saving mechanism and are made possible by Microsoft’s recent preview of automation tasks under the Virtual Machine page.

Note: To understand more about why you would want to deallocate a virtual machine in Azure vs. just shutting it down, you can read more here.

The Case for Automated Tasks

You would want to use this walkthrough if you are trying to reduce costs on a Matillion instance by only having the virtual machine on during the times that are required for it to run its scheduled tasks. This will help greatly cut down the Azure compute costs and the Matillion costs.

By automating this process, you can ensure your VM is on at a certain time and your Matillion instance only shuts off after it has completed its scheduled tasks. If you had the shutdown on a schedule as well and you set the schedule too aggressively, you may run the risk of shutting down the server while a job was still executing, and if you built in too much buffer time, you’re paying for compute you aren’t using.

Prerequisites

Before you perform the following steps, you will want to ensure the Matillion virtual machine has a static IP address assigned to it. If you don’t have a static IP configured, when you deallocate the VM and start it again, the public IP address will change. Here is an article to follow to ensure you have the static IP configured properly.

How to Perform the Process

Step One 

Logged into the Azure portal, navigate to your virtual machine that is running Matillion. Scroll down on the left side, and you will see Tasks (preview) under the Automation section as highlighted below:

Click on the Add Task button in the middle of the screen:

This will bring up four automation templates to choose from. Although this walkthrough will only show you how to create the shutdown task, it is recommended that you also create a start task and put that on a schedule to run about 15 minutes before the first schedule in Matillion is set to begin.

Click Select on Deallocate Virtual Machine:

You should now see a wizard asking you to enter your credentials for the two services your task will need. Simply click the ellipsis, and sign in as highlighted below for both services. Once this is done and the services have a green check in the status column, click Next:

You will now be on the Add a task screen of the wizard. Even though we are not intending to use a schedule for shutting down the Matillion VM, just enter an arbitrary time and time zone (leaving it as UTC has been known to fail when creating the task). Once this is configured, verify the occurrence is 1 and the frequency is Day.

You can also configure the task to email on its successful and failed runs, though additional permissions may be needed.

Click Review + create:

Once your task has been created, you should see it in the Tasks (preview) pane. If this is for the Scheduled Auto Start task, you are done. To change the time, you can open the ellipsis, click Edit in-line to open the wizard again and adjust the time if needed.

Step 2 – Adjust Logic App 

To change the deallocation task from a schedule to an API endpoint trigger, we will need to modify the Logic App this wizard created. Open the ellipsis and click Open in Logic Apps for the deallocation task:

You will now see the Logic Apps Designer pane. On the Recurrence component, open the ellipsis and click Delete:

You should now see a search box to look for another component to use as the trigger. Search for Request and select it:

The only thing you need to do to configure this component is to add a new parameter of Method and specify that the call should be a Post:

After the Request component, click the plus (+) sign between it and the Deallocate component. Search for a response and configure the message you would like to return on a successful Post request. I also have added a delay to ensure Matillion will not shut down immediately and has time to finish writing the logs to disk before the actual deallocation occurs. You can do this by simply using the plus (+) button again and searching for the Delay component:

At this point, to generate the Endpoint URI and SAS key, just click Save on the top bar and open the Request component. Click the copy icon to copy the URI to your clipboard:

Step 3 – Create Matillion Job

We now can go to Matillion and create a job. Create an orchestration job and add a Python component to the canvas. Switch the interpreter to Python 2, and open the script window. Paste the following below and replace the URI Address with your copied URI:

import response
#Make API call
response = request.post('URI Address')
data = response.content
print(data)

Now you should be able to execute the orchestration job, and your Matillion instance will start deallocating after the wait timer you configured in the Logic App.

You could create a shared job out of this orchestration, so you can easily add it as a component to the end of your final scheduled job for the day. This will assist the client in managing their VM and Matillion spend automatically. You should also couple this deallocation method with an auto-start task in Azure that is scheduled for about 15 minutes before the first scheduled job kicks off.

Save on Time and Cost

With these two tasks configured, we will be able to better control the compute costs in Azure, as well as the Matillion Credit costs, in a more dynamic and cost-effective way.