This blog post is Human-Centered Content: Written by humans for humans.

In the field of data engineering, it sometimes feels like new tools are being introduced merely for the sake of novelty, only to have the exact same benefits and pitfalls as the previous new, shiny tool. When I first heard about “dlt” (data load tool), I will admit that I was a bit skeptical.

Why would I learn how to use yet another tool to load data from source to destination? Aren’t there already dozens of viable tools out there that already do this pretty well (Matillion, Fivetran, Openflow, Airflow, etc.)?

What is dlt, Anyway?

To clear any confusion right off the bat, I am not talking about Delta Live Tables within Databricks — these are completely separate entities (which just happen to use the exact same three letter acronym).

dlt stands for Data Load Tool. In their own words, dlt is an “open-source Python library that loads data from various, often messy data sources into well-structured datasets (https://dlthub.com/docs/intro)”. It was first launched in 2023, and it has only grown in popularity over the past few years.

dlt utilizes Python to create code-first pipelines, which has certain advantages:

Oh, and did I mention that it’s completely, 100% free?

In ELT terms, dlt is the Extract and the Load. The raw data is loaded into a data warehouse, where other code-first tools (such as dbt) can be used to Transform the data into its final form. This structure of dlt + dbt can provide a powerful and flexible framework for the entire ELT process.

But there is a large chasm that lies between theory and practice. I’ll walk you through a simple step-by-step guide to create your very first dlt pipeline from raw API source data into Snowflake. So let’s cross over!

Step 0: Pipeline Overview

Step 1: Setting Up Environments

Snowflake Instance — Free Trial Account

Visual Studio Code — Virtual Python Environment

VS Code is the IDE we will be using for this example, but feel free to use your code editor of choice. Below are the recommended dltHub installation steps on a Windows machine:

Step 2: Snowflake Authentication With dlt

For the purposes of this demo, we are placing our snowflake credentials in secrets.toml. In real-world production environments, you should follow security best practices such as using environment variables, a secrets manager (e.g. AWS Secrets Manager or Azure Key Vault), or Snowflake key-pair authentication. But for now, we’ll just hard-code values in secrets.toml.

Note: Never commit secrets.toml to Git. Add .dlt/ to your .gitignore file when using version control if it wasn’t automatically added with the dlt init [...] command.

Add your Snowflake credentials to secrets.toml, filling in the correct values. If you ran dlt init [...] earlier, then all you need to do is modify all of the "<configure me>" values.

[destination.snowflake.credentials]
database = "DLT_PRACTICE_DB"       # name of Step 1 Snowflake database
password = "your_password"
username = "your_username"
host = "your_account_identifier"	# ex: “ABCDEFG-HJK01234” (see note below)
warehouse = "COMPUTE_WH"           # default warehouse on trial accounts
role = "ACCOUNTADMIN"		     # determines ownership of schema/tables

Step 3: Create the Pipeline

We are going to create a new python file in our project called my_first_pipeline.py — this is an extremely simple example, so we are not going to touch pagination, authentication or chaining requests today (see dlthub’s RESTClient documentation to learn more).

For now, simply paste in the following code into this newly created file:

my_first_pipeline.py

import dlt
import requests


# -------------------------------------------------------
# SOURCES
# A "resource" is one endpoint / one table worth of data.
# The @dlt.resource decorator tells dlt "this is a data source".
# "yield" sends records one at a time to dlt — think of it
# like returning data lazily rather than all at once.
# -------------------------------------------------------

@dlt.resource(name="users", write_disposition="replace")
def get_users():
    """Fetches all users from JSONPlaceholder."""
    response = requests.get("https://jsonplaceholder.typicode.com/users")
    response.raise_for_status()  # raises an error if the request failed
    yield response.json()        # yield the list of user dicts to dlt

@dlt.resource(name="posts", write_disposition="replace")
def get_posts():
    """Fetches all posts from JSONPlaceholder."""
    response = requests.get("https://jsonplaceholder.typicode.com/posts")
    response.raise_for_status()
    yield response.json()

@dlt.resource(name="comments", write_disposition="replace")
def get_comments():
    """Fetches all comments from JSONPlaceholder."""
    response = requests.get("https://jsonplaceholder.typicode.com/comments")
    response.raise_for_status()
    yield response.json()


# -------------------------------------------------------
# PIPELINE
# This is the runner. It connects your resources to a destination.
# - pipeline_name:  just a label, also used for internal state tracking
# - destination:    where data goes — "snowflake", "duckdb", etc
# - dataset_name:   the schema your tables will live in inside Snowflake
# -------------------------------------------------------

pipeline = dlt.pipeline(
    pipeline_name="jsonplaceholder_pipeline",
    destination="snowflake",
    dataset_name="raw"
)


# -------------------------------------------------------
# RUN
# Pass your resources to pipeline.run() as a list.
# dlt will infer the schema, create the tables, and load the data.
# -------------------------------------------------------

if __name__ == "__main__":
    load_info = pipeline.run([get_users(), get_posts(), get_comments()])
    print(load_info)  # shows what was loaded, row counts, any errors

• Note about the dlt.pipeline(...) function:
  • dataset_name = "raw" is where you are declaring the schema name to be “RAW” within your Snowflake database which you created. You could name this whatever you like (e.g. “example_api_data_bronze”).
  • destination = "snowflake" is how the code knows to reference the secrets.toml file for proper Snowflake credentials.

Step 4: Execute the Pipeline

Step 5: Validate Results in Snowflake

Step 6: Experiment with Additional Features

Congratulations on creating your very first dlt pipeline into Snowflake!

Here are some ideas to experiment with as you venture forward with dlt:

Conclusion

If you followed along up to this point and tried your hand at some mini experiments, you are undoubtedly aware of the powerful flexibility of dlt. I found it remarkably refreshing to move away from clunky drag-and-drop user interfaces of the past and embrace the future of data ingestion: Code-first solutions.

Now that you have added another data ingestion tool to your toolbelt, you must use it wisely. Because remember, the best tool for the job always depends on the full context of the situation. Practical data engineering relies on your ability to know when to wield a hammer, and when to wield a screwdriver.

Never stop learning.