Sling with dbt: Load Raw Tables Before dbt Runs

Fritz Larco avatar
Fritz Larco
•
Cover for Sling with dbt: Load Raw Tables Before dbt Runs

Sling is the extract and load step before dbt. It copies tables from databases, files and APIs into a raw schema of your warehouse. dbt then reads that schema as sources and builds models. The two tools do not call each other. They share the warehouse.

Source DB --(sling run)--> raw schema --(dbt build)--> models

This guide shows the full cycle with real runs: a SQLite application database, a DuckDB warehouse, Sling for the load, and dbt Core with dbt-duckdb for the models. For Postgres, Snowflake or BigQuery, only the connection changes. You will see:

How do I load raw tables for dbt with Sling?

Write one replication file with one stream for each source table. Install Sling with curl -fsSL https://slingdata.io/install.sh | bash or pip install sling. Then define the connections in ~/.sling/env.yaml:

connections:
  APP_DB:
    type: sqlite
    instance: /data/app.db
  WAREHOUSE:
    type: duckdb
    instance: /data/warehouse.duckdb

The replication loads orders incrementally and reloads the small customers table on each run:

# replication.yaml
source: APP_DB
target: WAREHOUSE

defaults:
  mode: incremental
  object: raw.{stream_table}
  primary_key: [id]
  update_key: updated_at

streams:
  main.orders:
  main.customers:
    mode: full-refresh

{stream_table} puts each source table name into the object, so the targets are raw.orders and raw.customers. The first run creates both tables:

$ sling run -r replication.yaml
INF [1 / 2] running stream main.orders
INF inserted 10000 rows into "raw"."orders" in 4 secs [2,377 r/s] [513 kB]
INF [2 / 2] running stream main.customers
INF inserted 500 rows into "raw"."customers" in 1 secs [485 r/s] [25 kB]
INF Sling Replication Completed in 5s | APP_DB -> WAREHOUSE | 2 Successes | 0 Failures

After the first run, orders is incremental. Sling reads only the rows with an updated_at greater than the highest value in raw.orders, and merges them on id. dbt does not need to remove duplicates from the raw table. The upserts guide shows the merge in detail, and the replication modes reference lists all modes.

How does dbt read the tables that Sling loads?

Declare the raw schema as a dbt source. The sources.yml below also adds two dbt tests on raw.orders.id:

# shop/models/sources.yml
version: 2

sources:
  - name: raw
    schema: raw
    tables:
      - name: orders
        columns:
          - name: id
            data_tests: [unique, not_null]
      - name: customers

A model uses the sources with {{ source() }}:

-- shop/models/customer_revenue.sql
select
    c.id as customer_id,
    c.country,
    count(*) as paid_orders,
    sum(o.amount) as revenue
from {{ source('raw', 'orders') }} o
join {{ source('raw', 'customers') }} c on c.id = o.customer_id
where o.status = 'paid'
group by 1, 2
$ dbt run --project-dir shop
Found 1 model, 2 sources, 512 macros
1 of 1 START sql table model main.customer_revenue ............................. [RUN]
1 of 1 OK created sql table model main.customer_revenue ........................ [OK in 0.08s]
Done. PASS=1 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=0 TOTAL=1

We ran this with dbt Core 1.12.7 and dbt-duckdb 1.11.1.

How do I handle schema changes before dbt sees them?

Sling adds new source columns to the raw table on incremental runs. The add_new_columns target option is true by default. Set adjust_column_type: true if Sling must also widen a column type when the source type grows:

defaults:
  target_options:
    add_new_columns: true      # default
    adjust_column_type: true   # default is false

A dbt model with an explicit column list does not change when a new column comes in. A model with select * gets the new column on its next run.

Can Sling use my dbt profiles.yml?

Yes. Sling reads ~/.dbt/profiles.yml, or profiles.yml in the folder that DBT_PROFILES_DIR names. Each output becomes a connection named PROFILE_TARGET. Sling also expands {{ env_var('...') }} values. For this profile:

shop:
  target: prod
  outputs:
    prod:
      type: postgres
      host: db.example.com
      user: dbt_user
      password: "{{ env_var('DBT_PG_PASSWORD') }}"
      port: 5432
      dbname: analytics
      schema: raw
$ sling conns list
+-----------+-----------------+-------------------+
| CONN NAME | CONN TYPE       | SOURCE            |
+-----------+-----------------+-------------------+
| APP_DB    | DB - SQLite     | sling env yaml    |
| LOCAL     | FileSys - Local | built-in          |
| SHOP_PROD | DB - PostgreSQL | dbt profiles yaml |
| WAREHOUSE | DB - DuckDB     | sling env yaml    |
+-----------+-----------------+-------------------+

Use target: SHOP_PROD in the replication, and Sling writes to the same database that dbt reads. You keep one set of warehouse credentials. In our test, Sling did not load a dbt-duckdb output that uses path. For DuckDB, define the connection in the Sling env.yaml with instance, as above. The environment docs list all connection sources.

How do I run dbt after Sling?

Run Sling first, then dbt, in the scheduler that you already have. If a stream fails, Sling exits with a non-zero code, and the scheduler does not start dbt.

Airflow. Two BashOperator tasks:

from datetime import datetime
from airflow import DAG
from airflow.operators.bash import BashOperator

with DAG("elt_pipeline", start_date=datetime(2026, 1, 1), schedule="@daily") as dag:
    load = BashOperator(
        task_id="sling_load",
        bash_command="sling run -r /opt/pipelines/replication.yaml",
    )
    transform = BashOperator(
        task_id="dbt_build",
        bash_command="dbt build --project-dir /opt/dbt/shop",
    )
    load >> transform

If you prefer Python, the sling package runs the same file: Replication(file_path="/opt/pipelines/replication.yaml").run(). Dagster also has a Sling integration, dagster-sling.

A Sling pipeline. If you do not have a scheduler, one pipeline file can run the load and then dbt. The command step runs any program:

# pipeline.yaml
steps:
  - type: replication
    path: replication.yaml
    id: load

  - type: command
    if: state.load.status == "success"
    command: [dbt, build, --project-dir, shop]
    print: true

Before this run, we added 100 orders to the source and changed the status of 2 old orders:

$ sling run -p pipeline.yaml
INF [1 / 2] running stream main.orders
INF inserted 102 rows into "raw"."orders" in 1 secs [74 r/s] [5.0 kB]
INF [2 / 2] running stream main.customers
INF inserted 500 rows into "raw"."customers" in 1 secs [488 r/s] [25 kB]
INF Sling Replication Completed in 2s | APP_DB -> WAREHOUSE | 2 Successes | 0 Failures
1 of 3 PASS source_not_null_raw_orders_id ...................................... [PASS in 0.08s]
2 of 3 PASS source_unique_raw_orders_id ........................................ [PASS in 0.12s]
3 of 3 OK created sql table model main.customer_revenue ........................ [OK in 0.14s]
Done. PASS=3 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=0 TOTAL=3

Sling read 102 rows, not 10,350. The dbt unique test passed, because Sling merged the 2 changed orders on id and did not append them. The model total matches the source:

$ sling conns exec WAREHOUSE "select (select count(*) from raw.orders) as raw_orders, (select count(distinct id) from raw.orders) as distinct_ids, (select sum(paid_orders) from main.customer_revenue) as paid_orders_in_model"
| RAW_ORDERS | DISTINCT_IDS | PAID_ORDERS_IN_MODEL |
|      10350 |        10350 | 7847                 |

The source also had 7,847 orders with the status paid. Pipelines and the command step need a CLI Pro token or a Platform plan. A plain sling run -r is free.

When is sling build enough without dbt?

sling build is the SQL model runner in the Sling CLI (since v1.6.0). It runs Jinja SQL models in dependency order on the same connections that replications use. It has ref(), source(), incremental models and data tests, and it has no profiles.yml or packages. We ran the same customer_revenue.sql file, with no changes, from a folder with this sling_build.yml:

target: WAREHOUSE
$ sling build run
INF [1/1] marts.customer_revenue (full-refresh) .......................... START
INF [1/1] marts.customer_revenue (full-refresh) .......................... OK (339ms)
INF Build Completed in 339ms | 1 Successes | 0 Failures

The model total was 7,847 paid orders, the same as the dbt model. The first folder of the model path (marts/) sets the schema.

Use this rule:

  • Keep dbt if your team already has a dbt project, uses dbt packages or macros from the community, or uses dbt docs and the dbt ecosystem tools.
  • Try sling build for a small set of SQL models next to your replications, when you do not want a second tool to install and configure. Models in the dbt style (is_incremental(), {{ this }}, config()) work, as the build incremental docs show.

A pipeline can also run a build project with a type: build step. See the build docs.

How is this different from Fivetran and dbt?

Sling runs on your machine or your servers, and the data goes from the source to the warehouse directly. Fivetran is a managed service that bills on monthly active rows. With Sling and dbt Core, both tools are open source and no vendor is in the data path. For the cost and feature comparison, see Sling vs Fivetran. If you move from Airbyte, see migrating from Airbyte to Sling: it shows which dbt models you can delete after the move.

Frequently asked questions

Does Sling replace dbt? No. Sling does the extract and load step: it copies tables from databases, files and APIs into your warehouse. dbt reads those raw tables and transforms them. Sling also has sling build, a SQL model runner for simple projects, but a dbt project keeps working as it is.

Can Sling use the connections in my dbt profiles.yml? Yes. Sling reads ~/.dbt/profiles.yml, or the folder in DBT_PROFILES_DIR, and shows each output as a connection named PROFILE_TARGET, for example SHOP_PROD. It also expands env_var() values. In our test, a dbt-duckdb output with path was not loaded, so define DuckDB in the Sling env.yaml.

How do I run dbt after Sling in Airflow? Run sling run -r replication.yaml in one task and dbt build in the next task. Sling exits with a non-zero code when a stream fails, so Airflow does not start the dbt task.

Can Sling run dbt for me? Yes, with a Sling pipeline. A replication step loads the data and a command step runs dbt build. Pipelines need a CLI Pro token or a Platform plan. The plain sling run -r command is free.

Is Sling free to use with dbt Core? The Sling CLI is free and open source under GPL-3.0, and replications to any supported target are free. CLI Pro adds pipelines, hooks, API sources and state-based incremental loads.