How to Run Sling on Kubernetes with a CronJob

Fritz Larco avatar
Fritz Larco
•
Cover for How to Run Sling on Kubernetes with a CronJob

To run Sling on Kubernetes, use the official slingdata/sling image in a CronJob. Put the replication file in a ConfigMap and the env.yaml in a Secret. Sling reads the incremental checkpoint from the target table, so each pod is disposable and needs no volume for state.

This guide gives the manifests and the run output. It also covers the image tags, the credentials, and the settings that keep runs from overlapping.

This page is about the open-source CLI on a schedule. If you want the Sling Platform and its agents on Kubernetes, use the Kubernetes deployment guide in the docs.

Which Sling image do you use?

The official image is slingdata/sling on Docker Hub. Its entrypoint is sling, so container args are the CLI arguments. The image runs as the non-root user sling (UID 999).

The tags split by CPU architecture:

TagArchitecture
latest, v1.6.7linux/amd64
latest-arm64, 1.6.7-arm64linux/arm64

The arm64 tags have no v prefix. If your node pool uses arm64 nodes (for example AWS Graviton), the plain latest tag does not run on them. Pin a version in production, so that a new release does not change a running pipeline without a review.

How do the Kubernetes objects fit together?

Three objects hold all of the configuration:

  • A ConfigMap holds the replication file. It has no secrets, so you can read it and compare versions.
  • A Secret holds the env.yaml with the connections, and the optional SLING_CLI_TOKEN.
  • A CronJob runs the image on a schedule, mounts the ConfigMap, and reads the Secret into environment variables.

Sling reads the Secret through SLING_ENV_YAML. This variable holds the full body of an env.yaml file (see environment variables). Thus you do not need to mount a file into the Sling home folder.

Step 1: Put the replication in a ConfigMap

This replication loads one table incrementally. The primary_key makes each load a merge, and the update_key is the checkpoint column.

# replication.yaml
source: SOURCE_DB
target: TARGET_DB

defaults:
  mode: incremental
  object: main.{stream_table}

streams:
  main.events:
    primary_key: [event_id]
    update_key: created_at
kubectl create configmap sling-replication \
  --from-file=replication.yaml=replication.yaml

The names SOURCE_DB and TARGET_DB are connection names. The Secret in the next step defines them.

Step 2: Put the connections in a Secret

Write an env.yaml with the two connections. For a Postgres source and a Snowflake target, it looks like this:

# env.yaml (goes into a Secret, not into the image)
connections:
  SOURCE_DB:
    url: postgresql://etl_user:<password>@postgres.data.svc.cluster.local:5432/app

  TARGET_DB:
    type: snowflake
    account: myorg-myaccount
    user: SLING_SERVICE
    database: ANALYTICS
    warehouse: LOAD_WH
    role: LOADER
    private_key: |
      -----BEGIN PRIVATE KEY-----
      ...
      -----END PRIVATE KEY-----

Use key-pair authentication for Snowflake. A scheduled job cannot answer an MFA prompt. The private_key key accepts a PEM body, a Base64-encoded DER key, or a file path (see the Snowflake connection docs).

Create the Secret from the file:

kubectl create secret generic sling-env \
  --from-file=env.yaml=env.yaml \
  --from-literal=SLING_CLI_TOKEN="$SLING_CLI_TOKEN"   # optional, for CLI Pro features

Two other methods also work:

  • One variable for each connection. Sling reads a connection from an environment variable with the connection name, for example SOURCE_DB='postgresql://...'.
  • Secret references. In Sling v1.6.5 and later, a connection value can be ref+k8s://v1/Secret/<namespace>/<name>/<key>. In a pod, Sling reads the Secret through the API server with the service account token. See Secrets.

Step 3: Test with a one-time Job

Before you add a schedule, run the same container once as a Job.

# job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: sling-run
spec:
  backoffLimit: 1
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: sling
          image: slingdata/sling:v1.6.7
          args: ["run", "-r", "/config/replication.yaml"]
          env:
            - name: SLING_ENV_YAML
              valueFrom:
                secretKeyRef:
                  name: sling-env
                  key: env.yaml
            - name: SLING_CLI_TOKEN
              valueFrom:
                secretKeyRef:
                  name: sling-env
                  key: SLING_CLI_TOKEN
                  optional: true
            - name: SLING_LOGGING
              value: NO_COLOR
          volumeMounts:
            - name: replication
              mountPath: /config
              readOnly: true
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              memory: 2Gi
      volumes:
        - name: replication
          configMap:
            name: sling-replication

SLING_LOGGING=NO_COLOR removes the color codes from the pod logs. Use JSON if your log stack parses JSON lines. The resource values are a start point. Measure your own streams and change them.

kubectl apply -f job.yaml
kubectl logs -f job/sling-run

What does a run look like?

To get the output below, we ran slingdata/sling:v1.6.7 with the same arguments, environment variables, and mount as the Job. Each run used a new container. The connections in this lab are two SQLite files, so you can repeat it on a laptop. The source table has 60,000 rows.

INF Sling CLI | https://slingdata.io
INF Sling Replication | SOURCE_DB -> TARGET_DB | main.events
INF connecting to source database (sqlite)
INF connecting to target database (sqlite)
INF getting checkpoint value (created_at)
INF reading from source database
INF writing to target database [mode: incremental]
INF created table "main"."events_tmp"
INF streaming data
INF downloading sqlite 3.41 for linux/amd64
INF created table "main"."events"
INF inserted 60000 rows into "main"."events" in 0 secs [73,818 r/s] [2.1 MB]
INF execution succeeded

The downloading sqlite line is specific to SQLite and DuckDB connections. Sling gets the database binary into a new container on each run, so the pod needs outbound HTTPS for these targets. Postgres and Snowflake connections do not download a binary.

How does a new pod know where the last run stopped?

Sling reads the checkpoint from the target table, not from the pod disk. At the start of an incremental run, it gets the maximum update_key value in the target and reads only source rows after that value. The function is getIncrementalValueViaDB in core/sling/task_func.go.

To test this, we added 2,500 rows to the source and ran a second new container. It had no files from the first run:

INF getting checkpoint value (created_at)
INF inserted 2500 rows into "main"."events" in 0 secs [5,179 r/s] [89 kB]
INF execution succeeded

A third run, with no new source rows, did nothing and exited with code 0:

WRN no data or records found in stream. Nothing to do. To allow Sling to create empty tables, set SLING_ALLOW_EMPTY=TRUE
INF inserted 0 rows into "main"."events" in 0 secs [0 r/s]
INF execution succeeded

The target then had 62,500 rows and 62,500 distinct event_id values, so no row loaded two times:

select count(*), count(distinct event_id), max(created_at) from events;
-- 62500 | 62500 | 2026-11-08 06:56:40

This is why a database target needs no PersistentVolumeClaim for Sling. You can delete the pod, move it to a different node, or create the CronJob again. The next run starts from the correct row. For more about how the primary_key merge works, see upserts with Sling.

A file target is different. For S3, GCS, or Azure targets, Sling keeps incremental state in the location that SLING_STATE gives, for example MY_S3/sling-state. State-based incremental is a CLI Pro feature.

Step 4: Put the Job on a schedule

When the Job works, make a CronJob with the same pod template, a schedule, and a concurrency policy.

# cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: sling-nightly
spec:
  schedule: "0 2 * * *"         # every day at 02:00
  timeZone: "Etc/UTC"
  concurrencyPolicy: Forbid      # do not start a run while one is active
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 1
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: sling
              image: slingdata/sling:v1.6.7
              args: ["run", "-r", "/config/replication.yaml"]
              env:
                - name: SLING_ENV_YAML
                  valueFrom:
                    secretKeyRef:
                      name: sling-env
                      key: env.yaml
                - name: SLING_CLI_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: sling-env
                      key: SLING_CLI_TOKEN
                      optional: true
                - name: SLING_LOGGING
                  value: NO_COLOR
              volumeMounts:
                - name: replication
                  mountPath: /config
                  readOnly: true
              resources:
                requests:
                  cpu: 250m
                  memory: 512Mi
                limits:
                  memory: 2Gi
          volumes:
            - name: replication
              configMap:
                name: sling-replication

concurrencyPolicy: Forbid stops a second run while the first run is active. Thus two Sling processes do not write to the same table at the same time. The history limits keep the last pods, so you can read their logs after a failure. The timeZone field needs Kubernetes 1.27 or later.

kubectl apply -f cronjob.yaml
kubectl create job sling-nightly-manual --from=cronjob/sling-nightly   # run it now

How do you know that a run failed?

sling run exits with code 1 when a stream fails. In our lab, a connection to a missing SQLite file gave exit code 1. With restartPolicy: Never, the pod goes to Failed, and the Job tries again up to backoffLimit. Thus your usual Kubernetes alerts on failed Jobs also cover Sling.

kubectl get jobs shows the completions. kubectl logs on a pod shows the row count for each stream.

What to check before production

  • Network access. The pod must reach the source and the target. Use a Service DNS name in the cluster, or the normal hostname of a managed database.
  • Outbound HTTPS. The CLI validates SLING_CLI_TOKEN over the network when it starts. SQLite and DuckDB connections download a binary. Sling also sends anonymous usage data. Set SLING_DISABLE_TELEMETRY=true to stop it.
  • Secret management. Sealed Secrets, External Secrets, or a cloud secret manager all work, because the pod only reads a Kubernetes Secret.
  • One CronJob for each replication. Different tables can need different schedules. All CronJobs can use the same Secret.
  • Version pin. Change the image tag in a reviewed commit, not through latest.

Frequently asked questions

Do I need a persistent volume for Sling’s incremental state on Kubernetes?

Not for a database target. At the start of each incremental run, Sling reads the maximum update_key value from the target table. The pod keeps no state. A file target (S3, GCS, Azure) is different: it needs SLING_STATE, which is a CLI Pro feature.

How do I give the pod its connection credentials?

Put the full env.yaml in a Kubernetes Secret and pass it to the container in the SLING_ENV_YAML environment variable. You can also give each connection its own environment variable, or use a ref+k8s:// secret reference in Sling v1.6.5 and later.

Which Docker image tag do I use?

slingdata/sling:latest and the vX.Y.Z tags are linux/amd64 images. For arm64 nodes, use latest-arm64 or X.Y.Z-arm64 (no v prefix). Pin a version in production, for example slingdata/sling:v1.6.7.

What happens if a scheduled run starts before the previous run is complete?

Set concurrencyPolicy: Forbid on the CronJob. Kubernetes then skips the new run, so two Sling processes do not write to the same target table at the same time.

How does Kubernetes know that a Sling run failed?

sling run exits with code 1 when a stream fails. With restartPolicy: Never, the pod goes to the Failed state, and the Job retries up to backoffLimit.

Where do I put SLING_CLI_TOKEN?

Put it in the same Secret and expose it as an environment variable. The CLI validates the token over the network when it starts, so the pod needs outbound HTTPS. Air-gapped clusters need an offline token from Sling support.