Jenkins Reference

Declarative pipelines, a full Jenkinsfile for a PySpark project, credentials, triggers and pull requests, agents, shared libraries and best practices.

1. What Jenkins Is

Jenkins is an open-source automation server for CI/CD. A controller schedules work and stores configuration; agents (VMs, containers, Kubernetes pods) run the builds. Almost everything else (Git, Docker, Kubernetes, credentials, notifications) comes from plugins.

2. A Declarative Jenkinsfile

A CI/CD pipeline for a PySpark project: test on every branch, deploy to dev on main, and deploy to prod after manual approval.

pipeline {
  agent { docker { image 'python:3.11-slim' } }

  options {
    timeout(time: 45, unit: 'MINUTES')
    disableConcurrentBuilds()
    buildDiscarder(logRotator(numToKeepStr: '30'))
  }

  parameters {
    booleanParam(name: 'DEPLOY_PROD', defaultValue: false, description: 'Promote to prod after dev')
  }

  environment {
    DATABRICKS_HOST = 'https://adb-1234567890.11.azuredatabricks.net'
  }

  stages {
    stage('Install') {
      steps { sh 'pip install -r requirements-dev.txt' }
    }
    stage('Quality') {
      parallel {
        stage('Lint')  { steps { sh 'ruff check src tests' } }
        stage('Tests') { steps { sh 'pytest -q --junitxml=reports/junit.xml' } }
      }
    }
    stage('Build') {
      steps { sh 'python -m build --wheel' }
    }
    stage('Deploy dev') {
      when { branch 'main' }
      steps {
        withCredentials([string(credentialsId: 'databricks-dev-token', variable: 'DATABRICKS_TOKEN')]) {
          sh 'databricks bundle deploy -t dev'
        }
      }
    }
    stage('Deploy prod') {
      when { allOf { branch 'main'; expression { params.DEPLOY_PROD } } }
      steps {
        input message: 'Deploy to production?', submitter: 'release-managers'
        withCredentials([string(credentialsId: 'databricks-prod-token', variable: 'DATABRICKS_TOKEN')]) {
          sh 'databricks bundle deploy -t prod'
        }
      }
    }
  }

  post {
    always  { junit allowEmptyResults: true, testResults: 'reports/*.xml' }
    failure { echo "Build failed: ${env.BUILD_URL}" }   // or slackSend / emailext
  }
}

The Databricks CLI must be installed in the agent image. Use single quotes for shell strings so secrets are expanded by the shell, not interpolated by Groovy into the build log.

3. Pipeline Syntax Reference

In cron triggers, H spreads start times by a hash of the job name so all jobs do not fire on the same minute. Declarative syntax is preferred; Scripted pipelines (node { … }) give full Groovy but are harder to maintain.

4. Credentials & Secrets

Store secrets in the Jenkins credentials store (or an external vault via plugins) and reference them by ID. Jenkins masks bound values in console output.

environment {
  // username/password credential exposes REGISTRY (user:pass), REGISTRY_USR and REGISTRY_PSW
  REGISTRY = credentials('docker-registry')
}
steps {
  sh 'echo "$REGISTRY_PSW" | docker login -u "$REGISTRY_USR" --password-stdin myregistry.io'

  withCredentials([file(credentialsId: 'gcp-sa-key', variable: 'GOOGLE_APPLICATION_CREDENTIALS')]) {
    sh 'gcloud auth activate-service-account --key-file="$GOOGLE_APPLICATION_CREDENTIALS"'
  }
}

• Never echo secrets or pass them in Groovy double-quoted strings, which interpolate them before masking applies.
• Prefer short-lived credentials: OIDC federation to the cloud, or a Vault plugin, over long-lived static keys.
• Scope credentials to folders so only the right teams' jobs can use them.

5. Triggers, Branches & Pull Requests

• Webhooks (GitHub, GitLab, Bitbucket) trigger builds instantly on push or pull request. Prefer them to pollSCM, which repeatedly queries the repository.
• Multibranch pipelines build every branch and PR with its own Jenkinsfile and report status back to the PR, so merging can require a green build.
• Use when { changeRequest() } for PR-only stages and when { branch 'main' } for deployments.
• Scheduled runs: triggers { cron('H 2 * * 1-5') } for weekday nightly builds.
• Chain pipelines with build job: 'integration-tests', parameters: [string(name: 'VERSION', value: env.VERSION)].

6. Agents: Static, Docker & Kubernetes

agent {
  kubernetes {
    yaml """
apiVersion: v1
kind: Pod
spec:
  containers:
    - name: python
      image: python:3.11-slim
      command: ["sleep"]
      args: ["infinity"]
"""
    defaultContainer 'python'
  }
}

Do not run builds on the controller itself: it should only orchestrate. Set its executor count to 0.

7. Shared Libraries

When many repositories need the same pipeline steps, put them in a shared library (its own Git repo) instead of copying Jenkinsfiles:

# repository layout
vars/
  pysparkPipeline.groovy     # global step, callable as pysparkPipeline(...)
src/org/data/Utils.groovy    # helper classes
resources/                   # templates, scripts

// vars/pysparkPipeline.groovy
def call(Map cfg) {
  pipeline {
    agent { docker { image cfg.image ?: 'python:3.11-slim' } }
    stages {
      stage('Test')   { steps { sh 'pip install -r requirements-dev.txt && pytest -q' } }
      stage('Deploy') {
        when { branch 'main' }
        steps { sh "databricks bundle deploy -t ${cfg.target}" }
      }
    }
  }
}

// Jenkinsfile in each project
@Library('data-pipelines@v2') _
pysparkPipeline(target: 'dev')

Pin library versions with tags (@v2) so a library change cannot silently break every pipeline at once.

8. Best Practices

• Keep pipelines as code (Jenkinsfile + shared libraries) and Jenkins itself as code (Configuration as Code plugin), so the server can be rebuilt.
• Build once, deploy many: produce one versioned artifact (wheel, Docker image) and promote the same artifact through environments.
• Fail fast: run lint and unit tests first, run stages in parallel, and set timeouts on every pipeline.
• Use ephemeral agents (Docker/Kubernetes) for clean, reproducible builds.
• Publish test results and coverage, and archive only artifacts you need.
• Keep plugins few and updated; they are the main source of security issues and upgrade pain.
• Back up JENKINS_HOME (or rely on Configuration as Code plus Git) and restrict who can configure jobs.

9. Jenkins vs Other CI/CD Tools

Concepts carry over directly: a Jenkins stage ≈ a GitHub Actions job, agents ≈ runners, the credentials store ≈ repository or environment secrets.