Git Reference

Core concepts, everyday commands, merging vs rebasing, undoing mistakes, remotes, branching strategies, pull requests, and repository hygiene for data projects.

1. Core Concepts

Git is a distributed version control system: every clone holds the full history. A commit is a snapshot of the whole project plus a pointer to its parent(s), identified by a SHA hash. A branch is just a movable pointer to a commit, and HEAD points to the branch (or commit) you have checked out.

 working tree ──git add──► staging area (index) ──git commit──► local repository ──git push──► remote
      ▲                                                               │
      └──────────────── git restore / git switch ◄────────────────────┘   ◄── git fetch / pull

2. Everyday Commands

git clone git@github.com:org/data-pipelines.git
git status                         # what changed, what is staged
git switch -c feature/scd2-merge   # create and switch to a branch (older: git checkout -b)
git add src/scd2.py                # stage a file (git add -p to stage hunk by hunk)
git commit -m "Add SCD2 merge for dim_customer"
git push -u origin feature/scd2-merge
git log --oneline --graph --decorate -20
git diff                           # unstaged changes
git diff --staged                  # what will be committed
git show HEAD                      # last commit's changes
git blame src/scd2.py              # who last changed each line

Write commit messages in the imperative, explaining why: "Fix duplicate rows when source has late updates" beats "fix bug". Keep commits small and focused so they are easy to review and revert.

3. Branching, Merging & Rebasing

A fast-forward merge happens when main has not moved since you branched: Git just moves the pointer, with no merge commit. Pull-request platforms also offer squash merge, which turns a branch into one commit on main.

Resolving a conflict

git switch feature/scd2-merge
git fetch origin
git rebase origin/main            # or: git merge origin/main
# CONFLICT in src/scd2.py: edit the file, keep the right lines, remove the markers
#   <<<<<<< ... ======= ... >>>>>>>
git add src/scd2.py
git rebase --continue             # (git merge --continue for a merge)
git rebase --abort                # give up and return to the state before the rebase
git push --force-with-lease       # needed after rebasing a branch you already pushed

The golden rule: never rebase commits that others have already pulled. --force-with-lease refuses to overwrite the remote if someone else pushed in the meantime, unlike plain --force.

4. Undoing Things

Rule of thumb: reset for local, unpushed work; revert for anything others may have pulled.

5. Working with Remotes

git remote -v                       # list remotes
git fetch origin                    # download new commits; does not touch your files
git pull                            # fetch + merge into the current branch
git pull --rebase                   # fetch + rebase (avoids needless merge commits)
git push origin feature/x
git push origin --delete feature/x  # delete a remote branch
git branch -vv                      # local branches and the upstream each one tracks
git fetch --prune                   # drop references to branches deleted on the remote

git fetch is always safe: it only updates remote-tracking branches like origin/main. git pull also changes your current branch. Many teams set git config --global pull.rebase true.

6. Branching Strategies

For data pipelines, GitHub flow plus CI/CD works well: main is always deployable, each PR runs unit tests and data-quality checks, and the same bundle or image is promoted from dev to prod with different configuration.

7. Pull Requests & Code Review

• Keep PRs small (a few hundred lines at most) with a description of what changed, why, and how it was tested.
• Protect main: require PRs, passing CI checks and at least one approval; disallow direct pushes and force pushes.
• Use CODEOWNERS to auto-request reviews from the owners of specific paths (e.g. /sql/finance/ @finance-data-team).
• For data changes, reviewers should check: idempotency, partition and merge keys, NULL handling, backfill impact, and cost (full scans, shuffles).
• Link PRs to tickets; squash-merge to keep main's history readable.

8. Stash, Cherry-pick, Tags & Bisect

git stash push -m "wip: dq checks"    # shelve uncommitted changes
git stash list
git stash pop                         # re-apply and drop the latest stash

git cherry-pick <sha>                 # copy one commit onto the current branch (e.g. a hotfix)

git tag -a v1.4.0 -m "Release 1.4.0"  # annotated tag for a release
git push origin v1.4.0

git bisect start                      # binary-search history for the commit that broke something
git bisect bad                        # current commit is broken
git bisect good v1.3.0                # this one worked
# test each commit Git checks out, mark it good/bad, then:
git bisect reset

9. Repository Hygiene for Data Projects

# .gitignore
__pycache__/
*.pyc
.venv/
.env                 # local secrets
*.parquet
*.csv
data/
spark-warehouse/
metastore_db/
.ipynb_checkpoints/
.databricks/
*.log

• Never commit secrets (keys, tokens, passwords). If one is committed, rotate it immediately: removing it in a later commit does not remove it from history. Add secret scanning (GitHub secret scanning, gitleaks) in pre-commit hooks and CI.
• Do not commit data files. Keep data in object storage; use Git LFS or DVC only for small reference files or model artifacts.
• Add pre-commit hooks for formatting and linting (ruff, black, sqlfluff) so style is never a review comment.
• Commit notebooks as source (.py) or strip outputs before committing, so diffs stay readable and data does not leak.

10. Common Interview Questions