Show HN: Fitness Functions for Dbt Projects (github.com)

3 points by bartschuijt •

Managing data transformation (e.g. dbt) projects at scale can be challenging: DAG violations, that one model that grew to 2K lines, duplicated logic.

Inspired by the book 'Building Evolutionary Architectures' I built tff to help data transformation projects evolve in a desired direction. It comes with a github bot that you can plugin to your CI pipeline.

It aims to let transformation projects evolve in the desired direction by: * ensuring layer boundary integrity; * detect duplicate CTE logic; * catch violations locally and in CI

Core concepts: rules apply to models; checks apply to architecture and DAG checks.

Curious to read what you think. What can be improved? What's missing?

Looks very promising!

Now I have to decide which of the ERR lints we want to fix and which to exclude :)

Thanks for taking a look! Locally you could set up pre-commit/pre-push hooks to only assess changed files (`tff check --since main`). I just realized you cannot assert changed files directly yet. In CI you can configure the bot to do the same: https://tff.readthedocs.io/en/latest/ci_cd/?h=only#1-crucial....

Yep that's exactly what I would want to do

I was impressed that it 'just worked' on my dbt project

We're currently using sqlfluff linting but it was a bit finicky to get the pieces configured.

sqlfluff does have linting inside VS Code too though, when it works (python path issues...)

Hacker News