Lightdash Community

Why we're building Lightdash on dbt

Great data visualisation needs a great data transformation layer. So we built Lightdash on top of dbt. Introducing such a huge technical dependency wasn’t an easy choice but we’re really excited to build on dbt and here’s why:

  • SQL is the language of data
  • We’re building on open-source
  • We’re plugging into a standard
  • It’s better to do one thing and do it well
  • Communities win together

SQL is the language of data :abc:

Under the hood, dbt is simply running SQL statements. That’s amazing because SQL has been around for 47 years and isn’t going anywhere. SQL has a thriving community with a lot of collective knowledge on how to solve most common problems. Unlike scripting languages such as python, SQL is also easy to understand making it easy for engineers and Excel whizzes to collaborate :handshake:.

With Lightdash we threw away any ideas for a domain-specific language, and stuck with SQL. With Lightdash and dbt, analysts can write in SQL and their users then simply pick from the pre-defined metrics.

We’re building on open-source :earth_africa:

If you’re relying on any technical component to be a crucial part of your tech stack, it’s a huge benefit if that component is open source. dbt’s core is completely open-source1. By having access to the source code, you are protected in case the code owners decide to stop support. More exciting, you get the chance to contribute to the road map or extend the code however you see fit.

Lightdash is open-source at it’s core meaning you can see how we write queries, handle credentials, and enforce security. It enables us to be completely transparent and also lowers the barrier for our users to request features and report bugs.

We’re plugging into a standard :electric_plug:

dbt has changed the way that we think about data pipelines. Before dedicated transformations tools, everyone was building their own pipeline DAGs and testing infrastructure with no standardisation. It was a huge engineering headache for those that remember the dark days.

As of a year ago, over 2,000 companies using it weekly with 3x growth YoY2. If somebody today is building an analytics stack, they’re likely using dbt somewhere.

By plugging into this standard, Lightdash enables people to use their existing familiar tools and invest in the future. And if you throw Lightdash away, you are still left an industry-standard set of data transformations.

It’s better to do one thing and do it well :1st_place_medal:

dbt follows the unix philosophy of doing one thing and doing it well (it also has a cute name à la awk, sed). By just focusing on the “T” in ELT, dbt has become a powerful tool for a huge variety of use cases. Simply: it helps manage batch SQL-based transformations, in an idempotent way, and it just works.

By leveraging dbt, Lightdash has a first-class transformation layer. With more and more people using dbt, and the fact that dbt does this bit so well, we thought it was time for a BI tool to be adding to this step, not trying to be some hybrid replacement for it.

Communities win together :dancing_women:

The dbt community, at the intersection of analytics and engineering, has created a whole new domain of analytics engineering. Conversations always feel productive with community members. It’s also just a whole lotta fun.

At Lightdash we’re excited about how analytics engineers use dbt to serve the rest of their business. We wrote about this before3 and want to nurture a complimentary community that brings data users to the conversation because we believe that teams are more likely to find solutions than individuals.

We’re really excited to build a visual layer for dbt and work directly with the community to make the perfect tool for modern analytics engineers. Join us for the ride and join the Lightdash Community4.

Links

  1. https://github.com/fishtown-analytics/dbt
  2. https://blog.getdbt.com/four-years-in-from-misfits-to-mainstream/
  3. https://community.lightdash.com/t/why-are-we-building-a-community/18
  4. https://community.lightdash.com/invites/eh9WEKMDkU
3 Likes