Dagster Labs (Elementl) llms.txt

The actual file

Source Markdown

The saved llms.txt text, displayed as source. The file belongs to its publisher; the live endpoint may change.

Showing the first 18000 characters. Load the complete file to read the remainder.

# Dagster

> Dagster is an open-source data orchestrator for building, running, and observing data and AI pipelines. You define data assets (tables, files, ML models, reports) as Python functions, and Dagster runs them at the right time, tracks lineage between them, and monitors their quality and freshness. Dagster+ is the managed commercial platform built on the open-source project. Dagster is part of Prefect and continues under its own name and open-source license.

Core concepts:

- Assets: the data products a pipeline produces, declared in Python with their upstream dependencies.
- Resources: configurable connections to external systems such as warehouses and APIs.
- Schedules, sensors, and declarative automation: run assets on a cadence, on external events, or when upstream data changes.
- Partitions and backfills: split assets by time or another dimension and recompute any slice.
- Components: YAML-configured building blocks for common integrations such as dbt, Fivetran, and Sling.
- Dagster+: hosted control plane with serverless or hybrid deployment, branch deployments, a data catalog, alerting, and cost insights.

Pages backed by authored content (blog posts, customer stories, integrations, guides, learning articles, glossary terms, and events) are also available as plain Markdown: add `.md` to the page URL or send `Accept: text/markdown`. The links below point to those Markdown copies. [llms-full.txt](https://dagster.io/llms-full.txt) contains the full text of the product and learning content in one file.

## Documentation

- [Dagster docs llms.txt](https://docs.dagster.io/llms.txt): Index of the Dagster documentation for agents
- [Dagster docs llms-full.txt](https://docs.dagster.io/llms-full.txt): Full text of the Dagster documentation
- [Dagster docs](https://docs.dagster.io/): Guides, tutorials, and API reference
- [GitHub repository](https://github.com/dagster-io/dagster/): Source code, issues, and release notes for open-source Dagster

## Product

- [Platform overview](https://dagster.io/platform-overview): How Dagster works as a control plane for data and AI pipelines
- [Components](https://dagster.io/platform-overview/components): The building blocks of a Dagster project: assets, schedules, sensors, and more
- [Data orchestration](https://dagster.io/platform-overview/data-orchestration): Scheduling, automation, and dependency management for pipelines
- [Data catalog](https://dagster.io/platform-overview/data-catalog): Searchable catalog of assets with ownership, lineage, and metadata
- [Data quality](https://dagster.io/platform-overview/data-quality): Asset checks, freshness policies, and alerting
- [Cost insights](https://dagster.io/platform-overview/cost-insights): Track compute and warehouse spend per asset
- [Dagster+ for enterprise](https://dagster.io/enterprise): Security, governance, scale, and support for large teams
- [Dagster and Databricks](https://dagster.io/databricks): Orchestrating Databricks workloads with Dagster
- [Security](https://dagster.io/security): Security practices and compliance

## Pricing

- [Pricing](https://dagster.io/pricing): Dagster+ plans: Solo ($10/month plus usage), Starter ($100/month plus usage), and Pro (custom). Includes a 30-day free trial
- [Dagster+ trial](https://dagster.io/lp/dagster-plus-trial): Start a free trial of managed Dagster+

## Solutions

- [ETL/ELT pipelines](https://dagster.io/solutions/etl-elt-pipelines): Orchestrating ingestion and transformation
- [AI and machine learning](https://dagster.io/solutions/ai): Data prep, training, and deployment pipelines for ML and AI
- [Data modernization](https://dagster.io/solutions/data-modernization): Migrating from legacy orchestrators and platforms
- [Data products](https://dagster.io/solutions/data-products): Building and operating reliable data products
- [Finance](https://dagster.io/use-case/finance): Dagster in financial services
- [Life sciences](https://dagster.io/use-case/life-sciences): Dagster for research and lab data
- [Retail and e-commerce](https://dagster.io/use-case/retail-e-commerce): Dagster for inventory, sales, and ERP data
- [Software and technology](https://dagster.io/use-case/software-technology): Dagster for software engineering teams

## Comparisons

- [Dagster vs Airflow](https://dagster.io/vs/dagster-vs-airflow)
- [Dagster vs AWS Step Functions](https://dagster.io/vs/dagster-vs-aws-step-functions)
- [Dagster vs Azure Data Factory](https://dagster.io/vs/dagster-vs-azure-data-factory)
- [Dagster vs dbt Cloud](https://dagster.io/vs/dagster-vs-dbt-cloud)

## Company and support

- [Dagster is now part of Prefect](https://dagster.io/prefect): Announcement of Dagster becoming part of Prefect
- [Customer FAQ: Prefect acquisition](https://dagster.io/dagster-customer-faq): What the acquisition means for Dagster users and customers
- [Support](https://dagster.io/support): Ways to get help with Dagster
- [Dagster+ support](https://dagster.io/support/dagster-plus)
- [Contact](https://dagster.io/contact)
- [Request a demo](https://dagster.io/request-a-demo)
- [Partners](https://dagster.io/company/partners)
- [Community](https://dagster.io/community): Slack, GitHub, and community events
- [Dagster University](https://courses.dagster.io/): Free self-paced courses on Dagster
- [Brand kit](https://dagster.io/brand)
- [Status](https://dagstercloud.statuspage.io): Dagster+ service status

## Reports and ebooks

- [Forrester Total Economic Impact report](https://dagster.io/forrester-tei-report): Forrester's study of the ROI of Dagster+
- [Modernizing your data platform for AI](https://dagster.io/ai-modernization-guide)
- [How to build a data platform](https://dagster.io/how-to-build-data-platforms-ebook)
- [How to scale data teams](https://dagster.io/how-to-scale-data-teams-ebook)

## Customer stories

- [All customer stories](https://dagster.io/customers)
- [Flo Energy's Data Platform for Critical Energy Data](https://dagster.io/customers/flo-energys-data-platform-for-critical-energy-data.md): Learn why modern data orchestration is about more than scheduling and how Dagster's declarative automation continuously determines what needs to run based on the state of your data assets.
- [How Magenta Telekom Built the Unsinkable Data Platform](https://dagster.io/customers/how-magenta-telekom-built-the-unsinkable-data-platform.md): Learn how Magenta Telekom replaced fragmented, manual data workflows with a modular, Dagster-powered platform that reduced onboarding from 3 months to 1 day and laid the foundation for AI-driven...
- [Scaling fintech: How smava achieved zero downtime with Dagster](https://dagster.io/customers/smava-case-study.md): German fintech smava eliminated alert fatigue and achieved zero downtime by migrating to Dagster, automating 1000+ dbt models through YAML-driven code generation and reducing developer onboarding...
- [Zero Incidents, Maximum Velocity: How HIVED achieved 99.9% pipeline reliability with Dagster](https://dagster.io/customers/hived-case-study.md): Learn how UK logistics company HIVED achieved 99.9% pipeline reliability with zero data incidents over 3 years using Dagster, while reducing turnaround time for new data products to just 3-7 days.
- [Small Market, Big Data: Tampa Bay Rays' Analytics Advantage with Modern Orchestration](https://dagster.io/customers/tampa-bay-rays-case-study.md): How the Tampa Bay Rays process game data 50-70% faster with Dagster, delivering insights in 15 minutes instead of waiting for 9 a.m. batch jobs. Modern Moneyball analytics.
- [PostHog Powers Customer-Facing Web Analytics with Dagster](https://dagster.io/customers/posthog-case-study.md): Learn how PostHog's engineering team used Dagster to eliminate weekend firefighting, improve dashboard performance for enterprise customers, and reduce feature shipping time by 80%.
- [From "YAML Mess" to Unified Control: How Otto Transformed Their Data Platform with Dagster](https://dagster.io/customers/otto-case-study.md): Learn how Otto transformed their data platform from fragmented tools to unified control with Dagster. 6 teams, improved stability, faster debugging, and better collaboration.
- [Digitizing an Analog Industry: How Ida Leverages Multi-Tenant Data Orchestration with Dagster](https://dagster.io/customers/ida-case-study.md): How Ida built a multi-tenant Dagster architecture to process millions of rows daily, turning chaotic fresh produce data into AI predictions that prevent food waste.
- [How Clippd Built Organization-Wide Data Visibility and Saved 8+ Hours Per Week](https://dagster.io/customers/clippd-case-study.md): Sports analytics platform Clippd automated daily data operations, eliminated 8+ hours of weekly manual work, and democratized data access across teams.
- [From scattered scripts to unified platform: How Dagster powers data decisions for 4.5 million Belgian citizens](https://dagster.io/customers/federation-wallonie-bruxelles-case-study.md): Belgian ministry FWB modernized their data platform with Dagster, doubling pipeline speed and enabling self-service analytics for 4.5M citizens.
- [How easyJet Holidays Achieved 15x Faster Data Pipelines with Dagster](https://dagster.io/customers/easyjet-holidays-case-study.md): Learn how easyJet Holidays reduced pipeline execution from 2.5 hours to 10 minutes with Dagster, eliminated manual support, and democratized analytics.
- [How US Foods Eliminated Data Silos and Achieved Near-Perfect Reliability with Dagster](https://dagster.io/customers/us-foods-case-study.md): Learn how Fortune 500 company US Foods transformed their fragmented data architecture with Dagster, achieving 99.996% uptime across 750+ deployments.
- [Connect 211's Small Team, Big Impact: Building a Community Resource Data Platform That Serves Millions](https://dagster.io/customers/connect-211-case-study.md): Connect 211 cut pipeline troubleshooting by 80% and scaled from startup to largest health data publisher in 20 states with modern orchestration.
- [Big Cartel Brought Fragmented Data into a Unified Control Plane with Dagster](https://dagster.io/customers/big-cartel-case-study.md): Learn how Big Cartel eliminated data silos and built a scalable, observable data platform using Dagster's unified control plane—from chaos to clarity in 6 months.
- [How Vanta Eliminated Data Bottlenecks with Dagster](https://dagster.io/customers/vanta-case-study.md): Dagster is a central piece of everything we do. Without it, we would not have been able to get to the level of self-service we are at today.
- [StashAway Unified Their Global Data Platform with Dagster](https://dagster.io/customers/stashaway-case-study.md): Our team is lean, but Dagster lets us handle thousands of assets at high velocity—we couldn’t have done that with any other orchestration platform.
- [Three Weeks, 140 Iterations: How Group 1001 redefined their data platform](https://dagster.io/customers/group-1001-case-study.md): Group 1001 used Dagster to rapidly iterate and deploy a robust data platform—140 iterations in just three weeks.
- [Routing LLM prompts with Dagster and Not Diamond](https://dagster.io/customers/routing-llm-prompts-with-not-diamond.md): Improve accuracy and cost-efficiency in LLM workflows by routing prompts with Dagster and Not Diamond's orchestration strategy.
- [Case Study: Analytiks - Fast-Track AI Projects With Managed Dagster+](https://dagster.io/customers/analytiks-case-study.md): Analytiks used Managed Dagster+ to accelerate enterprise AI initiatives and data infrastructure setup.
- [Case Study: From Disconnected Data to a Unified Platform](https://dagster.io/customers/servco-case-study.md): Built-in observability and cataloging let this team open data access to a wider range of data professionals.
- [Case Study: KIPP - Building a Resilient Data Platform with Dagster](https://dagster.io/customers/kipp-case-study.md): A solo data engineer at KIPP built a resilient Dagster platform to drive organization-wide data access.
- [Case Study: Mejuri - Building an eCommerce Data Platform](https://dagster.io/customers/mejuri-case-study.md): Mejuri relies on Dagster for a robust data platform that scales with their growing ecommerce business.
- [Case Study: The Lean and Efficient One-Person Data Team of Erewhon](https://dagster.io/customers/erewhon-case-study.md): See how a solo engineer built a fast, efficient data stack at Erewhon using Dagster orchestration.
- [Case Study: Artemis - Powering the Crypto Markets](https://dagster.io/customers/artemis-case-study.md): Artemis built a Dagster-powered platform for reporting on the $2.5T crypto market.
- [Case Study: How Petal Incrementally Adopted a Data Orchestrator](https://dagster.io/customers/petal-case-study.md): Fintech company Petal gradually adopted Dagster to scale its platform without disrupting daily operations.
- [Case Study: A Start-up’s Rite of Passage - Establishing the Data Platform](https://dagster.io/customers/zippi-case-study.md): How a growing startup built a scalable data platform using Dagster to future-proof its data operations.
- [Case Study: BenchSci - A Leap Forward with Dagster](https://dagster.io/customers/benchsci-leap-forward-with-dagster.md): BenchSci uses Dagster to accelerate biomedical research and bring therapies to market faster.
- [Case Study: Abstracting Pipelines for Analysts with a YAML DSL](https://dagster.io/customers/simplisafe-case-study.md): SimpliSafe's team uses YAML DSL and Dagster to empower analysts while reducing engineering overhead.
- [Case Study: Catalyst Cooperative - Liberating Public Utility Data with Dagster](https://dagster.io/customers/catalyst-cooperative-case-study.md): Catalyst uses Dagster to clean and share energy data with researchers, policymakers, and journalists.
- [Case Study: Empirico - Enabling Large-scale, Multi-cloud Computing with Dagster](https://dagster.io/customers/empirico-case-study.md): Empirico uses Dagster to abstract infrastructure in large-scale, conditional multi-cloud environments.
- [Case Study: Orchestrating Data Science at Zephyr AI](https://dagster.io/customers/zephyr-ai-case-study.md): Zephyr AI uses Dagster to analyze massive healthcare and DNA datasets for actionable AI insights.
- [Case Study: How EvolutionIQ Rebuilt its ML Platform for Enormous Productivity](https://dagster.io/customers/evolution-iq-case-study.md): EvolutionIQ rebuilt its ML stack on Dagster to gain developer productivity and avoid common data pitfalls.
- [Case Study: Incrementally Adopting Dagster at Mapbox](https://dagster.io/customers/incremental-adoption-mapbox.md): Mapbox adopted Dagster while keeping Airflow, improving developer workflows and team efficiency.

## Integrations

- [All integrations](https://dagster.io/integrations)
- [Airflow](https://dagster.io/integrations/airflow.md): Accelerate the migration of Airflow DAGs to Dagster assets with opinionated tooling.
- [Bring Open Metadata Standards/APIs to your Dagster project](https://dagster.io/integrations/dagster-open-metadata.md): Configure and schedule metadata and profiler workflows from the OpenMetadata UI.
- [Build Bash/shell commands into your pipeline](https://dagster.io/integrations/dagster-shell.md): Execute a Bash or shell command directly or from a script file.
- [Dagster + Anthropic](https://dagster.io/integrations/dagster-anthropic.md): Integrate Anthropic’s Claude AI models into Dagster pipelines for advanced conversational AI capabilities.
- [Dagster + Clickhouse](https://dagster.io/integrations/dagster-clickhouse.md): Read and write natively to ClickHouse from software-defined assets.
- [Dagster + Datadog](https://dagster.io/integrations/dagster-datadog.md): Publish metrics to Datadog and centralize monitoring metrics.
- [Dagster + DuckDB](https://dagster.io/integrations/dagster-duckdb.md): Read and write natively to DuckDB from software-defined assets.
- [Dagster + GCP BigQuery](https://dagster.io/integrations/dagster-gcp-bigquery.md): Integrate Dagster with GCP BigQuery.
- [Dagster + GCP GCS](https://dagster.io/integrations/dagster-gcp-gcs.md): Integrate Dagster with Google Cloud Storage.
- [Dagster + Github](https://dagster.io/integrations/dagster-github.md): Integrate with GitHub Apps and automate operations in repositories.
- [Dagster + Great Expectations](https://dagster.io/integrations/dagster-great-expectations.md): Yield an expectation and its output with all relevant metadata.
- [Dagster + Hightouch](https://dagster.io/integrations/dagster-hightouch.md): Trigger syncs and monitor them until they complete.
- [Dagster + Jupyter](https://dagster.io/integrations/dagster-jupyter.md): Eliminate the tedious productionization of Jupyter notebooks.
- [Dagster + Microsoft Teams](https://dagster.io/integrations/dagster-microsoft-teams.md): Keep your team up to speed with Teams messages.
- [Dagster + MLflow](https://dagster.io/integrations/dagster-mlflow.md): Streamline productionizing, maintaining, and monitoring machine-learning models.
- [Dagster + Modal](https://dagster.io/integrations/dagster-modal.md): Run serverless Python functions at scale with Modal.
- [Dagster + Pandas](https://dagster.io/integrations/dagster-pandas.md): Implement validation on pandas DataFrames.
- [Dagster + Pandera](https://dagster.io/integrations/dagster-pandera.md): Generate Dagster Types from Pandera dataframe schemas.
- [Dagster + Polars](https://dagster.io/integrations/dagster-polars.md): Use Polars eager or lazy DataFrames as inputs and outputs in Dagster.
- [Dagster + Prometheus](https://dagster.io/integrations/dagster-prometheus.md): Integrate with Prometheus through the prometheus_client library.
- [Dagster + Ray](https://dagster.io/integrations/dagster-ray.md): Scale Python workloads with Ray’s distributed computing framework.
- [Dagster + Salesforce](https://dagster.io/integrations/dagster-salesforce.md): Query records, perform bulk operations, and sync Salesforce customer data.
- [Dagster + SharePoint](https://dagster.io/integrations/dagster-sharepoint.md): Aut