This page provides links to public articles, videos, and podcasts related to
Dataflow.

### Announcements

To read about announcements and updates, see the following resources:

- Dataflow news: [Google Cloud blog](https://cloud.google.com/blog/products/data-analytics)
- Dataflow updates: [Dataflow release notes](https://cloud.google.com/dataflow/docs/release-notes)
- Apache Beam updates: [Apache Beam SDK release notes](https://cloud.google.com/dataflow/docs/resources/release-notes-apache-beam)

### Dataflow ecosystem

- [Dataflow, the backbone of data analytics](https://cloud.google.com/blog/topics/developers-practitioners/dataflow-backbone-data-analytics)
- Dataflow Under the Hood: the origin story: [Part 1](https://cloud.google.com/blog/products/data-analytics/how-cloud-batch-and-stream-data-processing-works), [Part 2](https://cloud.google.com/blog/products/data-analytics/cloud-batch-and-stream-processing-for-analytics), [Part 3](https://cloud.google.com/blog/products/data-analytics/dataflow-vs-other-stream-batch-processing-engines)
- Visit the Apache Beam open source [docs](https://beam.apache.org/documentation/) to learn more about developing a unified model, defining pipelines, and running pipelines in Dataflow or one of the Apache Beam supported distributed backends.

### Customer stories

### Public articles

- [Collection of Dataflow stories](https://medium.com/google-cloud/search?q=Dataflow) - Medium Publication
- [Dataflow a Leader in 2021 Streaming Analytics](https://cloud.google.com/blog/products/data-analytics/dataflow-leader-forrester-wave-streaming-analytics-report-2021) - Forrester Wave
- [Building a tool which provides real-time feedback on live audiences](https://www.cloudpro.co.uk/it-infrastructure/cloud-deployment/8011/itv-and-google-cloud-partner-to-fortify-streaming) - ITV
- [Monitoring your Dataflow pipelines](https://medium.com/google-cloud/monitoring-your-dataflow-pipelines-80b9a2849f7a) - Medium Publication
- [Execution Model for Highly Scalable, Low-Latency Data Processing](https://medium.com/google-cloud/streaming-engine-execution-model-1eb2eef69a8e) - Medium Publication
- [Accelerating Machine Learning Model Inference on Dataflow with GPUs](https://developer.nvidia.com/blog/accelerating-machine-learning-model-inference-on-google-cloud-dataflow-with-nvidia-gpus/) - Nvidia
- [Executing computation against large datasets](https://engineering.pandora.com/investing-in-pandoras-core-differentiators-1fbec589e1d) - Pandora
- [Optimized the Largest Dataflow Job Ever for Wrapped 2020](https://engineering.atspotify.com/2021/02/how-spotify-optimized-the-largest-dataflow-job-ever-for-wrapped-2020/) - Spotify
- [Processing billions of events in real time using Dataflow](https://blog.twitter.com/engineering/en_us/topics/infrastructure/2021/processing-billions-of-events-in-real-time-at-twitter-) - Twitter
- [Dataflow In the Smart Home Data Pipeline](https://nest.tech/google-cloud-dataflow-in-the-smart-home-data-pipeline-5ae71781b856) - Nest
- [Streaming JSON messages into BigQuery JSON-type column](https://medium.com/@olejniczak.lukasz/streaming-json-messages-into-bigquery-json-type-column-7b9702a49a36) - Medium

### Case studies

- AXA Switzerland: [Using Google Cloud analytics solutions to boost internal processes and develop services](https://cloud.google.com/customers/axa-switzerland)
- Bayer Crop: [Improving soil health and crop management with geospatial analytics on BigQuery and Dataflow](https://cloud.google.com/blog/products/data-analytics/how-bayer-crop-science-uses-bigquery-and-geobeam-improve-soil-health)
- Dow Jones: [Built a knowledge Graph of key events documented in over 30 years of news content](https://cloud.google.com/customers/dow-jones/)
- HSBC: [Embracing the cloud to lower risk exposure through rapid insight and analysis capabilities](https://cloud.google.com/customers/hsbc-risk-advisory-tool/)
- Nintendo: [Using Dataflow and Pub/Sub to collect and analyze game usage logs in BigQuery](https://cloudplatform-jp.googleblog.com/2017/06/nintendo-super-mario-run-google-cloud.html)
- Quantiphi: [Built a serverless real-time credit card fraud detection solution](https://cloud.google.com/blog/products/data-analytics/how-to-build-a-fraud-detection-solution)
- SoFi Stadium: [Building a fan ready personal concierge app to tailor game day experiences for every user](https://cloud.google.com/blog/products/media-entertainment/sofi-stadium-powers-return-live-entertainment-google-cloud)
- Spotify: [Experiments with stream processing on Dataflow](https://cloud.google.com/blog/products/gcp/spotifys-experiments-with-stream-processing-on-google-cloud-dataflow)
- Subaru Corporation: [Using Google Cloud AI and Machine learning to accelerate development](https://cloud.google.com/customers/subaru)
- Telus: [Accelerate modernization with data science](https://cloud.google.com/blog/products/data-analytics/telus-accelerates-modernization-with-data-science)
- Tokopedia: [Created a Customer Data Platform on Google Cloud](https://cloud.google.com/blog/products/data-analytics/how-tokopedia-created-a-customer-data-platform-cdp-on-google-cloud)
- Tyson Foods: [Reimagined their Data Platform by developing Ingestion as a Service](https://cloud.google.com/blog/topics/consumer-packaged-goods/tyson-foods-embraces-aibi-analytics-to-enable-predictive-insights)
- Vodafone: [Using Google Cloud to safely share mobile phone data](https://cloud.google.com/blog/products/data-analytics/vodafone-picks-google-cloud-to-safely-share-mobile-phone-data)

### Videos

#### Brightcove: Serverless data processing with Dataflow and Pub/Sub

[Video](https://www.youtube.com/watch?v=3BrcmUqWNm0)

#### Sky: Real-Time Stream Analytics with Dataflow

[Video](https://www.youtube.com/watch?v=7lJyq1hw_KI)

#### Spotify: Migrating from music intelligence pipeline and microservices to Dataflow and Pub/Sub

[Video](https://www.youtube.com/watch?v=h7IM0EF_18g)

#### Two Sigma: Advancing Serverless Data Processing in Dataflow

[Video](https://www.youtube.com/watch?v=h7IM0EF_18g)

#### Tyson Foods: Generating real-time value with streaming analytics

[Video](https://www.youtube.com/watch?v=ijEIDyHRctI)

#### Veolia: Using Dataflow in production to transform, clean, and insert tens of thousands of files into our BigQuery data lake

[Video](https://www.youtube.com/watch?v=AymRqE2-SVY)

#### Verizon: Building an internet-scale analytics platform on Google Cloud

[Video](https://www.youtube.com/watch?v=Kjo-Zig7yKw)

### Technical guidance

### Intro Videos


#### Dataflow in a minute

[Video](https://www.youtube.com/watch?v=XdsuDOQ9nkU)

<br />


#### What is Dataflow?

[Video](https://www.youtube.com/watch?v=KalJ0VuEM7s)

<br />

#### Google Cloud Tech: 1) What is Apache Beam?

[Video](https://www.youtube.com/watch?v=65lmwL7rSy4)

#### Google Cloud Tech: 2) How to use WordCount in Apache Beam

[Video](https://www.youtube.com/watch?v=RTIOW1fIhkM)

#### Google Cloud Tech: 3) How to process stream data on Apache Beam

[Video](https://www.youtube.com/watch?v=oJ-LueBvOcM)

#### Google Cloud Tech: 4) How to process late data on Apache Beam

[Video](https://www.youtube.com/watch?v=MuFA6CSti6M)

#### Google Cloud Tech: 5) Joining data in Apache Beam

[Video](https://www.youtube.com/watch?v=O5pcahJ_KwE)

#### Apache Beam a unified model for batch and streaming data processing

[Video](https://www.youtube.com/watch?v=7DZ8ONmeP5A)

#### Short intro to Dataflow data pipelines

[Video](https://www.youtube.com/watch?v=m1fkc4rBfjg)

#### Easy access to stream analytics with Google Cloud

[Video](https://www.youtube.com/watch?v=4k-0wHOUy0A)

### Articles

- [Building the data engineering driven organization](https://cloud.google.com/blog/products/data-analytics/building-the-data-engineering-driven-organization)
- [Create templates from any Dataflow pipeline](https://cloud.google.com/blog/products/data-analytics/create-templates-from-any-dataflow-pipeline)
- [Dataflow Templates for Elastic Cloud](https://cloud.google.com/blog/products/data-analytics/dataflow-templates-for-elastic-cloud)
- [Dataflow Pipelines: deploy and manage data pipelines at scale](https://cloud.google.com/blog/products/data-analytics/manage-your-dataflow-jobs-at-scale-with-dataflow-pipelines)
- [Dataflow Auto Sharding for BigQuery delivers 3x performance](https://cloud.google.com/blog/products/data-analytics/3x-dataflow-throughput-auto-sharding-bigquery)
- [Export Google Cloud data into Elastic Stack with Dataflow templates](https://cloud.google.com/blog/products/data-analytics/export-google-cloud-data-into-elasticsearch-with-dataflow-templates)
- [Extend your Dataflow template with UDF](https://cloud.google.com/blog/topics/developers-practitioners/extend-your-dataflow-template-with-udfs)
- Exactly once processing in Dataflow: [Part 1](https://cloud.google.com/blog/products/data-analytics/after-lambda-exactly-once-processing-in-google-cloud-dataflow-part-1), [Part 2](https://cloud.google.com/blog/products/gcp/after-lambda-exactly-once-processing-in-cloud-dataflow-part-2-ensuring-low-latency), [Part 3](https://cloud.google.com/blog/products/gcp/after-lambda-exactly-once-processing-in-cloud-dataflow-part-3-sources-and-sinks)
- [Give your data processing a boost with Dataflow GPU](https://cloud.google.com/blog/products/data-analytics/give-your-data-pipelines-a-boost-with-gpus)
- [Handling duplicate data in a streaming pipeline using Dataflow \& Pub/Sub](https://cloud.google.com/blog/products/data-analytics/handling-duplicate-data-in-streaming-pipeline-using-pubsub-dataflow)
- [Learn Apache Beam patterns with Clickstream processing of Google Tag Manager data](https://cloud.google.com/blog/products/data-analytics/learn-beam-patterns-with-clickstream-processing-of-google-tag-manager-data)
- [Machine learning patterns with Apache Beam and the Dataflow Runner](https://cloud.google.com/blog/products/data-analytics/ml-inference-in-dataflow-pipelines)
- [Streaming data into BigQuery using BigQuery Storage Write API (gRPC)](https://cloud.google.com/blog/products/data-analytics/streaming-data-into-bigquery-using-storage-write-api)
- [Simplify and automate data processing with Dataflow Prime](https://cloud.google.com/blog/products/data-analytics/simplify-and-automate-data-processing-with-dataflow-prime)
- [Three ways Dataflow delivers ROI to Customers](https://cloud.google.com/blog/products/data-analytics/3-ways-dataflow-delivers-roi-to-customers)
- [Use real-time anomaly detection reference patterns to combat fraud](https://cloud.google.com/blog/products/data-analytics/using-automated-ml-streaming-architecture-to-find-anomalies)
- [Using TFX inference with Dataflow for large scale ML inference patterns](https://cloud.google.com/blog/topics/developers-practitioners/using-tfx-inference-dataflow-large-scale-ml-inference-patterns)
- [Why you should be using Flex templates for your Dataflow deployments](https://cloud.google.com/blog/topics/developers-practitioners/why-you-should-be-using-flex-templates-your-dataflow-deployments)
- [Writing Dataflow pipelines with scalability in mind](https://cloud.google.com/blog/products/gcp/writing-dataflow-pipelines-with-scalability-in-mind)
- Guide to common Dataflow use-case patterns: [Part 1](https://cloud.google.com/blog/products/data-analytics/guide-to-common-cloud-dataflow-use-case-patterns-part-1), [Part 2](https://cloud.google.com/blog/products/data-analytics/guide-to-common-cloud-dataflow-use-case-patterns-part-2)

### Data \& Analytics Videos

#### Apache Beam Summit: Handling Duplicate Data in Streaming Pipelines using Dataflow and Pub/Sub

[Video](https://www.youtube.com/watch?v=IYffTThXldI)

#### Cloud OnAir: Stream Processing with Dataflow: SDKs \& Architectures

[Video](https://www.youtube.com/watch?v=2vZc2AT_vjg)

#### Cloud Next: Apache Beam - Portable and Parallel Data Processing

[Video](https://www.youtube.com/watch?v=owTuuVt6Oro)

#### Cloud Next: Streaming analytics on Google Cloud

[Video](https://www.youtube.com/watch?v=qU5sOsybQsw)

#### Cloud Next: Build Your Own Event Analytics Pipeline using BigQuery, Dataflow, and Kubernetes

[Video](https://www.youtube.com/watch?v=7WLCWo8KRkE)

#### Cloud Next: Apache Kafka Ecosystem As-a-Service on Google Cloud with Confluent Cloud

[Video](https://www.youtube.com/watch?v=Tl9-j3XG8bQ)

### Troubleshooting and monitoring

### Videos

#### Dataflow Observability, Monitoring, and Troubleshooting

[Video](https://www.youtube.com/watch?v=e87LNQmc-l8)

#### Apache Beam College: Dataflow Monitoring

[Video](https://www.youtube.com/watch?v=Y_P9zQ3U7L8)

#### Apache Beam College: Dataflow Logging

[Video](https://www.youtube.com/watch?v=uVUo3NtosUE)

#### Apache Beam College: Troubleshooting and debugging Apache Beam and Dataflow

[Video](https://www.youtube.com/watch?v=H98M1eZp-7E)

### Articles

- [Use the Dataflow monitoring interface](https://docs.cloud.google.com/dataflow/docs/guides/using-monitoring-intf)
- [Build production-ready data pipelines using Dataflow: Monitoring data pipelines](https://cloud.google.com/architecture/building-production-ready-data-pipelines-using-dataflow-monitoring)
- [Pipeline troubleshooting and debugging](https://docs.cloud.google.com/dataflow/docs/guides/troubleshooting-your-pipeline)

### Podcasts

- [Google Cloud Podcast Episode 81 - Dataflow with Frances Perry](https://www.youtube.com/watch?v=GDKDXvpK284)
- [Software Engineering Daily Podcast - Dataflow with Eric Anderson](https://softwareengineeringdaily.com/2016/09/16/cloud-dataflow-with-eric-anderson/)
- [Software Engineering Radio Podcast Episode 272: Apache Beam](https://beam.apache.org/get-started/resources/videos-and-podcasts/#software-engineering-radio-podcast-episode-272-apache-beam)