Google Cloud Dataflow Tutorial: Easy Guide, 2 Real Examples

Your online shop gets a few hundred orders a day, and a small script handles the reporting just fine. Then a festival sale arrives. Orders jump to lakhs, the script runs out of memory halfway through, and the morning report never shows up.

Google Cloud Dataflow is built for that kind of problem. You describe what should happen to your data, and Google spreads the work across as many machines as the job needs.

Last reviewed: November 2026

In this guide:

What is Google Cloud Dataflow?
Batch vs streaming
Why use it?
When to use it
How to use it
Example 1: Batch pipeline
Example 2: Streaming pipeline
Pricing
FAQs

What Is Google Cloud Dataflow?

Google Cloud Dataflow is a managed service for processing data. It reads data from a source, changes it in the way you want, and writes the result to a destination. That is the whole idea, whether you are handling one small file or a nonstop stream of events.

Google Cloud Dataflow tutorial for processing batch and streaming data

Think of a packaging factory. Raw goods arrive at one end of a conveyor belt. Workers clean them, sort them and pack them, and finished boxes leave at the other end. When more goods arrive, the manager brings in more workers. When it gets quiet, they go home.

Google Cloud Dataflow is that manager. You design the conveyor belt, and Dataflow handles the workers.

You will see these words often:

Term	Simple meaning	Factory example
Pipeline	All the steps from start to finish	The full conveyor belt
Source	Where data comes from	The goods arriving
Transform	One step that changes the data	Cleaning or sorting
Sink	Where the results go	The finished boxes
Worker	A virtual machine doing the work	A factory worker

How Is Dataflow Related to Apache Beam?

This confuses almost everyone at first. Apache Beam is the open-source library you use to write a pipeline. Google Cloud Dataflow is the service that runs it. Beam is the recipe, and Dataflow is the kitchen.

The nice part is that you can test the same Beam code on your laptop before you send it to Google Cloud Dataflow. We will do exactly that below.

Batch vs Streaming in Google Cloud Dataflow

Point	Batch	Streaming
Data	A fixed amount, such as yesterday’s file	Nonstop, such as live orders
When it stops	When the work is finished	When you stop it
Result speed	Later, when the job ends	Within seconds or minutes
Real example	Nightly sales report	Live sales dashboard

A simple way to remember it: batch is the monthly electricity bill, and streaming is the live meter.

Why Use Google Cloud Dataflow?

No cluster to manage: Google Cloud Dataflow creates worker VMs for a job and removes them when it finishes.
Automatic scaling: It adds workers when the load grows and removes them when they are not needed.
One model for both modes: Batch and streaming pipelines use the same way of thinking.
Exactly-once processing: By default, each record counts once, which matters for things like sales totals.
Ready-made templates: Common jobs, such as Pub/Sub to BigQuery, need no Beam code at all.
Clear monitoring: The console shows each pipeline step, so you can see where a job is stuck.

When to Use Google Cloud Dataflow

Google Cloud Dataflow is a good fit when:

You need live data in BigQuery for dashboards.
Your files are too large for one script or one machine.
You need to clean and join data before loading it into a warehouse.
You process events from apps, websites, payment systems or sensors.

You probably do not need it when:

Your data is already in BigQuery and SQL can do the job.
A small script can process your file in a few minutes.
Your team already has Spark code. Dataproc may be the easier choice.

Be honest with yourself here. Google Cloud Dataflow is powerful, but starting workers has a cost, and Google itself advises against very small jobs.

Google Cloud Dataflow vs Dataproc vs BigQuery

Tool	Choose it when
Dataflow	You want managed batch or streaming pipelines with Apache Beam
Dataproc	You already use Spark or Hadoop
BigQuery	The data is in tables and SQL is enough

How to Use Google Cloud Dataflow

There are three common ways to start with Google Cloud Dataflow:

Templates: Ready-made pipelines. Fill in a few values and run.
Job builder: Build a pipeline in the Google Cloud console.
Apache Beam code: Write custom logic in Python, Java or Go.

Before you run the examples, turn on the APIs. Use a test project, and replace PROJECT_ID and PROJECT_NUMBER with your own values.

gcloud services enable dataflow.googleapis.com compute.googleapis.com \
  logging.googleapis.com storage.googleapis.com bigquery.googleapis.com \
  pubsub.googleapis.com cloudresourcemanager.googleapis.com

Next, give the default worker account the roles from Google’s tutorial. These are fine for learning, but they are too broad for production.

for role in roles/dataflow.admin roles/dataflow.worker roles/storage.admin \
  roles/pubsub.editor roles/bigquery.dataEditor; do
  gcloud projects add-iam-policy-binding PROJECT_ID \
    --member="serviceAccount:PROJECT_NUMBER-compute@developer.gserviceaccount.com" \
    --role="$role"
done

For real projects, create a separate worker service account with smaller roles. Our Google Cloud IAM guide explains how.

Finally, create a bucket for your files. Bucket names are unique worldwide, so choose your own name.

gcloud storage buckets create gs://shop-dataflow-demo-2026 --location=asia-south1

Example 1: Build a Daily Sales Report with a Batch Pipeline

Problem: Each night, the shop exports all orders to a CSV file. The file has cancelled orders and some broken rows. The owner just wants paid sales by city.

Step 1: Create orders.csv with this sample data. Notice the cancelled order and the bad amount in A1006.

order_id,city,amount,status
A1001,Delhi,1200,PAID
A1002,Mumbai,850,PAID
A1003,Delhi,300,CANCELLED
A1004,Pune,1500,PAID
A1005,Mumbai,650,PAID
A1006,Delhi,bad_value,PAID
A1007,Pune,900,PAID

Step 2: Install Apache Beam.

python -m venv env
source env/bin/activate
pip install "apache-beam[gcp]"

Step 3: Save this pipeline as sales_by_city.py.

import argparse
import csv

import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions


def parse_order(line):
    order_id, city, amount, status = next(csv.reader([line]))
    return {"city": city.strip(), "amount": amount, "status": status.strip()}


def has_valid_amount(order):
    return order["amount"].isdigit()


def run():
    parser = argparse.ArgumentParser()
    parser.add_argument("--input", required=True)
    parser.add_argument("--output", required=True)
    args, beam_args = parser.parse_known_args()

    with beam.Pipeline(options=PipelineOptions(beam_args)) as pipeline:
        (
            pipeline
            | "Read orders" >> beam.io.ReadFromText(args.input, skip_header_lines=1)
            | "Parse rows" >> beam.Map(parse_order)
            | "Keep valid amounts" >> beam.Filter(has_valid_amount)
            | "Keep paid orders" >> beam.Filter(lambda order: order["status"] == "PAID")
            | "Make city pairs" >> beam.Map(lambda order: (order["city"], int(order["amount"])))
            | "Add sales by city" >> beam.CombinePerKey(sum)
            | "Format output" >> beam.MapTuple(lambda city, total: f"{city},{total}")
            | "Write results" >> beam.io.WriteToText(args.output, file_name_suffix=".csv", header="city,total_sales")
        )


if __name__ == "__main__":
    run()

Read the middle part from top to bottom. Each line starting with | is one step on the conveyor belt: read, parse, clean, group, then write.

Step 4: Test it on your own computer first. This costs nothing on Google Cloud and catches silly mistakes early.

python sales_by_city.py --input orders.csv --output output/sales

Open the new file in the output folder. We ran this pipeline ourselves, and the output was:

city,total_sales
Delhi,1200
Mumbai,1500
Pune,2400

Delhi shows 1200 because the cancelled order and the bad row were dropped. The rows may appear in a different order on your machine.

Step 5: Run the same code on Google Cloud Dataflow. Upload the file, then change only the paths and add the Dataflow options.

gcloud storage cp orders.csv gs://shop-dataflow-demo-2026/input/orders.csv

python sales_by_city.py \
  --input gs://shop-dataflow-demo-2026/input/orders.csv \
  --output gs://shop-dataflow-demo-2026/output/sales \
  --runner DataflowRunner \
  --project PROJECT_ID \
  --region asia-south1 \
  --temp_location gs://shop-dataflow-demo-2026/temp/ \
  --job_name sales-by-city \
  --max_num_workers 2 \
  --save_main_session

Step 6: Open the Dataflow jobs page and click your job. You will see each step as a box in a graph. When the status shows Succeeded, the workers are gone and your report is in the bucket.

The --max_num_workers option is your safety limit. Keep it low while testing, because it caps how far the job can scale.

To run this every night, you can start the job on a schedule. Our Google Cloud Workflows guide shows how to connect steps like these.

Example 2: Send Live Orders to BigQuery with a Streaming Pipeline

Problem: During a sale, the team wants a dashboard that shows orders as they happen. Waiting for a nightly report is too late.

Solution: The app sends each order to Pub/Sub, a messaging service. A Google Cloud Dataflow template reads those messages and writes them to BigQuery. You do not write any pipeline code here.

Step 1: Create the Pub/Sub topic and subscription.

gcloud pubsub topics create live-orders

gcloud pubsub subscriptions create live-orders-sub --topic=live-orders

Step 2: Create the BigQuery dataset and table. The columns must match the JSON messages.

bq mk --location=asia-south1 --dataset PROJECT_ID:shop

bq mk --table PROJECT_ID:shop.live_orders \
  order_id:STRING,city:STRING,amount:INTEGER

Step 3: Start the template.

gcloud dataflow jobs run live-orders-job \
  --gcs-location=gs://dataflow-templates/latest/PubSub_Subscription_to_BigQuery \
  --region=asia-south1 \
  --staging-location=gs://shop-dataflow-demo-2026/temp \
  --max-workers=2 \
  --parameters=inputSubscription=projects/PROJECT_ID/subscriptions/live-orders-sub,outputTableSpec=PROJECT_ID:shop.live_orders

Step 4: Wait until the job is running, then send a test order.

gcloud pubsub topics publish live-orders \
  --message='{"order_id":"A2001","city":"Delhi","amount":1500}'

Step 5: Check BigQuery. The row can take about a minute to appear.

bq query --use_legacy_sql=false \
  'SELECT city, SUM(amount) AS total_sales
   FROM `PROJECT_ID.shop.live_orders`
   GROUP BY city'

What if the app sends a broken message? It does not vanish. The template writes failed rows to an error table named live_orders_error_records, so you can check them later.

Step 6: Stop the job. Please do not skip this. A streaming job keeps running, and you keep paying, until you stop it.

gcloud dataflow jobs list --region=asia-south1

gcloud dataflow jobs drain JOB_ID --region=asia-south1

Drain stops reading new messages and finishes the data already in progress. Cancel stops right away, so data in progress can be lost. For streaming jobs, try drain first.

More Real-Time Google Cloud Dataflow Scenarios

Scenario	Type	What Dataflow does
Mobile app activity	Streaming	Moves app events to BigQuery for near real-time reports
Online shop analytics	Streaming	Sends order and clickstream data to analytics tables
Website or server logs	Batch or streaming	Cleans large log volumes for analysis
Factory sensors	Streaming	Processes constant device readings
Data warehouse loading	Batch	Cleans and joins files before loading them

Google Cloud Dataflow Pricing

You pay for the resources your job uses, billed per second. These list prices appear to be for the Iowa region, and Mumbai can differ, so check the official pricing page before you plan a budget.

Resource	Batch	Streaming
vCPU per hour	$0.056	$0.069
Memory per GiB per hour	$0.003557	$0.003557
Shuffle data per GiB	$0.011	$0.018

Here is a rough example for one worker with 2 vCPUs and 7.5 GiB of memory, counting only CPU and memory:

Batch job for 1 hour: About $0.14.
Streaming job for a full month: About $120, because it never stops.

Your real bill can also include disks, shuffle data, Streaming Engine, BigQuery, Pub/Sub and Cloud Storage. The big lesson is simple: the danger is a streaming job you forgot about, not a batch job that took a few extra minutes.

If a batch job is not urgent, look at Flexible Resource Scheduling, which uses lower vCPU and memory rates.

Google Cloud Dataflow Mistakes to Avoid

Forgetting a streaming job: Stop test jobs when you finish, and set a billing alert.
Testing on the full dataset first: Run a small sample locally, then a small cloud job.
Keeping services in different regions: Run the job close to its bucket and BigQuery dataset.
Using no worker limit: Set a maximum number of workers, especially while learning.
Using latest templates in production: Google warns they can include breaking changes, so use a dated version.
Using Dataflow for a small job: If a simple script or SQL query works, keep it simple.

Conclusion

Google Cloud Dataflow takes data from one place, processes it at any size, and delivers it where you need it. Use batch for scheduled reports and streaming for live data. Let Google handle the workers while you focus on the logic.

Start with Example 1 on your laptop, since it costs nothing on Google Cloud. Once that makes sense, try Example 2 in a test project, and remember to stop the job. For more details, read the official Google Cloud Dataflow documentation.

FAQs

Q1: Is Google Cloud Dataflow free?
No. You pay for the resources your jobs use. You can test Apache Beam code on your own computer without running a Dataflow job.

Q2: What is the difference between Google Cloud Dataflow and Apache Beam?
Apache Beam is the library you use to write a pipeline. Dataflow is the Google Cloud service that runs it.

Q3: Do I need to know Java to use Google Cloud Dataflow?
No. You can use Python or Go, or run a ready-made template without writing a pipeline.

Q4: What is the difference between drain and cancel in Google Cloud Dataflow?
Drain stops new input and finishes the work in progress. Cancel stops right away, so in-progress data can be lost.

Q5: Is Google Cloud Dataflow an ETL tool?
It is commonly used for ETL. It can extract data, transform it and load it into systems such as BigQuery.

Q6: Should I use Google Cloud Dataflow or Dataproc?
Choose Dataflow for new managed pipelines built with Apache Beam. Choose Dataproc if you already have Spark or Hadoop code.

Google Cloud Dataflow: What It Is, When to Use It, and How to Use It