Google Cloud Dataflow
E97117
Google Cloud Dataflow is a fully managed service for developing and executing batch and streaming data processing pipelines, based on Apache Beam, within the Google Cloud ecosystem.
Aliases (1)
- Dataflow ×1
Statements (63)
| Predicate | Object |
|---|---|
| instanceOf |
Google Cloud Platform service
→
cloud data processing service → managed service → |
| abstracts |
autoscaling
→
cluster management → job monitoring infrastructure → resource provisioning → |
| basedOn |
Apache Beam
→
|
| billingModel |
pay-as-you-go
→
|
| consoleUrl |
https://console.cloud.google.com/dataflow
→
|
| deploymentModel |
serverless
→
|
| developedBy |
Google
→
|
| documentationUrl |
https://cloud.google.com/dataflow
→
|
| executionModel |
unified batch and streaming model
→
|
| feature |
SQL-based pipelines via Dataflow SQL
→
autoscaling of workers → classic templates → dynamic work rebalancing → exactly-once processing semantics for many operations → flex templates → pipeline visualization in Google Cloud console → shuffle service → stateful processing → streaming engine → windowing and triggers → worker logging and metrics → |
| languageSupport |
Go
→
Java → Python → |
| logsTo |
Cloud Logging
→
|
| monitoredBy |
Cloud Monitoring
→
|
| offers |
availability across multiple Google Cloud regions
→
regional job execution → |
| partOf |
Google Cloud Platform
→
|
| provides |
fully managed execution environment
→
|
| resourceType |
Dataflow job
→
Dataflow template → Dataflow worker → |
| securityFeature |
Customer-managed encryption keys support
→
IAM-based access control → VPC Service Controls support → |
| supports |
batch data processing
→
streaming data processing → |
| supportsIntegrationWith |
BigQuery
→
Bigtable → Cloud Datastore → Cloud Functions → Cloud Logging → Cloud Monitoring → Cloud SQL → Firestore → Google Cloud Storage → Pub/Sub → Spanner → Vertex AI → |
| supportsUseCase |
ETL pipelines
→
IoT data processing → data warehousing ingestion → event processing → log processing → machine learning data preparation → real-time analytics → |
| uses |
Apache Beam SDKs
→
|
Referenced by (4)
| Subject (surface form when different) | Predicate |
|---|---|
|
Google BigQuery
→
Google Cloud Pub/Sub → |
integratesWith |
|
Google I/O 2014
→
|
announced |
|
Google Cloud
("Dataflow")
→
|
hasComponent |