What would you like to happen?
The BigTable connector via BigtableIO.Read currently only supports reading from BigTable tables via withTableId(). There is no way to read from the new continuous materialized view tables (CMV), which are a separate resource type (materializedViews/ vs tables/).
The underlying google-cloud-bigtable Java client already supports CMVs — Query.create() and sampleRowKeys() both accept materialized view resource names as of v2.55.0. However, BigtableIO.Read has no option to specify a materialized view name, so this capability is not accessible from Beam pipelines.
Use case:
We use BigTable CMVs to maintain pre-aggregated index views over our data. We need to read from these views in Dataflow pipelines for downstream processing. Currently we can only access CMVs via the REST API or the Java client directly, but not through BigtableIO.
Expected behavior:
A withMaterializedViewName(String) method (or similar) on BigtableIO.Read, mutually exclusive with withTableId(), that routes the read through the existing client support for materialized views.
Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
Issue Components
What would you like to happen?
The BigTable connector via
BigtableIO.Readcurrently only supports reading from BigTable tables viawithTableId(). There is no way to read from the new continuous materialized view tables (CMV), which are a separate resource type (materializedViews/vstables/).The underlying google-cloud-bigtable Java client already supports CMVs —
Query.create()and sampleRowKeys() both accept materialized view resource names as of v2.55.0. However,BigtableIO.Readhas no option to specify a materialized view name, so this capability is not accessible from Beam pipelines.Use case:
We use BigTable CMVs to maintain pre-aggregated index views over our data. We need to read from these views in Dataflow pipelines for downstream processing. Currently we can only access CMVs via the REST API or the Java client directly, but not through BigtableIO.
Expected behavior:
A
withMaterializedViewName(String)method (or similar) onBigtableIO.Read, mutually exclusive with withTableId(), that routes the read through the existing client support for materialized views.Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
Issue Components