feat: add historical backfill support to SummaryRules#1102
Open
feat: add historical backfill support to SummaryRules#1102
Conversation
Add spec.backfill and status.backfill fields to the SummaryRule CRD, enabling users to request processing of arbitrary historical time ranges. Key design: - requestId for resume/restart semantics - Separate cursor from live scheduling (does not touch LastSuccessfulExecution) - Bounded concurrency via maxInFlight (default 1, low-priority background task) - No skipped intervals: failed Kusto async ops are re-queued for retry - No overlapping intervals: sequential cursor + dedup guard - Generation pinning: spec changes mid-backfill fail the request - Shared backfill package (pkg/crd/summaryrule/backfill/) used by both the ADX Exporter and Ingestor, avoiding logic duplication Files changed: - api/v1/summaryrule_types.go: BackfillSpec, BackfillStatus, BackfillOperation types - api/v1/conditions.go: ConditionBackfill constant - api/v1/zz_generated.deepcopy.go: DeepCopy for new types - pkg/crd/summaryrule/backfill/: shared backfill processing logic + 25 tests - adxexporter/summaryrule.go: wire backfill into Reconcile loop - ingestor/adx/tasks.go: wire backfill into Run loop - kustomize/bases/ + operator/manifests/crds/: CRD schema updates - docs/designs/summary-rules.md: backfill usage documentation - docs/crds.md: updated CRD reference table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces historical backfill support for SummaryRule by extending the CRD (spec.backfill + status.backfill) and wiring shared backfill processing logic into both the ADX Exporter reconciler and the Ingestor SummaryRule task loop.
Changes:
- Add new
BackfillSpec/BackfillStatus(+ phases/operation tracking) to theSummaryRuleAPI and regenerate deepcopy + CRD schemas. - Introduce
pkg/crd/summaryrule/backfillshared package (with tests) to generate windows, gate concurrency, submit async ops, and track progress. - Wire backfill processing into
adxexporter/summaryrule.goreconcile andingestor/adx/tasks.gorun loop; update docs and CRD reference.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/crd/summaryrule/backfill/backfill.go | Core shared backfill processing (window generation, polling, retries, completion). |
| pkg/crd/summaryrule/backfill/backfill_test.go | Unit tests covering backfill state transitions, batching, retries, and boundaries. |
| api/v1/summaryrule_types.go | Adds spec.backfill / status.backfill and supporting types. |
| api/v1/conditions.go | Adds ConditionBackfill constant for condition-based signaling. |
| api/v1/zz_generated.deepcopy.go | DeepCopy generation for new API types. |
| adxexporter/summaryrule.go | Calls shared backfill processing from reconciler; adapters for submit/status. |
| ingestor/adx/tasks.go | Calls shared backfill processing from task loop; adapters for submit/status. |
| operator/manifests/crds/summaryrules_crd.yaml | CRD schema updates for backfill fields. |
| kustomize/bases/summaryrules_crd.yaml | CRD schema updates for backfill fields (kustomize base). |
| docs/designs/summary-rules.md | Adds usage/semantics documentation for backfill. |
| docs/crds.md | Updates CRD reference table to mention backfill. |
- honor retryable async operations during backfill polling - fail invalid backfill specs instead of leaving stale status - always persist a valid nextWindowStart when marking backfill failed - mirror backfill phase into status.conditions via ConditionBackfill - document and validate the maxInFlight cap of 20 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add spec.backfill and status.backfill fields to the SummaryRule CRD, enabling users to request processing of arbitrary historical time ranges.
Key design:
Files changed: