Orchestration Without the Logo Soup

Interview-grade notes for the technical (it-ml) track. The concepts above are tool-agnostic; these are the sharp edges you hit once you operate a real scheduler.

Cron (time-triggered) vs event-triggered. A cron schedule fires on a wall-clock cadence — 0 2 * * * means "02:00 every day" (minute, hour, day-of-month, month, day-of-week). It is simple and predictable but blind: it fires whether or not the upstream data has actually arrived, so a cron DAG that assumes yesterday's file is present will run on an empty input if the file is late. An event-triggered (sensor / data-aware) run instead fires when a condition is met — a file lands, a message arrives, an upstream dataset is marked ready. Event triggers couple the run to data readiness rather than to the clock, at the cost of needing something to watch for the event. Many production DAGs combine both: a cron cadence with a sensor at the top that waits (up to a timeout) for the data before releasing the rest of the graph.

The scheduling interval vs the execution date. A subtle, much-misunderstood point: a batch scheduled "for Monday" typically runs after Monday closes (early Tuesday), because only then is Monday's data complete. Orchestrators model this as a distinct logical/execution date (the period the run is for) separate from the wall-clock run time (when it actually executes). Every job should be parameterised on the logical date and read only that period's data — never today() — or the same code produces different results depending on when it happens to run, breaking both determinism and backfills.

Retries, backoff, and idempotency contracts. Retry policy is usually max_retries + a backoff curve (fixed, linear, or exponential — exponential with jitter is the default that avoids synchronised retry storms against a shared source). A retry re-executes the task, so the task must be idempotent at task grain, not just the pipeline overall: writing to a temp location and atomically swapping on success (write-then-rename), or upserting by key, both give this; appending does not. Distinguish transient failures (retry) from permanent ones (a schema violation — retrying is pointless, fail fast and alert).

DAG execution order. The scheduler runs a topological sort of the DAG: a task becomes eligible only when every upstream task has reached a success (or an explicitly allowed) state, and independent branches are dispatched concurrently up to a configured parallelism / pool limit. Failure semantics are policy: a failed task normally marks its downstream as upstream_failed and skips them, but you can declare trigger rules (run on all-success, one-success, all-done, etc.) for cleanup tasks that must run even after a failure. Two operational hazards worth knowing: backfilling a wide date range can overwhelm a shared source if every logical date is dispatched at once (bound it with per-pool concurrency), and overlapping runs of the same DAG (a run still going when the next fires) need max_active_runs = 1 or idempotent writes to avoid two runs clobbering the same period.