Automating and orchestrating ML pipelines

Why pipelines

A notebook is a one-off. A pipeline is a repeatable, parameterised graph of steps (ingest, validate, transform, train, evaluate, register, deploy). Benefits: reproducibility, automation, caching, auditability, testing and the ability to retrain on a trigger. Google describes MLOps maturity as manual (level 0), pipeline automation with continuous training (level 1), and CI/CD plus automated pipeline deployment (level 2).

Developing end-to-end pipelines

Typical stages

1. Data extraction (BigQuery, Cloud Storage, Dataflow).
2. Data validation: schema, ranges, missing values, skew and drift against a reference statistic. Fail or branch the pipeline on bad data.
3. Preprocessing / feature engineering (Dataflow, BigQuery, TFT).
4. Training (custom job, AutoML, hyperparameter tuning).
5. Evaluation: compute metrics on a held-out set, compare to a threshold and to the currently deployed model.
6. Model validation: fairness, slice metrics, infrastructure check (does it load and respond within latency?).
7. Conditional step: register and deploy only if the candidate beats the champion.
8. Metadata and logging of everything.

Data and model validation

• Data validation (TFX StatisticsGen, SchemaGen, ExampleValidator, or TensorFlow Data Validation / TFDV) computes statistics, infers a schema and flags anomalies, training-serving skew and drift. Great Expectations and BigQuery assertions do similar jobs.
• Model validation (TFX Evaluator, Vertex AI Model Evaluation): evaluate against absolute thresholds, against a baseline model, and by slice (subgroups). A blessed model passes through to the Pusher. The pipeline’s Condition step gates deployment.
• Add unit tests for pipeline components and data contract tests before any retraining.

Consistent preprocessing between training and serving

Training-serving skew is a leading cause of production failures. Fixes, from strongest to weakest:

1. Put preprocessing in the model: TF Transform graphs, Keras preprocessing layers, scikit-learn Pipeline objects, or BQML TRANSFORM. The artifact applies the same logic at inference.
2. Shared feature definitions in Feature Store: both training (offline) and serving (online) read the same feature definitions.
3. One shared library or container imported by both training and the serving code.
4. Run batch and streaming through the same Apache Beam code (Dataflow).
5. Avoid rewriting transformations in Java/JS for the serving team.

Also log the serving inputs so they can be compared with training statistics (the basis of skew monitoring).

Components, parameters, triggers and compute

• A component is a containerised step with typed inputs and outputs (artifacts: datasets, models, metrics; parameters: strings, numbers). Lightweight Python components (@dsl.component) wrap a function. Container components wrap any image. Google Cloud Pipeline Components (GCPC) are prebuilt steps (create dataset, custom training job, hyperparameter tuning, model upload, deploy, batch prediction, BigQuery query).
• Parameters (pipeline arguments) hold things like the date range, learning rate and project. Runs from the same compiled template with different parameters.
• Compute needs: set CPU, memory, GPU and accelerator type per component. Heavy work belongs in a managed job (custom training, Dataflow, BigQuery) launched by a light component, rather than in the pipeline step itself.
• Caching: unchanged steps with identical inputs are skipped. Turn caching off for steps with external side effects or when data in place changed.
• Triggers (below). Cloud Build runs builds and tests and can submit a pipeline. Cloud Run (functions or services) can host glue code: receive a Pub/Sub message or Eventarc event and call the pipeline submit API.

Orchestration frameworks

Rule of thumb: ML lifecycle (train, evaluate, deploy) is Vertex AI Pipelines. Broad data-platform scheduling with ML as one task is Cloud Composer. Portable Kubernetes-based ML is Kubeflow Pipelines.

System design with TFX or the Kubeflow DSL

• TFX components: ExampleGen (ingest and split), StatisticsGen, SchemaGen, ExampleValidator, Transform, Trainer, Tuner, Evaluator, InfraValidator, Pusher, BulkInferrer. They run on Apache Beam, and on Google Cloud the Dataflow runner supplies scale for ExampleGen, Transform and Evaluator.
• KFP DSL defines the graph in Python: @dsl.pipeline, @dsl.component, dsl.Condition, dsl.ParallelFor, dsl.Collected, set_cpu_limit, set_accelerator_type, set_caching_options. Compile to a YAML (or JSON) template, store it in Artifact Registry, and run with PipelineJob.
• Dataflow steps in a pipeline: use DataflowPythonJobOp/Flex Template launchers or a TFX component running on the Dataflow runner.
• Separate pipeline definition (code in Git) from pipeline runs (parameters). Version compiled templates.

Hosting third-party pipelines (for example MLflow)

• MLflow tracking and registry can run on Cloud Run or GKE, with a Cloud SQL backend store and a Cloud Storage artifact store. Authenticate with IAP or service accounts.
• Alternatively use Vertex AI Experiments and Model Registry as the managed equivalent, and migrate or dual-log.
• Other third-party orchestrators (Airflow on Composer, Argo, Jenkins, Kubeflow) can run on GKE or Compute Engine. Choose Vertex-native services unless a requirement says otherwise.

Hybrid and multicloud strategies

• Kubeflow Pipelines / TFX are portable: the same pipeline definition runs on GKE, on-prem Kubernetes (GKE Enterprise / Anthos) or other clouds. Vertex AI Pipelines is Google Cloud only.
• Containerise everything (Artifact Registry) so steps run anywhere. Store data where policy requires and use Interconnect / VPN and BigQuery Omni for cross-cloud analytics.
• Use open formats (Parquet, ONNX, TF SavedModel) and open frameworks to avoid lock-in. Cross-cloud Interconnect reduces egress and latency.
• Edge: train in the cloud, export TF Lite models, deploy to devices, and send telemetry back.
• Trade-off: portability costs operational complexity. Without a requirement, prefer managed Vertex AI.

Automating model retraining

Determining a retraining policy

Choose the policy using: how fast the world changes, label delay, training cost, risk of a bad auto-deployed model. Retraining too often costs money and can destabilise. Too rarely causes stale models. Gate automatic retraining: validate data, compare the candidate with the champion, and require approval or a canary rollout for sensitive models. Decide the training window (growing vs sliding) and whether to fine-tune from the previous model (warm start) or retrain from scratch.

CI/CD for model deployment

ML needs CI, CD and CT (continuous training).

Good practice: separate dev / staging / prod projects, deploy pipeline code from Git (infrastructure and pipelines as code, for example with Terraform), use service accounts for each environment, scan images, pin dependency versions, and never deploy from a laptop. Jenkins works if it is the existing CI server and can call gcloud, the SDK or REST. Cloud Build is the native choice. Model deployment tests: schema compatibility, latency under load, a prediction sanity test, and comparison to the champion.

Tracking and auditing metadata

Auditing questions follow a pattern: “which dataset and code produced the model in production, and which models used this dataset?” Answer: ML Metadata lineage plus Model Registry links, with versioned datasets and Git commits as the artifact references. Model cards and evaluation records attached to the registry document intended use and limits.

• Hook into model versioning: register each trained model as a new version, record the training pipeline run ID and dataset URI in labels or metadata.
• Dataset versioning hooks: snapshot the BigQuery table (or save the query + timestamp with FOR SYSTEM_TIME AS OF) per run, and record the version in the run parameters.
• Compare artifacts: use the Experiments table and the metadata lineage graph to see differences between model versions and what upstream data changed.