MLOps case study — EDA to deployment

Bots don't hide from one feature.
They hide from the ensemble.

A full-circle MLOps system for flagging cheaters from player behavioral telemetry — actions per minute, click-interval consistency, reaction time, headshot %, movement entropy — built on a full MLOps toolchain (Kedro, MLflow, Optuna, Dash, Docker) applied to player behavioral telemetry and a multi-method anomaly-detection ensemble.

5,000 players bots / cheaters — tighter, denser cluster

No single feature cleanly separates the two groups — this is one axis-pair from the EDA notebook. It's why detection here runs a multivariate ensemble, not a threshold rule.

Raw sessions to a review queue

five Kedro / entrypoint stages

feature_eng and monitoring both read the same player-level features; training logs every run to MLflow and only promotes a challenger if its weak-label AUC is at least as good as the champion's. Nothing downstream of inference ever bans a player automatically — it hands a ranked list to a human.

Three lenses on the same anomaly

rank-averaged, not voted

Distance-based

Mahalanobis distance

How far a player sits from the "normal" cluster, measured in units that account for how features correlate with each other — not a naive Euclidean distance.

Density-based

Isolation Forest

Anomalies are easier to isolate with random splits than normal points are — so they land at shallower average tree depth. Contamination rate tuned with Optuna.

Reconstruction-based

PCA reconstruction error

Compresses each player into a few principal components, then rebuilds them. Normal players reconstruct cleanly; unusual behavior leaves a large residual.

Only about 6% of true cheaters in this dataset are ever caught and labeled — which means every metric this system reports, weak_label_auc, top_k_capture_rate, is a lower bound against an unlabeled majority, not a real precision/recall guarantee.

That asymmetry is why inference never auto-bans. A false positive here means suspending a legitimate, skilled player — a far more expensive mistake than a missed detection — so the model's only output is a ranked queue for a human reviewer.

common/metrics.py — docstrings state the lower-bound caveat at every call site

Honest tradeoffs

what's simplified, on purpose

Ensemble of three vs. one "best" model: Distance, density, and reconstruction methods have different blind spots. Rank-averaging is a deliberately conservative combination — a max() would over-trigger on any single method's false positive.
SQLite over Postgres: No server process, one portable file, easy in Docker — but weak under concurrent writes. Fine at this scale; flagged in code as the first thing to swap for real production throughput.
CSV-backed catalog bridging a live DB: The real-time loop re-exports a CSV snapshot on each detected change so the file-based Kedro catalog stays in sync with SQLite — a simplified stand-in for how OLTP databases bridge to batch ML pipelines.
Optuna against 7 labeled positives: The tuning objective is a noisy, small-sample estimate. A real anti-cheat team would validate any configuration change against a fresh manual-review batch before trusting an Optuna "winner."
Polling, not an event queue: Real-time inference polls for row-count changes rather than reacting to a message bus. Deliberately simpler than a full event-driven system — "good enough" for many production systems in practice.
No Prefect orchestration layer: The entrypoints are the same programmatic KedroSession building blocks a Prefect flow would wrap in @task/@flow — left as a natural extension rather than complexity a single-VM deployment doesn't need yet.

Toolchain

same stack, new domain

Kedro MLflow Optuna Dash SQLite Docker / Compose GitHub Actions (CI + CD) pytest ruff ty pre-commit SHAP uv

Development milestones → repo location

20 milestones, mapped 1:1

Milestone	Covered	Lives here
01	Project scaffolding — uv / git	`pyproject.toml`, `uv.lock`
02–03	EDA, anomaly-detection taxonomy, SHAP	`notebooks/01 - EDA - Cheat Detection.ipynb`
04–06	Baseline model, tuning, experiment tracking	`pipelines/training/`
07	Feature engineering pipeline structure	`pipelines/feature_eng/`
08	Config-driven hyperparameter search	`training/nodes.py`
09	MLflow hooks, champion/challenger	`hooks.py`, `pipelines/inference/`
10	Data management module	`src/app_data_manager/`
11	Programmatic KedroSession entrypoints	`entrypoints/training.py`, `inference.py`
12	Multi-page Dash UI, callbacks	`src/app_ui/`
13	Model monitoring, data drift	`pipelines/monitoring/`
14	Real-time streaming, pytest	`app_stream_data.py`, `tests/`
15	Code quality, pre-commit	`.pre-commit-config.yaml`
16	CI with GitHub Actions	`.github/workflows/ci.yml`
17	Docker deep-dive	`Dockerfile`, `docker-compose.yml`
18	Cloud deployment, CD	`.github/workflows/cd.yml`
19	Prefect orchestration	not implemented — see tradeoffs above
20	Full docker-compose stack	`docker-compose.yml`

Bots don't hide from one feature.They hide from the ensemble.