ARMS Kafka Workload Classifier
Apache Kafka cluster workload classifier
2 min readupdated
A lightweight model built on LogisticRegression from the scikit-learn library (paired with StandardScaler for feature normalization) that identifies an Apache Kafka cluster's current operating mode using four system metrics. It classifies workloads into real-time streaming (REAL_TIME_EVENT_DRIVEN) and resource-heavy batch processing (BATCH_DATA_INTENSIVE). Created by Nwaoghor Praise Chukunweiken as part of a master's thesis, it serves as the decision module for the ARMS system to select the right recovery strategy upon broker failure instead of relying on a single rigid configuration. Useful for DevOps engineers and distributed systems developers experimenting with adaptive message broker management.
Key features
- Operates on just four Prometheus input metrics: request rate variance, incoming message variance, disk write rate, and available memory.
- Inference takes less than 0.1 ms on CPU, and the model ships as two compact joblib/pickle files.
- Paired with ARMS logic, it selects aggressive rebalancing (
QUICK_REBALANCE) for real-time workloads or a conservative mode (RESOURCE_OPTIMIZED) for batch workloads at 90%+ confidence, falling back to gradualCONTROLLED_GRADUALin borderline cases. - The public repository includes a ready-to-use Docker stack with Kafka, Prometheus, and Grafana, an
inference.pyscript, and code for retraining on custom data.
What you can do
- Integrate the classifier into a Prometheus monitoring pipeline to automatically switch Kafka broker rebalancing settings as traffic profiles shift.
- Retrain the model on metrics from your own production cluster using the repository's
train_model.pyscript.
Example
import joblib, pandas as pd
from huggingface_hub import hf_hub_download
repo = "nwaoghorpraise/arms-kafka-workload-classifier"
model = joblib.load(hf_hub_download(repo, "kafka_logistic_regression_model.pkl"))
scaler = joblib.load(hf_hub_download(repo, "kafka_feature_scaler.pkl"))
snapshot = pd.DataFrame([[96.19, 5.50e7, 235.35, 5.176e9]], columns=["request_rate_variance", "node_disk_write_bytes_total", "message_rate_variance", "node_memory_MemAvailable_bytes"])
print(model.predict_proba(scaler.transform(snapshot))[0])
Read more
DetailsHoldout accuracy (600 samples) is 0.9617, and 5-fold cross-validation accuracy is 0.9677.
Within ARMS, it cuts mean recovery time for real-time workloads by 6.14 s (p < 0.001).
Trained on 3,000 synthetic snapshots from a single 3-broker Docker cluster; fine-tuning on your own metrics is recommended for production.
Requires
scikit-learn 1.6.x; files are saved in pickle format.









Comments