What ML Engineers Actually Do
ML Engineers are responsible for the entire lifecycle of ML models in production—everything that happens after a Data Scientist says "the model works in my notebook."
A Day in the Life
Model Deployment & Serving (Core Responsibility)
The primary job of an ML Engineer is getting models into production and keeping them running:
- Serving infrastructure — Building REST APIs, gRPC endpoints, batch inference jobs, and real-time prediction pipelines
- Model packaging — Containerizing models with Docker, managing dependencies, ensuring reproducible deployments
- A/B testing frameworks — Canary deployments, gradual rollouts, feature flags for model versions
- Latency optimization — Model quantization, pruning, caching strategies, GPU optimization
- Edge deployment — Mobile inference (TensorFlow Lite, Core ML), IoT devices, on-device processing
MLOps Infrastructure
ML Engineers build and maintain the infrastructure that makes ML sustainable:
- CI/CD for ML — Automated testing for models, data validation, deployment pipelines that handle model artifacts (not just code)
- Experiment tracking — MLflow, Weights & Biases, or custom systems for tracking experiments, hyperparameters, and results
- Model registry — Versioning models, managing model lineage, tracking which model is deployed where
- Feature stores — Centralized feature management ensuring consistency between training and serving (Feast, Tecton)
- Automated retraining — Pipelines that detect model degradation and trigger retraining when needed
Production Monitoring & Reliability
Unlike traditional software, ML systems can fail silently—the model still returns predictions, but they're increasingly wrong:
- Data drift detection — Monitoring input distributions, alerting when production data differs from training data
- Model performance tracking — Tracking prediction quality over time, not just system metrics
- Prediction logging — Capturing predictions for debugging, auditing, and feedback loops
- Fallback mechanisms — What happens when the model fails? Graceful degradation, simpler backup models
- Error handling — Input validation, outlier detection, schema enforcement
ML Engineer vs. Data Scientist: Understanding the Distinction
This is the most common source of confusion in ML hiring. Getting it wrong leads to frustration on both sides.
Data Scientists
- Focus: Model development, experimentation, statistical analysis, understanding what's possible
- Environment: Jupyter notebooks, research mode, iterating on model accuracy
- Success metrics: Model accuracy, AUC-ROC, business insights derived from data
- Typical day: Running experiments, analyzing feature importance, presenting findings to stakeholders
- Tools: pandas, scikit-learn, Jupyter, visualization libraries, statistical packages
ML Engineers
- Focus: Production systems, reliability, scalability, making ML actually work in the real world
- Environment: Production codebases, CI/CD pipelines, monitoring dashboards
- Success metrics: Model latency, uptime, deployment frequency, cost efficiency
- Typical day: Debugging production issues, optimizing serving infrastructure, building deployment automation
- Tools: Docker, Kubernetes, TensorFlow Serving, MLflow, Prometheus, feature stores
The Overlap Zone
Some professionals can do both—these are sometimes called "Full Stack ML Engineers" or "Applied Scientists." But most ML systems benefit from specialization:
- Data Scientists who can deploy simple models to production
- ML Engineers who understand model internals enough to optimize deployment
Hiring mistake: Expecting one person to do both at a high level. The skill sets are different. Research shows teams are more productive when roles are specialized.
Skill Levels: What to Expect
Career Progression
Curiosity & fundamentals
Independence & ownership
Architecture & leadership
Strategy & org impact
Junior ML Engineer (0-2 years)
Capabilities:
- Deploys models using established patterns and existing infrastructure
- Writes Python code following team conventions
- Monitors models using existing dashboards
- Debugs basic deployment issues with guidance
- Understands fundamental ML concepts
Learning areas:
- Production architecture design
- Complex MLOps pipeline construction
- Performance optimization
- Handling production incidents independently
Mid-Level ML Engineer (2-5 years)
Capabilities:
- Designs ML serving systems from scratch
- Implements monitoring and alerting for models
- Optimizes model latency and cost
- Handles production incidents independently
- Makes informed trade-off decisions (batch vs. real-time, cost vs. latency)
- Mentors junior engineers on deployment practices
Growing toward:
- Platform architecture
- Cross-team technical leadership
- Strategic tooling decisions
Senior ML Engineer (5+ years)
Capabilities:
- Architects ML platforms that scale across teams
- Sets MLOps standards and best practices for the organization
- Makes build vs. buy decisions for ML infrastructure
- Mentors Data Scientists on production best practices
- Influences technical roadmap and vendor selections
- Handles complex, ambiguous technical challenges
Demonstrates:
- System thinking at organizational scale
- Business impact awareness
- Technical leadership without authority
What to Look For by Use Case
Different ML applications require different skill emphases:
Real-Time Inference (Recommendations, Fraud Detection, Personalization)
- Priority skills: Low-latency serving, model optimization, caching strategies, streaming inference
- Interview signal: "How would you serve predictions with <10ms latency at 10K requests/second?"
- Key tools: TensorFlow Serving, TorchServe, Triton Inference Server, Redis, ONNX Runtime
- Trade-offs: Accuracy vs. latency, model complexity vs. serving cost
Batch Processing (Analytics, Risk Scoring, Periodic Updates)
- Priority skills: Distributed computing, cost optimization, Spark/Dask experience
- Interview signal: "How would you process predictions for 100 million records daily?"
- Key tools: Spark, Airflow, dbt, batch inference frameworks
- Trade-offs: Throughput vs. freshness, cost vs. processing time
Computer Vision (Image/Video Processing)
- Priority skills: GPU optimization, model compression, preprocessing pipelines
- Interview signal: "How would you deploy a real-time object detection model?"
- Key tools: TensorRT, ONNX, OpenCV, specialized vision serving frameworks
- Trade-offs: Accuracy vs. speed, GPU cost vs. latency
NLP / Large Language Models
- Priority skills: Token optimization, prompt engineering infrastructure, model quantization, context management
- Interview signal: "How would you serve a large language model efficiently for 1000 concurrent users?"
- Key tools: Hugging Face Transformers, vLLM, TensorRT-LLM, quantization libraries
- Trade-offs: Model size vs. serving cost, latency vs. quality
Where to Find ML Engineers
High-Signal Channels
- GitHub: Contributors to MLflow, Feast, TFX, Kubeflow, Ray, or other MLOps projects
- MLOps Community: Active participants in ML infrastructure discussions
- KubeCon and MLOps World: Conference speakers and attendees
- Tech blogs: Engineers writing about production ML challenges
- Company alumni: Engineers from companies known for production ML (Uber, Netflix, Spotify, Stripe)
Talent Pools by Background
| Background | Strengths | Growth Areas |
|---|---|---|
| Backend Engineers → ML | Strong production skills, reliability mindset | ML fundamentals, model understanding |
| Data Scientists → MLE | ML knowledge, model intuition | Software engineering rigor, production operations |
| DevOps → MLOps | Infrastructure expertise, operational skills | ML-specific challenges, model monitoring |
| Research → Production | Deep ML understanding | Production mindset, software engineering |
Red Flags When Sourcing
- Only research or academic experience with no production deployments
- "ML Engineer" titles that were actually Data Science work
- Can't discuss latency, reliability, or monitoring
- No experience with containerization or cloud infrastructure
- Overemphasis on algorithms without deployment context
Common Hiring Mistakes
1. Confusing ML Engineers with Data Scientists
The mistake: Posting an "ML Engineer" role but actually needing someone to build models in notebooks.
The fix: Be clear about whether you need someone to train models or deploy them. If you need both, either hire two specialists or explicitly seek a "Full Stack ML Engineer" with appropriate expectations.
2. Overweighting Research Credentials
The mistake: Preferring PhD candidates and published papers when you need production systems.
The fix: Academic ML research and production ML are different disciplines. A researcher who published at NeurIPS may struggle with Docker and Kubernetes. Ask about production deployments, not publication records.
3. Ignoring Software Engineering Skills
The mistake: Hiring someone who knows ML but can't write production code—no tests, no code review experience, no debugging skills.
The fix: ML Engineers are software engineers first. Require strong Python proficiency (production code, not notebooks), testing practices, and familiarity with software development workflows.
4. Not Testing MLOps Knowledge
The mistake: Interviewing only on ML algorithms, not on deployment, monitoring, or operations.
The fix: Include questions about CI/CD for ML, model monitoring, data drift, retraining pipelines. These differentiate ML Engineers from Data Scientists.
5. Unrealistic Tech Stack Requirements
The mistake: Requiring PyTorch AND TensorFlow AND JAX AND specific cloud platforms AND specific tools.
The fix: Strong engineers learn new tools quickly. Focus on fundamental skills (Python, containerization, APIs) and one ML framework. Be flexible on the rest.
Interview Approach
Technical Assessment Areas
System Design (Must Include)
- "Design a system to serve ML recommendations for an e-commerce site with 1M daily users"
- "How would you build a fraud detection system that needs to respond in under 100ms?"
- Look for: latency considerations, caching strategies, fallback mechanisms, monitoring approach
MLOps Scenarios (Must Include)
- "How would you handle model retraining when underlying data changes?"
- "A model's predictions have degraded in production. Walk me through your investigation."
- Look for: systematic debugging, understanding of data drift, monitoring awareness
Production Experience (Must Include)
- "Tell me about a production ML system you built and operated"
- Look for: specific scale numbers, challenges faced, lessons learned
Code Assessment
Option A: Take-home exercise
- Build a simple model serving API with monitoring
- Pay for exercises over 2-3 hours
Option B: Live coding
- Debug a model serving issue
- Extend an existing inference pipeline
- Review ML deployment code for production readiness
Red Flags in Interviews
- Only notebook experience, never deployed to production
- Can't discuss latency, SLAs, or reliability
- Doesn't understand software engineering fundamentals
- Focuses on model accuracy without considering production constraints
- No experience with monitoring or debugging production systems
- Blames "the infrastructure team" for deployment issues
Developer Expectations
| Aspect | ✓ What They Expect | ✗ What Breaks Trust |
|---|---|---|
| ML Infrastructure & Tooling | →Access to proper ML infrastructure: GPU compute, experiment tracking, model registry, feature stores (or willingness to build them). Modern MLOps tooling that enables productivity rather than fighting infrastructure. | ⚠No dedicated ML infrastructure—everything is manual scripts and ad-hoc processes. No budget for proper tooling. Expecting engineers to do production ML on laptop CPUs. Legacy systems with no path to modernization. |
| Clear Role Boundaries | →Clear division between ML Engineering and Data Science responsibilities. ML Engineers deploy and operate models; Data Scientists train them. Collaboration between roles, but not doing both jobs. | ⚠Expecting ML Engineers to also be Data Scientists—training models, doing analysis, AND handling production. Role confusion leading to "jack of all trades, master of none." Constant context switching between research and production. |
| Production Operations | →Reasonable on-call expectations with proper runbooks and incident response processes. Blameless post-mortems. Time allocated for improving reliability and reducing toil. Recognition that ML systems need specialized monitoring. | ⚠Constant firefighting with no time to fix root causes. Blaming individuals for model failures that are systemic issues. No investment in monitoring or observability. Expecting 24/7 availability without compensation. |
| Technical Growth & Impact | →Opportunity to work on interesting scale challenges. Input on architecture decisions. Learning budget for ML infrastructure conferences (MLOps World, KubeCon). Clear path to senior/staff levels. | ⚠Dead-end role maintaining one model forever. No input on technical decisions. Architecture handed down by people who don't understand ML. No professional development budget. |
| Realistic Expectations | →Understanding that ML systems are different from traditional software—they require ongoing maintenance, monitoring, and iteration. Patience for infrastructure work that enables long-term velocity. | ⚠Treating ML as "deploy once and forget." No understanding of model drift, retraining needs, or the operational complexity of ML. Expecting instant results without infrastructure investment. |