MLOps and its levels of maturity

Dmitry Mittov

The mediator between ML models and Production Services must be the MLOps

Metropolis GmbH

CEO Jon Fredersen

Waterfall Project Value

Agile Project Value

Four Key Metrics

%%{init: {"theme": "dark", "themeVariables": {"fontSize": "12px"}, "flowchart":{"htmlLabels":false}}}%%
stateDiagram-v2
    state "Mean Time To Restore" as mttr
    state "Lead Time To Change" as lttc
    state "Deployment Frequency" as df
    state "Change Failure Rate" as cfr
    state "Productivity" as p
    mttr --> p
    lttc --> p
    df --> p
    cfr --> p

MLOps

CTO Rotwang accepts the challenge

ML Tools

Creating a Company MLOps Platform

Official Acceptance

Onboarding in Teams

Possible Pitfalls

• Prioritirisation
• Onboarding Strategy
• Platform team responsibility
• Misunderstanding the needs of the ML teams
• …

ML in production

The tower of Babel

SWE Grot Protects Production Environment

MLOps Maturity Models

• Google model
  • 3 levels
  • Focus: Architecture
• Microsoft model
  • 5 levels
  • Focus: Process & People

Microsoft MLOps levels

Level	Name
0	No MLOps
1	DevOps no MLOps
2	Automated Training
3	Automated Model Deployment
4	Full MLOps Automated Retraining

Level 0: No MLOps

• High failure rate
• Long time to restore

Level 1: DevOps no MLOps

• (partially) Failure Rate
• Time to Restore

Level 2: Automated Training

• Failure Rate
• Lead Time to Change

Level 3: Automated Model Deployment

• Deployment Frequency

Level 4: Full MLOps Automated Retraining

Removes the toil work:

• Deployment Frequency
• Mean Time to Restore

Happy end

Conclusions

• Roadmap > Final State
• Iterate faster with smaller steps
• Measure the impact