Once you’ve settled the “build vs. buy” debate, your research enters a new phase: how to choose a machine learning operations (MLOps) vendor.
That first decision is thorny enough. (Hint: Unless MLOps is a core competency, save your money, time, energy, and focus.) But choosing an MLOps vendor will start a cascade of questions that can rapidly overwhelm even the savviest business leader. “What MLOps vendors are out there?” “How can I tell them apart?” “What do I really need to know?”
These are important questions. You’re about to enter into a relationship that both you and your vendor hope will thrive for years. Yet, every business leader has a story about a poor vendor choice—all of which resulted in weak results, wasted time, and a lost opportunity for value generation.
So, how can you separate the wheat from the chaff to get the MLOps platform your organization needs?
We’ve put together this list of questions we’d ask if we were selecting an MLOps vendor (instead of using the Striveworks MLOps platform that we built). As you start evaluating your options, make sure your short list has good, trustworthy answers to these questions—or you may end up picking the wrong guy in a striped shirt.
1. How does your MLOps platform handle data governance and compliance?
It may seem strange to start your exploration of MLOps capabilities here, but data governance and compliance are among the most foundational features for reliable AI and machine learning (ML). Organizations that operate in regulated spaces already know this fact. But all organizations should take note of data governance as the AI landscape shifts in coming years.
AI/ML are in their early stages. They’ve only recently left the lab to find useful application in the real world. In fact, many AI frameworks and use cases are so new that legislation hasn’t caught up to them—yet. MLOps platforms that struggle with data governance are at serious risk of displacement when regulators get around to inspecting and establishing guidelines around AI auditability and oversight. If your MLOps vendor is cavalier about security, privacy, and data lineage, you’re at risk of losing access to your critical models when the regulators find out.
Follow-up questions:
- How does your platform handle data encryption?
- Do you have role-based access controls?
- How do you handle versioning and data lineage?
2. Is your platform open or closed?
MLOps isn’t the end point for your data—it’s part of the process for transforming your data into insights. Ideally, this grunt work would disappear into the background. For that to happen, you need an MLOps platform that integrates easily with other solutions you use—data sources, annotation suites, orchestration tools, business intelligence studios, etc. An open platform ensures that you can configure the workflow you need, plugging in MLOps capabilities as appropriate. Closed platforms (you know who you are) prevent these integrations and impose heavy burdens to extracting data, switching vendors, or customizing your solutions. As you screen vendors, envision how MLOps will sit within your workflow—and whether or not the vendor lock-in of a closed platform is worth a “killer feature.”
Follow-up questions:
- What’s your data ownership policy?
- Are your model and data formats proprietary?
- Do you have an open application programming interface (API)?
- What integrations does your platform have?
3. Does your platform have no-code support for any MLOps processes?
User experience is the ultimate make-or-break characteristic of a new technology. A user-friendly workflow is essential for your team to adopt an MLOps platform and use it to advance their productivity. If the experience is lacking, you may as well just use Jupyter Notebook.
A low-code to no-code user interface can greatly accelerate standard parts of the model development process, such as dataset creation, annotation, and training. At the same time, you want to maintain options for code-first users who need more fine-grained control over model building, deployment, and maintenance. Before investing in a relationship with an MLOps vendor, consider how a platform will streamline steps in your team’s workflow to help them work faster or more effectively toward your business outcomes.
Follow-up questions:
- What no-code elements does your platform offer?
- Is a software development kit (SDK) necessary to build and deploy an ML model with your platform?
- What options do you have for code-first users?
- How does your platform save time for both experts and novice users?
4. How does your MLOps platform perform at 50 models? At 5,000?
Today, you may only have a few models in production. But if you are reviewing MLOps platforms, you obviously plan to expand that number in the near future—and you need a platform that grows with you. Ask your candidate vendors how their platforms perform at each tier of service. Can they handle serving thousands of concurrent models without a decline in performance? What are the technological requirements needed for that increase?
Consider pricing as well. Does the vendor charge a standard licensing fee? Or, do they charge per model, GPU hour, or inference? Depending on your use case, you may find that certain pricing models make sense at the start of your MLOps journey—but become prohibitively expensive as your adoption scales.
Follow-up questions:
- Can you explain your process for model licensing? Data licensing?
- How does auto-scaling ensure high availability while minimizing compute costs?
5. Can you deploy on premises?
Flexible deployment may not matter to most organizations, but it’s a critical capability for those who need it. Most corporate enterprises are best off with deploying their MLOps capabilities on a commercial cloud. But plenty of industries, such as defense, intelligence, aid agencies, NGOs, mining, etc., deal with highly sensitive data or operate in remote locations far from robust, affordable internet connectivity. These types of organizations need the ability to deploy machine learning tools on prem—potentially in a disconnected environment. Many MLOps platforms are reliant on commercial cloud technologies to operate, blocking them from projects like delivering a large language model (LLM) and retrieval-augmented generation (RAG) pipeline on an air-gapped, top-secret network. You already know if you need this flexibility for your data—but it’s important to recognize that not all vendors have it.
Follow-up questions:
- What cloud providers have you deployed into?
- Does your platform have an authority to operate in IL4, IL5, IL6, or IL7 environments?
6. What are your areas of expertise in MLOps?
Like all vendors, MLOps providers will be stronger in certain areas based on their technology and customer profiles. Certain companies may have strengths with structured data, others with unstructured data. Some specialize in handling the sensitivities needed for public sector customers while others are commercial-focused. One platform may shine with early stages of the MLOps workflow, like data preparation and annotation. Others may lead the industry in post-production—monitoring, evaluation, and retraining.
There’s no right or wrong answer for which solution is better—just a spectrum of strengths and weaknesses that make platforms right or wrong for your applications. Consider your biggest needs. Are you just getting models into production? Do you have a primary data type you interact with? Are managing drift and remediating degraded models major concerns? Your answers will point you toward one solution over another.
Follow-up questions:
- Does your platform support video streaming?
- Does your platform support multispectral and hyperspectral imagery?
- Does your platform support specialized image formats, such as NITF and TIFF?
- Does your platform fully use metadata from files?
-
How do I deploy models into production with your MLOps platform?
-
How do I maintain models in production using your MLOps platform?
7. How involved is your customer success team?
Any good MLOps vendor will assign you a dedicated account manager when you become a customer. But every customer success and support organization operates differently. Find out how many customers each account manager oversees and what their standard level of involvement is. No one wants a vendor that disappears once your system is up and running. But too much involvement from an account manager can also send up a red flag, indicating problems with the software or other difficulties with self-service.
Follow-up questions:
- How can you prove the value of the platform?
- Can I see your product documentation?
- What training do you offer?
- What’s your experience delivering professional services?
- Which phases of the model life cycle can your team help support?
8. Do you have a case study with a customer who is similar to me?
When free trials aren’t reasonable (a common challenge with enterprise B2B solutions), case studies are your best assurance that an MLOps platform will work for you. Success stories and client testimonials from potential vendors are worth evaluating closely for key details. Specifically, consider:
- What data types were involved?
- What did the vendor do (vs. what did the customer do)?
- What industries were involved?
- What kind of results did the project produce?
- How reproducible are the results?
Obviously, case studies are marketing tools meant to make the vendor look good. But they also contain essential details that suggest whether or not a vendor has the experience to help you in a similar way as their other customers.
Follow-up questions:
- Who was involved with this case study?
- What gave rise to the results?
- Can I speak with your customers?
9. Can you share your product roadmap?
Innovation isn’t mandatory in all areas: A mousetrap is still effective over 100 years after its invention. But AI and ML are changing so rapidly that you want to partner with an MLOps vendor that has clear foresight into developments coming down the pike. Technological obsolescence can ruin the value of your AI models, so you want to make sure that your vendor is investing and developing capabilities that are meaningful to you over the long haul. Look at your prospective vendors’ roadmaps to confirm that they plan to add new, relevant capabilities in the coming months—and that those capabilities meet your expectations for growth with your AI practice.
Follow-up questions:
- What’s your area of focus over the next quarter? And the next year?
- What capabilities are you adding for X?
10. What’s the time to value for your MLOps platform?
Cost makes a difference. That said, for a tool that rapidly accelerates results and profits as much as AI, total cost of ownership is the wrong metric. Instead, explore each platform’s time to value.
AI use cases, models, and data all vary widely. At the same time, each MLOps vendor may use a different pricing model—usage-based pricing, subscription models, model-deployment pricing, or something else entirely. One vendor’s low up-front cost may work best for some scenarios while another’s monthly subscription may make sense for others. Ideally, you want a custom estimate of the time to value for your most challenging problem. From that number, you can extrapolate the value possible from deploying additional models and putting your platform to full use.
Follow-up questions:
- Do you have cost comparisons with platforms?
- What cost-saving programs or features are offered?
***
Choosing an MLOps vendor is a major decision for the maturity of your AI program and your organization as a whole. AI/ML can deliver transformative capabilities, and MLOps platforms are instrumental in putting models into production and keeping them working at scale.
Let these questions guide you in selecting an MLOps vendor that delivers what you need for your organization. With the rise in AI, more options are available than ever before with the expertise in data types, workflows, integrations, deployment options, and industry specifics to propel your AI program forward.
Want to know more about the Striveworks MLOps platform? Request a demo today.
