MSP On AI Vendor Vetting Risks: Don’t Let ‘Three Chatbots In A Trench Coat’ Sell You AI
“I can’t tell you how many vendors say, ‘We’ve got AI.’ But when I ask how many MSPs they’ve worked with, the answer is zero. That’s not innovation. That’s a science fair project,” says Dawn Sizer, CEO of 3rd Element Consulting.
With AI everywhere and embedded into almost every technology, one MSP CEO urged her peers to understand and recognize the risks of buying into AI too fast.
“This isn’t like picking your next PSA,” said Dawn Sizer, CEO of Mechanicsburg, Pa.-based 3rd Element Consulting. “Vet your AI vendors like you’re building a spaceship. Because if it breaks in orbit, you’re not coming back easily.”
Sizer spoke to a packed room of MSPs at CRN parent The Channel Company’s 2025 XChange NexGen conference, being held in Houston this week, about practical questions MSPs should be asking when vetting an AI vendor.
One of her central messages was to stop accepting vague answers or tech buzzwords in place of real, accountable information.
“I want to know who’s flying the ship,” she said. “What does the team look like? Are there real AI engineers on staff, or is this just three chatbots in a trench coat?”
What she’s seeing is many vendors trying to break into the MSP market by gluing together AI tools without real use cases, funding or operational security.
“I can’t tell you how many vendors say, ‘We’ve got AI.’ But when I ask how many MSPs they’ve worked with, the answer is zero,” she said. “That’s not innovation. That’s a science fair project.”
And vetting AI vendors is not the same as vetting traditional vendors, according to Sizer. Key questions include how long a vendor’s funding will last, who’s backing the company and whether it is built to scale or just to sell.
Another aspect MSPs should consider is the legal side of AI: Are they incorporated in your country? Do they hold data overseas? Does an MSP own the data once it’s put it in?
“These are not fine-print questions. These are front-page questions,” she said. “Ask to see their architecture. If they can’t draw you a diagram, even a napkin sketch, they don’t understand their own system or they don’t want you to. Either way, walk away.”
She even stressed data security and storage, challenging MSPs in the room to consider how much they really know about where their clients’ data ends up.
“Everyone in here is using AI in some way, shape or form. But do you know where your data goes from Point A to Point C?” she said. “Do you know how many hops it makes? Who can see it? If you don’t, that’s a huge risk.”
Sizer also touched on emergency controls, bias mitigation and proven security practices and why MSPs should dig even deeper to learn about a vendor’s AI practices.
“Ask them how they stop the AI if it starts doing weird things. Ask about hallucinations. Ask if you can unplug it. This is not the time to ‘hope for the best,’” she said.
To close out the session, she encouraged MSPs to start being strategic investigators as they are on the frontlines when it comes to their clients. Don’t just buy the shiny new thing because it’s shiny, she said, and make sure the AI tools solve real problems.
“Make sure your people trust it,” Sizer said. “Make sure you can trust the people behind it. Otherwise, you’re not flying the ship, you’re strapped to the outside of it.”
Travis Woods, CEO of San Francisco-based Fort Point IT, agreed with Sizer’s sentiment and said MSPs should look deeper when shopping for AI vendors.
“My biggest takeaway is that as MSPs who are rapidly adopting AI, we need to ask deeper questions about our vendors and require more transparency to ensure that our customers’ data, our own data, is being protected as it should be,” he said.