Komprise Intros High-Performance, Secure Data Ingestion For AI

‘The same things that help you understand your storage and optimize it, the same capabilities, are applicable to AI, except it's now about the value of the data. It's not about the storage cost of the data. It's about what's in the files. Which files can I bring to AI. Which files can I use AI to get better information on. Those kinds of things. What we've been adding to our product are all these AI data workflow and AI data management capabilities,’ says Komprise co-founder Krishna Subramanian.

Unstructured data management software developer Komprise Tuesday unveiled the general availability of Komprise Intelligent AI Ingest, a new data ingestion engine aimed at high-performance curation of unstructured data.

Komprise Intelligent AI Ingest decreases the risk and cost of making unstructured data available for RAG (retrieval augmented generation) and LLM pipelines, said Krishna Subramanian (pictured), co-founder of the Campbell, Calif.-based company, in an interview with CRN.

“When we look across all the storage that's not databases, anything that is keeping documents, images, videos, network storage, cloud accounts, SaaS accounts, all these places where you could have unstructured data, you can simply point Komprise at them, and Komprise takes an inventory of all the data in these environments,” Subramanian said. “It then builds a catalog, a database of all the metadata about all this data. And with that, we can show businesses how much data is there, what's being used, what's not being used, what's in the data, all these things so that IT can save money on storage by moving the right data to the right place.”

Data for AI purposes require the same management, Subramanian said.

“The same things that help you understand your storage and optimize it, the same capabilities, are applicable to AI, except it's now about the value of the data,” she said. “It's not about the storage cost of the data. It's about what's in the files. Which files can I bring to AI. Which files can I use AI to get better information on. Those kinds of things. What we've been adding to our product are all these AI data workflow and AI data management capabilities.”

Komprise focuses on unstructured data, leaving management of structured data to companies like Snowflake, Subramanian said.

“The required features are different,” she said. “We can export our data to a structured solution if you want. But basically, unstructured data is a very different problem than structured, because it's billions of files. Think about yourself. Look at your phone or maybe look at your computer and all the folders you may have. Maybe you have many drafts of documents. Maybe you have something sitting in the cloud. Think about all these silos of information you have for unstructured data. Somebody needs to organize all that. Somebody needs to get the key information out of all of that,” Subramanian said.

“And 80 percent of it is noise. Somebody has to clean up all that noise, and that requires specialized capabilities. That's what we provide.”

When it comes to AI, ingesting the right data is actually quite important, Subramanian said.

“If you make AI process a million documents and 800,000 of those are junk, you're wasting the compute power of that AI,” she said. “You may actually get wrong answers from it because all its tokens and everything are filled up with nonsense, not with what you actually want. You may be leaking sensitive data you don't even know about. And you may even get wrong answers if you have an old version of a document. So feeding the right data to AI is actually quite important to improve accuracy and reduce costs.”

Komprise Intelligent AI Ingest searches and indexes the documents it ingests, Subramanian said.

“We can provide customers a way to say, ‘Okay, let me eliminate all the out-of-date stuff. Let me eliminate all the irrelevant stuff. Let me eliminate all the sensitive data, because I don't want to send that to AI,’” she said. “They can use filters to narrow it down to the right data. And it could be anything. It could be in AWS Bedrock or OpenAI or Copilot or an on-premises LLM. It's a way to filter across all your storage, find the right data, and then move it. And we move it two times faster as others in the testing that we did, because we optimize that move engine for ingestion.”

Komprise’s technology in general fits the security lens that goes into everything that Evotek, a San Diego, Calif.-based solution provider, which grew from $0 revenue 11 years ago to about $650 million today, said CEO, Chairman, and founder Cesar Enciso.

“Over 50 percent of our revenues comes from cybersecurity,” Enciso told CRN. “When I started the company, we hired 11 CISOs and we started really focusing on advisory services. To us, there’s a lot of risk in unstructured data. And so we were an early adopter of Komprise.”

Data is the core element of modern enterprises, and if a company doesn’t know where its data is, especially unstructured data, it's hard to drive expected business outcomes, Enciso said.

“Komprise Intelligent AI Ingest helps businesses start to lean into the AI revolution,” he said. “Our customers are asking us, ‘How do we scale AI responsibly?’ And what we like about Komprise right now is they solve a key challenge by ensuring only the right unstructured data makes it to the AI pipeline, which helps with accuracy and reducing risk.”

The new technology should help very large enterprise customers cut their data ingest injection time, which dramatically lowers the cost of AI processing, Enciso said.

There are a lot of companies that are ingesting a bunch of noise from an AI standpoint,” he said. “It's just not productive and it's expensive. The other part about AI is, a lot of people are bringing their own AI tools. How do we secure properly? How do we make sure that this doesn't become a security risk? And that's why we use a tool like Komprise.”

Komprise Intelligent AI Ingest is available to Komprise customers at no extra charge via the company’s “all you can eat” subscription model, Subramanian said.

Like all Komprise offerings, Komprise Intelligent AI Ingest is available only via the company’s channel partners, she said.

Enterprises with data-intensive environments are the main target of the technology, Subramanian said.

“Some of the customers using this are, like, the top 15 hospitals around the nation, and one of the top five banks,” she said. “Such customers don't want sensitive data getting leaked to AI. We filter not only all the duplicates or PII [personally identifiable information], but also company’s sensitive data. You can specify what's sensitive to your organization and we'll control that out.”