GenAI and ACE: A Game-Changer for Business Data

Conveyer
4 min readDec 13, 2023

By Shriraj Kulkarni

In the AI era, there is an ever-growing demand for information that is both comprehensive and easily digestible. That’s why Conveyer created the AI Conversion Engine (ACE), a transformative solution that redefines our approach to data processing — and opens the door to powerful new ways of leveraging complementary technologies such as generative AI (GenAI).

What truly sets ACE apart is its ability to convert unstructured or semi-structured data into a meticulously structured format called “Topics” — specific, precisely defined categories or themes that cohesively organize the data into a structured whole.

This new approach represents a fundamental shift in how we engage with data. By precisely categorizing and united previously unstructured or loosely structured information with structured information, this technology makes information more readily accessible, more interactive, and more engaging. In addition to generating topics, ACE can support the creation of all GenAI artifacts — including but not limited to FAQs, sentiment analysis, classification, keywords, entities, and many more — further enhancing the depth and breadth of data exploration and comprehension.

How Does ACE Unlock Value for Businesses?

Consider a practical example, where a company uses ACE to transform unstructured or semi-structured data into structured Topics. With the aid of GenAI, it then goes a step further by generating sentiment analysis for the topic data. This process not only enhances data organization, but also extracts valuable insights from the structured information, shedding light on the sentiments embedded within.

Let’s say the organization has extensive volumes of customer data (emails, call recordings, support logs, etc.), and they want to use that unstructured data to identify sentiment patterns to ultimately enhance overall customer satisfaction. The problem is, this data is typically unstructured, sprawling, and seemingly all but impossible to leverage effectively. For most businesses, trying to derive insights from such data is not worth the investment due to GPU shortage and high processing costs.

That’s where Conveyer AI steps in. Conveyer’s ACE can transform this complex data into a structured, coherent format and deliver tailored insights meticulously aligned with the organization’s specific operational needs and objectives.

After transforming unstructured data into a structured topic format and generating artifacts using GenAI tailored to each topic, we can now delve into illustrating the sentiment analysis for the data.

The sentiment analysis provides insights into whether the content carries a positive, negative, or neutral tone. Each topic receives a sentiment analysis score, which is a numeric representation of the strength of that sentiment within the topic. A negative score corresponds to a negative sentiment, indicating a more pessimistic tone, while a positive score signifies a positive sentiment, reflecting a more optimistic tone. A score of zero is indicative of a neutral sentiment, suggesting an absence of strong positive or negative emotions. This detailed analysis aids in understanding the nuanced emotional context associated with each specific theme or subject.

Once these artifacts, including sentiment scores, are obtained, the next step in addressing business challenges involves visualizing the data. In our approach, we utilize vector data visualization, facilitated by the data structuring platform Atlas Nomic AI.

This process begins with textual embedding generation — a crucial step that transforms textual information into numerical vectors. These vectors are then utilized to visualize the vector representation of the text data. This comprehensive approach, which integrates both text embedding and sentiment scores, adds a layer of depth to our understanding, enabling us to extract valuable insights from the structured and sentiment-analyzed data. The vectorization process enhances our ability to unravel complex patterns and relationships within the data, fostering a more holistic understanding of the information at hand.

The following code demonstrates the process of visualizing the Pinecone vector database index in Atlas.

"""
Visualizing your pinecone vector database index in Atlas
"""
import pinecone
import numpy as np
from nomic import atlas
import nomic
pinecone.init(api_key='YOUR PINECONE API KEY', environment='us-east1-gcp')
nomic.login('YOUR NOMIC API KEY')

#create and insert embeddings into your pinecone index
pinecone.create_index("quickstart", dimension=128, metric="euclidean", pod_type="p1")

index = pinecone.Index("quickstart")

num_embeddings = 999
embeddings_for_pinecone = np.random.rand(num_embeddings, 128)
index.upsert([(str(i), embeddings_for_pinecone[i].tolist()) for i in range(num_embeddings)])

#now pull the embeddings out of pinecone by id
vectors = index.fetch(ids=[str(i) for i in range(num_embeddings)])

ids = []
embeddings = []
for id, vector in vectors['vectors'].items():
ids.append(id)
embeddings.append(vector['values'])

embeddings = np.array(embeddings)

atlas.map_embeddings(embeddings=embeddings, data=[{'id': id} for id in ids], id_field='id')
Image 1 showcases the data preview, including both the structured information and the associated sentiment analysis results. This allows for a comprehensive view, revealing the organized data structure along with the emotional tones conveyed in the content.
Image 2 portrays data points characterized by a neutral sentiment. This dataset is accessible for download as an Excel sheet, providing an opportunity for in-depth analysis to address various business challenges. Users not only have the flexibility to filter for positive sentiments, but they can also tailor their exploration to include negative or neutral sentiments. The downloadable data sheet allows for dynamic filtering based on the emotional tones conveyed in the content. Furthermore, the application of semantic search functionalities in vector visualization later on enhances the analytical capabilities, enabling users to conduct nuanced searches and gain deeper insights from the structured and sentiment-analyzed data. This comprehensive approach empowers users to navigate the dataset with precision, uncovering patterns and relationships aligned with specific business objectives.

You can access the Sentiment Analysis | Atlas Nomic AI here.

The showcased data and its visualizations underscore the potency of Conveyer’s state-of-the-art technology. By transforming vast, unstructured datasets into structured formats, Conveyer empowers businesses to derive actionable insights from complex information sources such as customer emails and call recordings. This capability is particularly valuable when aiming to uncover customer satisfaction insights from disparate, unstructured everyday data sources.

Conveyer offers a powerful solution for tackling intricate business challenges and harnessing the potential within data-driven environments. Get in touch with the Conveyer team today to learn more.

References:

--

--

Conveyer

Conveyer is an AI platform that unlocks and transforms unstructured data and simplifies curation for business to fuel actionable insights.