Text Comparison with TF-IDF

One very common scenario I've encountered as a business systems developer is the need to compare qualitative data from two or more sources to determine if there are similarities or matches.

For example, there is a collection of Contact records in our CRM that we want to match with a collection from an email list or marketing campaign. Contact names, emails, phone numbers, and other information may not be consistent and could have alternate spellings.

Or let's say users have been entering notes in a CRM about customers and the specific products they are interested in adopting. There is also a product request portal that is totally separate from the CRM, but needs to be aligned with the customer data. Is there a way to match existing descriptions between the portal and the CRM to make this connection?

While you could certainly do this type of matching manually, who wants to be the one do this level of tedious comparison? Fortunately, there are a lot of automated Natural Language Processing (NLP) methods we could employ, and I've happened across one such method recently I wanted to share because I found it to be particularly useful.

Term Frequency–Inverse Document Frequency (TF-IDF), is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus. I'm not an expert regarding the mathematics supporting this strategy, and there is a lot of information readily available if you're interested in learning more or comparing various NLP strategies.

Instead, I want to focus on one specific application TF-IDF using scikit-learn's TfidfVectorizer to show how powerful it can be at comparing and relating qualitative data. While the applications of TF-IDF I came across demonstrated comparing a word or set of words to a collection of documents, I wanted to see what would happen if we compared documents to documents.

To illustrate this application, I started by asking a generative AI to come up with some writing prompt examples and then provide a short response to the prompt. The following table shows the original text responses:

Then I took those responses and asked the AI to create similar responses to the same topics, so I could get some text samples about the same general idea that were composed somewhat differently. I then rearranged the prompt-to-id mapping of the secondary responses:

So now I have unique blobs of text that are about the same set of core topics across two different data sets. Here are some specific examples from the two sets:

Childhood Memories

Original Text: When I was a child, my favorite memories were spent playing in the backyard with my friends. We would spend hours climbing trees, building forts, and having imaginary adventures. I remember the feeling of freedom and pure joy as we would explore the world around us. Those carefree days hold a special place in my heart and serve as a reminder of the importance of embracing our inner child.

Secondary Text: In my childhood, some of my fondest memories were of playing imaginative games with my siblings indoors during rainy days. We would create elaborate make-believe worlds, transforming our living room into magical kingdoms or outer space adventures. The joy and laughter we shared brought us closer together, and those cherished moments continue to put a smile on my face.

The Benefits of Mindfulness

Original Text: Mindfulness is the practice of being fully present in the moment and non-judgmentally aware of one's thoughts, emotions, and sensations. It has been shown to reduce stress, improve focus and attention, and enhance overall well-being. Mindfulness can help to cultivate a sense of calm and clarity, allowing individuals to better manage their emotions and navigate life's challenges. It promotes self-awareness and encourages a non-reactive mindset, enabling individuals to respond to situations with intention and kindness. Mindfulness practices, such as meditation and deep breathing, can be incorporated into daily routines to foster a greater sense of peace and balance. By practicing mindfulness, we can cultivate a greater sense of gratitude for the present moment and develop a deeper connection with ourselves and others.

Secondary Text: Mindfulness practices can indeed promote well-being, but it's essential to recognize that different individuals may find solace in diverse forms of meditation or mindfulness techniques. Embracing a range of practices allows for a more inclusive and accessible approach to well-being.

Now for the processing and comparison. First, lets import pandas, numpy, and TfidfVectorizer from scikit-learn. Then create two DataFrames, one from the original responses and another for the secondary.

import pandas as pd
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer


original = pd.read_csv('original_responses.csv')
secondary = pd.read_csv('secondary_responses.csv')

Next, while iterating through the rows of the original responses, we create the corpus of text samples to compare. This is just a list of the current text we're looking at and the entire collection of the secondary text responses.

for row in original.itertuples():
    id = row[1]
    prompt = row[2]
    text = row[3]

    corpus = [text]
    corpus.extend(secondary['text'].tolist())

Using the vectorizer we can now run the TF-IDF comparison, and transform the result matrices into a numpy array. From there we do a little cleanup to eliminate the comparison result of the text to itself, which would be a perfect match.

    vectorizer = TfidfVectorizer(min_df=1, stop_words='english')
    tfidf = vectorizer.fit_transform(corpus)
    pairwise_similarity = tfidf * tfidf.T

    arr = pairwise_similarity.toarray()
    np.fill_diagonal(arr, np.nan)

    input_idx = corpus.index(text)
    result_idx = np.nanargmax(arr[input_idx])

Finally, we can extract the matched text and trace it using its index in the secondary DataFrame.

    match_text = corpus[result_idx]

    match_idx = secondary['text'].str.find(match_text).to_list().index(0)
    match_id = secondary.iloc[match_idx]['id']
    match_prompt = secondary.iloc[match_idx]['prompt']

Using TF-IDF I was able to match the secondary text to the original text 100% of the time! I dropped the original prompt and the prompt from the matched text into a DataFrame with the output:

I'm sure there are some limitations, but in the cases where I have used this approach I've seen a pretty solid success rate. I look forward to exploring more and discovering new methods of text comparison that can deliver solid results without a ton of manual work.