---
title: How to Build NLP Topic Models to Truly Understand What Customers Want
description: Step-by-Step Process for Identifying top Topics with Gensim
image: https://dataknowsall.com/hubfs/topic_viz.png
---

[Skip to content](https://dataknowsall.com/blog/topicmodels.html#main-content)

[![Lambda with Text - Wide](https://dataknowsall.com/hs-fs/hubfs/Lambda%20with%20Text%20-%20Wide.png?width=3068&height=301&name=Lambda%20with%20Text%20-%20Wide.png)](https://dataknowsall.com)

- [Home](https://dataknowsall.com)
- [Blog](https://dataknowsall.com/blog/)
- [Services](https://dataknowsall.com/services)
- [Technologies](https://dataknowsall.com/technologies)

Open main navigation

Close main navigation

- [Home](https://dataknowsall.com)
- [Blog](https://dataknowsall.com/blog/)
- [Services](https://dataknowsall.com/services)
- [Technologies](https://dataknowsall.com/technologies)
- [Contact us](https://dataknowsall.com/contact)

[Contact us](https://dataknowsall.com/contact)

[All posts](https://dataknowsall.com/blog/all)

 March 5, 2024

# How to Build NLP Topic Models to Truly Understand What Customers Want

![Picture of Brian Roepke](https://app.hubspot.com/settings/avatar/67adab425dfba5d6fa7e242d82fbe2c4) By  [Brian Roepke](https://dataknowsall.com/blog/author/brian-roepke)  ·   6 minute read

## What is a Topic Model?

[Topic Modeling](https://en.wikipedia.org/wiki/Topic_model) is one of my favorite NLP techniques. It is a technique for discovering topics in a large collection of documents. It allows you to take unstructured text and bring structure into topics, which are collections of related words. A subject matter expert can then interpret these to give them additional meaning.

We scraped the data we'll be using for this analysis from the official Tesla User Forums. It's a collection of about 55,000 entries consisting of Tesla owners asking questions about their cars and the community providing answers. User forums are a great place to mine for customer feedback and understand areas of interest. We'll approach this as a *Product Manager* looking for areas to investigate.

## Latent Dirichlet Allocation (LDA)

The specific methodology used in this analysis is [Latent Dirichlet Allocation (LDA)](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation), a probabilistic model for generating groups of words associated with *n*-topics. In LDA, the algorithm looks at documents as a mixture of topics. The topics are a mixture of words, and the algorithm calculates the probability of each word associated with a topic. LDA is potentially the most common method for building topic models in NLP.

This post will not get into the details of LDA, but we'll be using the [Gensim](https://radimrehurek.com/gensim/) library to perform the analysis. If you would like to learn more about LDA, check out the paper by Blei, Ng, & Jordan (2003).

Two terms you will want to understand when evaluating LDA models are:

- **Perplexity**: Lower the perplexity better the model.
- **Coherence**: Higher the topic coherence, the topic is more human interpretable.

## Text Cleaning and Prep

We must clean the text used for Topic Modeling before analysis. The objective is to find as munch commonality amongst words, and therefore you want to avoid issues like **Run**, **run**, and **running** being interpreted as a different word. I've covered [Text Cleaning](https://dataknowsall.com/blog/textcleaning.html) in this series already. We applied the methods from this article directly to the data. I opted to use *Lemmatization* over *Stemming* since words tend to be more interpretable when brought to their root or lemma versus being chopped off with stemming.

We'll start by importing the necessary libraries and loading our data.

```
import numpy as npimport pandas as pdfrom collections import defaultdictfrom gensim import corpora, modelsfrom gensim.models import Phrasesfrom gensim.models import CoherenceModelfrom gensim.models.ldamodel import LdaModelimport pyLDAvisimport pyLDAvis.gensim_models as gensimvisimport matplotlib.pyplot as pltfrom matplotlib.lines import Line2Dimport seaborn as sns%matplotlib inline
```

```
df = pd.read_pickle('tesla_clean.pkl')
```

We first need to turn our column from the Data Frame into a giant **list**, a simple task that we can do with one line.

```
documents = df['Discussion_Clean'].to_list()
```

After we have our list of documents, we can create a list of **stop words** and remove them and tokenize each document. **Tokenizing** is the process of breaking the string up into individual words.

**Note**: I recommend building your stop word list. You can iterate over your model and remove words that add little to no value to your topics. The words you might wish to remove probably are not common stop words but are not helping you interpret the results. Towards the end of my stop list, you can see words like `troll` and `fish` which are not helpful.

`stoplist = ['the', 'a', 'an', 'of', 'and', 'or', 'in', 'for', 'to', 'at', 'by', 'from', 'with', 'on', 'as', 'but', 'is', 'are', 'was', 'were', 'be', 'been', 'am', 'i', 'me', 'my', 'we', 'our', 'you', 'fish', 'troll', 'ha', 'bye', 'ok', 'okay', 'andy', 'idiot'] # remove common words and tokenize texts = [[word for word in document.split() if word not in stoplist]          for document in documents]`

Next, we can remove words that **infrequently occur**; in this case, only words that only appear **once** in the corpus. We can adjust this number to your needs.

```
# remove words that appear only oncefrequency = defaultdict(int)for text in texts:    for token in text:        frequency[token] += 1texts = [[token for token in text if frequency[token] > 1]         for text in texts]
```

Another treatment you can add is **n-grams** to your documents. You can experiment with this on your own and see how it affects your results.

```
# Add bigrams to docs (only ones that appear 20 times or more).bigram = Phrases(texts, min_count=20)for idx in range(len(texts)):    for token in bigram[texts[idx]]:        if '_' in token:            # Token is a bigram, add to document.            texts[idx].append(token)
```

Next we create our **dictionary** and **corpus**. The corpus is a list of documents, and the dictionary lists unique words. Note the powerful function to remove words that are **extremes**. You can hone in on more meaningful topics depending on your documents by adjusting these values.

```
# Create the dictionarydictionary = corpora.Dictionary(texts)# Filter out words that occur less than X documents, # or more than X% of the documents.dictionary.filter_extremes(no_below=50, no_above=0.5)# Create the corpus.  This is a Term Frequency # or Bag of Words representation.corpus = [dictionary.doc2bow(text) for text in texts]print(f'Number of unique tokens: {len(dictionary)}')print(f'Number of documents: {len(corpus)}')
```

```
Number of unique tokens: 2462Number of documents: 54335
```

## Training the Topic Model

Next, we need to train the topic model. Training in this context is somewhat akin to hyperparameter tuning in a Machine Learning model. We will iterate through different parameters until the model converges across all (or almost all) of our documents. We need to turn logging into `DEBUG` mode to do this, and we can do this via the following code.

```
import logginglogging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.DEBUG)
```

Next, we can set parameters and attempt multiple passes until we get the results we want.

```
NUM_TOPICS = 10chunksize = 2000passes = 6iterations = 100eval_every = 1temp = dictionary[0]id2word = dictionary.id2tokenmodel = LdaModel(    corpus=corpus,    id2word=id2word,    chunksize=chunksize,    alpha='auto',    eta='auto',    iterations=iterations,    num_topics=NUM_TOPICS,    passes=passes,    eval_every=eval_every)
```

After running this, open up the log (if you're using VS Code, you should be able to open it in a new tab). Next, search for **converged** and look for a line like the following. Adjust the `iterations` and `passes` values until you see all the documents converged. After some playing with my model, I ended up with the following results.

```
2022-01-01 : DEBUG : 1998/2000 documents converged within 100 iterations2022-01-01 : DEBUG : 335/335 documents converged within 100 iterations
```

With these settings, you can now choose the best number of topics. The Gensim documentation [2](http://localhost:8000/topicmodels.html#fn:GEN) describes this process as well.

## Choosing the Best Number of Topics

To choose the best number of topics, we can calculate the **coherence value** for different topic-number settings and determine which has the **greatest value**. The below function simplifies this and can be a bit of a time-consuming process. Try to iterate through and get the best results. I wouldn't recommend running `100` processes stepping by `1` to begin.

```
def compute_coherence_values(dictionary, corpus, texts,                              cohere, limit, start=2, step=2):    coherence_values = []    for num_topics in range(start, limit, step):        model = LdaModel(corpus=corpus,                          id2word=dictionary,                          num_topics=num_topics,                         chunksize=chunksize,                         alpha='auto',                         eta='auto',                         iterations=iterations,                         passes=passes,                         eval_every=eval_every,                         random_state=42,)        coherencemodel = CoherenceModel(model=model,                                         texts=texts,                                         dictionary=dictionary,                                         coherence=cohere)        coherence_values.append(coherencemodel.get_coherence())    return coherence_values
```

Next, we'll create some parameters for stepping through the range of topics we want to test. Some corpus might have hundreds of topics; smaller ones will have few. You can play with these values to see how yours change. I started with stepping to `100` by `5`originally to narrow down the window and then stepped by `2` when finishing. We will also use these variables in the plot below, which is why they're separated.

```
limit=50start=2step=2
```

Let's run the above function calculating the **coherence value** for each number of topics. The result of the function returns a list of values.

```
coherence_values = compute_coherence_values(dictionary=dictionary,                                             corpus=corpus,                                             texts=texts,                                             cohere='c_v',                                             start=start,                                             limit=limit,                                             step=step)
```

We can now easily plot these values and calculate the **Max** value. I've plotted a vertical line to help visualize it easier.

```
plt.figure(figsize=(8,5))# Create a custom x-axisx = range(start, limit, step)# Build the line plotax = sns.lineplot(x=x, y=coherence_values, color='#238C8C')# Set titles and labelsplt.title("Best Number of Topics for LDA Model")plt.xlabel("Num Topics")plt.ylabel("Coherence score")plt.xlim(start, limit)plt.xticks(range(2, limit, step))# Add a vertical line to show the optimum number of topicsplt.axvline(x[np.argmax(coherence_values)],             color='#F26457', linestyle='--')# Draw a custom legendlegend_elements = [Line2D([0], [0], color='#238C8C',                           ls='-', label='Coherence Value (c_v)'),                   Line2D([0], [1], color='#F26457',                           ls='--', label='Optimal Number of Topics')]ax.legend(handles=legend_elements, loc='upper right')
```

We can see here that the Coherence Value peeks at `8`. Remember above we said:

> Higher the topic coherence, the topic is more human interpretable.

While this is great for guidance, you must still interpret the results and determine if they make sense. We'll utilize `8` topics as our final and visualize the results.

![topic_coherence](https://dataknowsall.com/hs-fs/hubfs/topic_coherence.png?width=2400&height=1500&name=topic_coherence.png)

## Final Run and Visualiztion of the Results

Finally, we can run our trained and tuned model and visualize our results!

```
temp = dictionary[0]id2word = dictionary.id2tokenmodel = LdaModel(    corpus=corpus,    id2word=id2word,    chunksize=2000,    alpha='auto',    eta='auto',    iterations=100,    num_topics=8,    passes=8,    eval_every=None)
```

Next, let's visualize the results. We'll use the `pyLDAvis` package to visualize the results.

```
pyLDAvis.enable_notebook()# feed the LDA model into the pyLDAvis instancelda_viz = gensimvis.prepare(model, corpus, dictionary, sort_topics=True)pyLDAvis.display(lda_viz)
```

![topic_viz](https://dataknowsall.com/hs-fs/hubfs/topic_viz.png?width=2366&height=1420&name=topic_viz.png)

## Conclusion

Topic Models are an incredibly powerful tool for interpreting unstructured text. Topic Modeling can be used on various sources such as social media, product reviews, customer support tickets, and more. As a former Product Manager, I've used topic modeling to inform roadmap decisions on where to invest resources and prioritize. Best of luck on your journey utilizing them!

 

 

 

 

Share: [facebook-f icon](http://www.facebook.com/share.php?u=https://dataknowsall.com/blog/topicmodels.html) [linkedin-in icon](http://www.linkedin.com/shareArticle?mini=true&url=https://dataknowsall.com/blog/topicmodels.html) [twitter icon](https://twitter.com/intent/tweet?url=https://dataknowsall.com/blog/topicmodels.html) [pinterest-p icon](http://pinterest.com/pin/create/link/?url=https://dataknowsall.com/blog/topicmodels.html) [envelope icon](mailto:?body=https://dataknowsall.com/blog/topicmodels.html)

## Related posts

[![Clustering Elbow Method](https://dataknowsall.com/hubfs/textclustering_elbow.png)](https://dataknowsall.com/blog/textclustering.html)

[NLP](https://dataknowsall.com/blog/tag/nlp)

### [Utilize Machine Learning to Automatically Detect Patterns in Text](https://dataknowsall.com/blog/textclustering.html)

[![](https://dataknowsall.com/hubfs/AI-Generated%20Media/Images/Two%20people%20sitting%20at%20laptops%20using%20a%20browser%20based%20application%20in%20different%20offices.jpeg)](https://dataknowsall.com/blog/streamlitauthmultipage.html)

[Data Engineering](https://dataknowsall.com/blog/tag/data-engineering)

### [Implementing Streamlit-Authenticator Across Multi-Page Apps](https://dataknowsall.com/blog/streamlitauthmultipage.html)

[![Text Cleaning](https://dataknowsall.com/hubfs/text_clean.jpg)](https://dataknowsall.com/blog/textcleaning.html)

[NLP](https://dataknowsall.com/blog/tag/nlp)

### [How to Clean Text Like a Boss for NLP in Python](https://dataknowsall.com/blog/textcleaning.html)

[![Lambda Google Workspace](https://dataknowsall.com/hs-fs/hubfs/Lambda%20Google%20Workspace.png?width=320&height=132&name=Lambda%20Google%20Workspace.png "Lambda Google Workspace")](https://www.dataknowsall.com)

Specialists in Data Engineering, Machine Learning, and Analytics Consultation

[linkedin-in icon](https://www.linkedin.com/in/broepke/) [twitter icon](https://twitter.com/broepke) [Follow us on Facebook](https://github.com/broepke)

**Pages**

- [Home](https://dataknowsall.com)
- [Blog](https://dataknowsall.com/blog/)
- [Services](https://dataknowsall.com/services)
- [Technologies](https://dataknowsall.com/technologies)

### Get Updates In Your Inbox

Copyright © 2026, Data Knows All

```json
{
  "@context" : "https://schema.org",
  "@type" : "BlogPosting",
  "author" : {
    "@type" : "Person",
    "name" : "Brian Roepke",
    "url" : "https://dataknowsall.com/blog/author/brian-roepke"
  },
  "dateModified" : "2024-03-05T12:53:20.024Z",
  "datePublished" : "2022-01-09T16:06:00.000Z",
  "headline" : "How to Build NLP Topic Models to Truly Understand What Customers Want",
  "image" : [ "https://dataknowsall.com/hubfs/topic_viz.png" ],
  "mainEntityOfPage" : {
    "@id" : "https://dataknowsall.com/blog/topicmodels.html",
    "@type" : "WebPage"
  },
  "publisher" : {
    "@type" : "Organization",
    "logo" : {
      "@type" : "ImageObject",
      "url" : "https://dataknowsall.com/hubfs/Lambda%20Google%20Workspace.png"
    },
    "name" : "Data Knows All, LLC"
  }
}
```

```json
{
  "@context" : "http://schema.org",
  "@type" : "Article",
  "author" : {
    "@type" : "Person",
    "name" : [ "Brian Roepke" ],
    "url" : "https://dataknowsall.com/blog/author/brian-roepke"
  },
  "datePublished" : "2022-01-09T16:06:00+0000",
  "description" : "Step-by-Step Process for Identifying top Topics with Gensim",
  "headline" : "How to Build NLP Topic Models to Truly Understand What Customers Want",
  "image" : "https://44115987.fs1.hubspotusercontent-na1.net/hubfs/44115987/topic_viz.png",
  "publisher" : {
    "@type" : "Organization",
    "logo" : {
      "@type" : "ImageObject",
      "url" : "https://44115987.fs1.hubspotusercontent-na1.net/hubfs/44115987/Lambda%20Google%20Workspace.png"
    },
    "name" : "Data Knows All"
  },
  "url" : "https://dataknowsall.com/blog/topicmodels.html"
}
```