We all love topic models, and one of the things we like about them is that we get these lovely, interesting, semantically-coherent topics. But one thing limiting the use of these topics is that we have no easy appellations for them. Wouldn’t it be great if we could say that this document is 20% “astronomy” rather than 20% the topic whose top words are “star planet quasar galaxy”?
To this end, I ran another Mechanical Turk experiment. I presented the top 6 words from a topic (in a shuffled order to different turkers for good measure). The turkers were asked whether or not the set of words makes sense together, and if they did to come up with a concise subject heading for them.
The topics I used were some of the topics learned by Nubbi on Wikipedia. I’ve put the responses here. The “Good topic %” is the proportion of people who judged that the set of words makes sense together. For some of the topics nearly everyone agrees that they’re good topics; for others almost everyone agrees that they’re terrible. More worrisome is that there are a few in the middle.
As for the labels, there was considerable variation. Some of it is accounted for by different levels of specificity (“Monarchy” vs. “European history” vs. “English history” vs. “Mary and Bothwell”). But while the labels are in the same milieu, there doesn’t seem to be any universal agreement on what makes a good topic label.
So the take away is:
- Not as many topics are “good” as I would’ve thought.
- Coming up with good labels for topics is no easy task.
Let’s chew on that next time we stick a corpus through the LDA wringer.
Please Scoop Me! 