Machine and Human Learning of Word Meanings

How do we ever come to know the meanings of words? Consider the following (likely apocryphal) story:

Walking along one day on the newly-discovered coast of Australia, Captain Cook saw an extraordinary animal leaping through the bush. "What's that?" he asked one of the aborigines accompanying him.

"Uh - gangurru." he replied - or something like that. Captain Cook duly noted down the name of the peculiar beast as 'Kangaroo'. Some time later, Cook had the opportunity to compare notes with Captain King, and mentioned the kangaroo.

"No, no, Cook", said King, "the word for that animal is 'meenuah' - I've checked it carefully.

"So what does 'kangaroo' mean?"

"Well, I think," said King "it probably means something like 'I don't know'..."

This story may have inspired Quine in proposing the "gavagai problem," which can be interpreted as suggesting that word learning is a nearly impossible task because any given word has in principle an infinite number of referents. To make this more clear, consider that gangurru (or meenuah, or "gavagai") could actually mean "marsupial," "good to eat," or "kangaroo jumping through a harvested field prior to 11:45 am." So, how do we recover word meaning from any real-world linguistic experience, when even straightforward and direct instruction of word meaning is actually so ambiguous?

One might assume that powerful constraints exist on the kinds of conclusions we draw from linguistic experience. Yet a fascinating machine learning technique, known as Latent Semantic Analysis, is capable of acquiring word meanings in ways that eerily resemble human learning - all without ever having undergone direct instruction, and with no word learning biases built-in.

As presented in this paper (from which this post is conceptually derived), Landauer & Dumais describe how Latent Semantic Analysis (LSA) can acquire word meaning simply by reading encyclopedia text. Essentially, the LSA algorithms derive a rating of similarity between every word and every other word by cataloguing the "number of times that a particular word type, say 'model,' appears in a particular paragraph, say this one." These values undergo a log transform, and then division by the entropy of that word with respect to that paragraph (entropy is a measure of how representative a word is of the paragraph in which it is found, with high values being less representative; basically, dividing by this term allows LSA to scale-down the importance of "contextually ill-defined" words). Ultimately, this processing results in an enormous matrix of similarity relationships.

That's "where the miracle occurs." Landauer & Dumais describe the use of "singular value decomposition" (SVD) to compress these similarity relationships into a more manageable data structure - one that represents the most "core" set of relationships that define when and how a certain word will be used. In slightly more technical language, SVD reduces the dimensionality of a data set by identifying the principle components or eigenvectors that compose it (the technique is similar to factor analysis, principle component analysis, and multidimensional scaling).

To use the example in the paper, Landauer & Dumais were able to compress semantic relationship data from over 30,000 encyclopedia articles into around 300 dimensions. In other words, the usage of each and every term out of the 4,000,000 words encountered in the encyclopedia is optimally describable if characterized along each of roughly 300 different continuums. One might claim that LSA has learned the set of "core semantic characteristics" that make up the meanings of English words. (Not only that, but this is just one of many possible dimensionality-reduction techniques, as the authors note!)

And the proof is in the pudding:

• after training, LSA performed at 64.4% correct on a multiple choice test of synonymity taken from TOEFL (in contrast, humans score around 64.5% on average on this test, which is frequently used as a college entrance examination of English proficiency in non-native speakers. By this metric, LSA would be admitted to many major universities!)
• calculations of the rate of word learning by 7th graders suggests that they acquire .15 words per 70-word text sample; analogous calculations of LSA's rate of acquisition show that LSA acquires .1500 words per text sample read
• the comprehension by college students of several versions of a text sample about heart function is precisely replicated by LSA, when comprehension is measured as the degree of semantic overlap between subsequent sentences;
• Humans initially show facilitated processing of all meanings of a previously-presented word, but after 300 ms show priming only of context-appropriate meanings; LSA shows similar effects insofar as similarity is higher between a homograph and two words related to different meanings of the homograph than between a homograph and unrelated words, and in that LSA considers words related to the context-appropriate definition of a homograph as more related than words related to the context-inappropriate definition of the homograph;
• Human reaction times in judgments of numerical magnitude suggest that the single digit numerals are represented along a "logarithmic mental number line;" LSA was able to replicate this effect in its ratings of similarity among the single digit numerals, which also conform to a logarithmic function

The authors note that novel variants on the SVD procedure may be capable of fitting even more data, and indicate a particular need for elaborations that specify an adaptive procedure for identifying a data set's optimal dimensionality.

In conclusion, LSA well-characterizes many aspects of word learning. How SVD-like computations may be situated in the brain, and the necessity of some of the simplifying assumptions of this model (such as perfect memory, and an empirical determination of optimal dimensionality) are as yet unanswered.

Nonetheless, latent semantic analysis seems like a promising and powerful approach for understanding human semantic learning at a cognitive level. Perhaps most importantly, it shows that Quine's "gavagai problem" might not be so intractable after all - a word's meaning can be understood as a function both of the contexts in which that word appears, the contexts in which it doesn't appear, and likewise for every other word to which it is related. This recursive relationship of word to context provides a nearly infinite amount of linguistic data from which word meaning might be derived.