Machine Learning Fun - Predicting Authorship from 40 Year-old Talks

I am continuing my fun with applying statistical and machine learning methods to General Conference talks. This post will explore if you can predict the author/speaker of a general conference talk today, but by using talks given over 25-40 years ago. Since Thomas S. Monson, Boyd K. Packer and L. Tom Perry are the most senior apostles and general authorities, I used them as case studies because they have the most general conference talks available. The oldest general conference talks available go back until 1971 on LDS.org.

The idea is this: Can the occurrence of certain words from talks given 25-40 years ago by these speakers be used to accurately predict if they are the author of a recently given talk -- but assuming we only had the text of the recent talk and knew nothing about who actually gave it? Let's find out...

Representing Talks as 0s and 1s

We first need to determine if we can statistically distinguish authorship between the three speakers from the talks given 25-40 years ago. I will be attempting to distinguish authorship based on the presence or absence of certain words in a talk.

I had the computer create four different dictionaries based on the most common words used among all three speakers. Each dictionary was a different size of unique words as follows: 128 words, 267 words, 542 words, and 1042 words. 

The dictionaries were derived from almost 120 total talks, with about 40 talks given from each speaker. The talks had over 8,000 unique words after combining similar words together to avoid double counting. For example, jump and jumping would be considered the same word; they would be counted as being one unique word 'jump'.

Based on those dictionaries, each talk was then turned into a spreadsheet-like document. Each row represents a certain talk, and each column represents a particular word. The cells of the spreadsheet contain a 0 if a certain word is absent, or a 1 if a word is present. This means we are numerically representing each talk as a bunch of 0s and 1s based on word occurrences. This method is like giving a fingerprint to each talk. In sum, we're hoping to see if there are patterns in the talks' "fingerprints" that correspond to certain speakers.

Visualizing Similarity/Differences Among Talks and Speakers

After creating the giant spreadsheet full of 0s and 1s, the talks were then converted mathematically into coordinates on a two-dimensional plane. The 120 talks were also previously labeled on the spreadsheet so we would know which speaker gave each talk.

Plotting the talks visually allows us to better understand how similar or different they are from each other. If a certain speaker's talks are all being grouped together closely, but also being far away from other speakers' talks, this will confirm we have statistically distinguished authorship based on word occurences! Thereafter, we can attempt to predict authorship of new talks without knowing who the author is in advance! This means we are leveraging experience gleaned from the 120 older talks to predict authorship of new talks given more recently!

Plots of Talks and Speakers

Below is how the 120 older talks look like from the 128 word dictionary. Each circle, triangle or square is an individual talk corresponding to a certain speaker.

You can see in the chart there is a lot of overlap among speakers. This means the 128 word dictionary isn't very effective to distinguish authorship since all the talks look the same (statistically speaking).

Below is how the talks look like from the 267 word dictionary:

This 267 word dictionary looks quite a bit better -- there is more separation among speakers but there is quite a bit of overlap still.

Here is how the talks look like from the 542 word dictionary:

The 542 word dictionary is looking pretty good! There is quite a bit of separation between speakers and only a little bit of overlap between Monson and Perry. This goes to show that these three speakers have particular word choices that give them a somewhat unique identity! It may not be possible to totally separate all the speakers statistically speaking, but it looks like combinations of at least 542 words can be a powerful way to identify authorship. However, let's see if we can improve this any further.

Here is how the talks look like from the 1042 word dictionary:

The 1042 word dictionary looks fantastic! There is a lot of separation among speakers and just barely a bit of overlap. Apparently, the presence or absence of the 1042 selected words appears to distinguish authorship very well! Now that we've determined that authorship among the three speakers is distinguishable, let's get on to the predictions.

Predicting the Author/Speaker of a New Talk

I collected 16 recent talks given between 2012-2014 by Monson (5 talks), Packer (5 talks) and Perry (6 talks). The computer was never told in advance who the true author was of each talk. To predict authorship, the texts of the new talks were also turned into spreadsheets of 0s and 1s using the 1042 word dictionary developed earlier (the 120 talks given between 1970-1990). Thereafter, those 0s and 1s of the new talks were converted into coordinates for a two-dimensional plane.

Now, at this point, prediction is easy. The speaker/author of the talk will be estimated based on where a talk lands on the two-dimensional plane and where its position is in relation to the 120 talks earlier. We saw earlier there were three distinct groups of talks, with each group corresponding strongly to a particular speaker. So, if a new talk -- whose speaker is unknown in advance to the computer -- lands firmly within the territory of the talks from speakers we knew from earlier, we will then predict that this was the author of the new talk!

Results and Plots of Selected Predictions

After following this method, the author/speaker for 13 of 16 talks was correctly predicted! That is an 81% success rate! This isn't half bad considering the simplicity of the methods employed. Overall, it appears the text in talks given from 25-40 years ago have predictive power to determine the author of a talk given today! These three speakers -- Thomas S. Monson, Boyd K. Packer, and L. Tom Perry -- appear to have been mostly consistent with their word choices over the years to allow this to happen!

Here are the plots of the three talks that had incorrectly predicted authors/speakers:

1) Love -- The Essence of the Gospel - April 2014

Predicted Speaker: L. Tom Perry 

Actual Speaker: Thomas S. Monson

This makes sense why the prediction went wrong. The blue diamond is right on the border between Monson and Perry.

2) Obedience to Law Is Liberty - April 2013

Predicted Speaker: Boyd K. Packer

Actual Speaker: L. Tom Perry

It looks like this particular talk by Perry had Packer-esque word occurrences. This blue diamond is quite a bit inside Packer territory.

3) The Doctrines and Principles Contained in the Articles of Faith - October 2013

Predicted Speaker: Boyd K. Packer

Actual Speaker: L. Tom Perry

Here is another talk that was on the border -- this time between Packer and Perry. Talks landing in the border zone introduce more ambiguity, so it makes sense the computer got this one wrong.

Here are plots of some talks that correctly predicted speakers:

1) Finding Lasting Peace and Building Eternal Families - October 2014

Predicted Speaker: L. Tom Perry

Actual Speaker: L. Tom Perry

This talk is firmly in Perry territory.

2) The Reason for Our Hope - October 2014

Predicted Speaker: Boyd K. Packer 

Actual Speaker: Boyd K. Packer 

This talk is firmly in Packer territory.

3) True Shepherds - October 2013

Predicted Speaker: Thomas S. Monson 
Actual Speaker: Thomas S. Monson 

This talk was in the ambiguous border zone between Monson and Perry, but the computer happened to get the author/speaker right. This one looks like a tough call visually speaking. 

In closing, this little project was a lot of fun. I should note, however, that it is much more important to actually read the general conference talks and apply their messages to our everyday lives. It is a great blessing to have prophets, seers, and revelators among us to give us the guidance we need today!

***

Previous posts related to statistics/text analytics of general conference talks:

Text Analytics Fun - Visualizing Sentiment Between General Conference Talks

http://thesmallplatesoftom.blogspot.com/2015/03/text-analytics-fun-visualizing.html

Statistics Fun - Visualizing Similarity Between General Conference Talks

http://thesmallplatesoftom.blogspot.com/2015/03/statistics-fun-visualizing-similarity.html

***

Summary of methodology:

• R was used for all analysis/visualization
• Each conference talk was copy and pasted into its own text file to make a corpus of documents.
• The corpus was prepared by removing numbers, punctuation, sparse words and applying stemming.

·         A term document matrix was created using binary representations for word occurrences or absences.

·         Distance was calculated from term document matrix using the asymmetric binary/Jaccard method.

·         Classical multidimensional scaling was applied to the Jaccard distances to project talks onto a two dimensional space ("X and Y coordinates").

• Author/speaker prediction was done using the IBk algorithm (nearest-neighbor) from the RWeka package. Three nearest neighbors were used in voting for classification.