Computational Textual Analysis: Is it literature?

Is there a definitive definition for literature? I was thinking to myself about this the other day, and decided to find out. Before I set out on this experiment I'll make it clear that I think the answer is very subjective, and that although there are aspects found within most books that are considered true literature, these are not found within all of them. The initial list of books that I considered literature (with which to train my software) are widely spread in terms of theme, publication date and writing style, but the key point is they are chosen, therefore a concept of literature already exists in the person who chooses a list. The program I set out to write would be able to determine whether or not any piece of text was (or wasn't) literature by matching the similarity of various features of a piece of text, to this concept of true literature. It should also be noted that the program is language independent.

I'll start by explaining briefly some of the characteristics that I decided to use in order to measure the level of literature in a piece of text:

• Sentence length
• Average word length
• Unique words
• Vocabulary Score (unique words / total words)
• Vocabulary Deviation
• Character Deviation
• Word Entropy
• Character Entropy

Sentence length and word length were calculated by splitting the text into components with the splitting parameter for sentences being a full stop, and for words being a space. Note that words can therefore include punctuation; "this" classes as a different word to "this." when it is at the end of a sentence. This gives us some information about where words are used in sentences.

Unique words are simply words without repeats, for example in the text chunk "hi hi there there there", the unique words are "hi" and "there". These can be used to provide what I've called a vocabulary score. The vocabulary score essentially allows us to create a measure of the range of words used, that is (almost) independent of the length of the text. I say almost, because the range of vocabulary in the English language is limited, therefore as text chunks become longer (for example in a very long novel such as War and Peace) the vocabulary score will automatically decrease.

The vocabulary and character deviation involve counting the number of repeats of each word and character - then calculating the standard deviation of this data set. The former is a good measure of how comfortable an author is using a wide range of words, as opposed to using them occasionally.

Finally two very important measures of "literaryness" are the entropy of words and characters. During initial experiments I found that these tend to stay fairly constant for the "literature" used for training. To calculate the entropy of a set the following calculation is used:

Entropy = - SUM [ Ki log2 ( Ki ) ],

where Ki is the probability of a member of the set (in this case a word or character) occurring.

The next step of the experiment was to find a transform of the values of the above features that yielded very similar results for all of my training set.

The following set of books were used in the training set:

• Franz Kafka, Metamorphosis
• Oscar Wilde, The Picture of Dorian Gray
• Sir Arthur Conan Doyle, The Hound of the Baskervilles
• Joseph Conrad, Heart of Darkness
• Jules Verne, Around the World in 80 Days
• J.D. Salinger, Catcher in the Rye
• Anthony Burgess, A Clockwork Orange
• Bram Stoker, Dracula
• George Orwell, Down and Out in Paris and London

And a value, which we'll call L was determined to be the smallest deviating transform of the textual features where:

L = (Word entropy + Character entropy - Word length) / Vocabulary score

Other features were found to range too much between the works.

Given an average L value the distance away from the mean was used to give any test text a literature score. Unfortunately this blog did not receive the title of literature, but then nor did Harry Potter and the Philosophers Stone, or Twilight, or a Mills and Boon title, A Fool For Love, so it seems to work quite nicely. 

A book I would consider to be literature, not on the training set was Kidnapped by Robert Louis Stevenson, and it did indeed score highly on the literature front which means the system works!

Below is the application, just click to launch it and post a comment with any results you might find:

Currently the results are a bit spurious, mostly because thresholds are fiddly to get right, but with a bit of work and some other parameters I have been thinking about, this kind of textual analysis really might help us spot the next classics! The program currently works best with whole books, there are a load of free eBooks that are now in the public domain which you can test the software with!

2D Dynamic Shadow Casting (and Lighting)

After having a little think about how to do this effectively, the answer was actually rather simple (which ins't entirely in keeping with the rest of this blog).  My previous attempts involved raytracing the 2D scene. By stepping through every single point in the scene, and calculating the path to it from the light source a per pixel map of brightness can be generated. I decided however that this was far to CPU intensive for high complexity scenes so went down a simpler route. The basic algorithm was:

• Trace a path from the light source to each vertex in each shape,
• Continue these paths outwards from the shape, thus casting a shadow
• Generate polygons to be filled in the shadow colour with pairs of these paths
• Draw a light mask (just a circle with a transparency gradient over the floor)

Just click the first image to launch the experiment. I am quite pleased with the results!

The image below shows the intermediary path tracing step:

By changing the light source mask a directional torch like effect is simple to reproduce, again just click to launch the app:

Colour Palette Creator

I am sure that there are hundreds of tools like this on the web, but I thought it would be quite interesting to produce a colour palette based on an image. Imaging you are designing a website and you really like the colours used in a particular piece of art, then you've come to the right place. This software will extract the most common colours in an image, and also give you the average colour.

The method used to calculate these is very simple. To calculate the average colour, simply sum all the RGB components of each pixel and divide by the total number of pixels. I do this using bitwise operators where:

R = HEX >> 16 & 0xFF
G = HEX >> 8 & 0xFF
B = HEX & 0xFF

and HEX = (R<<16 | G<<8 | B), useful stuff!

To calculate the palette I create an accumulator array which counts the number of each colour in the image. Before using this I reduce the number of colours in the image using some simple math.

I hope you find it quite interesting to have a play around with, just click the image to launch it. You can load your own images from your local computer to play around with it.