Voice Recognition in Flash

This evening I thought I'd take a crack at looking at voice recognition in flash. The idea for this came after the realisation that when using my phone I have almost entirely given up on using the keypad. Instead I simply utter a few words (When passers by are out of ear shot) and a voice appears at the other end. Wonderful! I haven't read up on voice recognition at all so in part I am using the knowledge I've gained by looking at image recognition and tracking software. Being a big Aphex Twin fan and having written a spectrograph to visualize this song in flash before I had at least a bit of a grounding. The idea would be to record a small sample of voice, using some microphone thresholding, draw a spectograph of the sample and then compare the spectograph to a database of possible patterns and determine which command or whatever is most similar. I'm pleased to say that I have completed the first half of this today and you can click to see a demo below (The top image will launch it):

The images are of a whistling sequence, a glass being rung, and me saying hello. Interestingly you can see the harmonics of the glass being rung (a few set frequencies). My voice is a bit harder to translate but using some analysis I'm sure it will be possible.

Anyway as usual the source code can be downloaded here and let me know if you come up with anything using it. How about auto-tuning?!

The plan from here is to save single voice clips as images, and train a neural network to find patterns in both the volume and the spectograph for a set number of commands. I will report my findings shortly!

Perceptron in Actionscript

I've always thought neural networks and the idea of computational learning was pretty cool but until now I'd never actually tried to program one. I decided to start out with something simple so I went for the single layer perceptron with the hope of building it up to something a bit smarter.

For those of you who don't know much about neural networks Wikipedia has done a pretty decent job in teaching me everything I know. All the project files are available here and sample source code is provided below. It is a simple solution to the NAND problem using two inputs.

The perceptron class essentially has two functions, train and evaluate. The train function is called 100 times on the training set which adjusts the respective weights on each input. When evaluate is called on a particular inputs an output is produced. The output trace from the below program looks like this:
Initiating new perceptron:

Training 100 times using training set...

Trained

Evaluating NAND[0,0] : 1

Evaluating NAND[1,0] : 1

Evaluating NAND[0,1] : 1

Evaluating NAND[1,1] : 0

This is a fast solution to these kinds of problems. I plan on implementing optimisations such as exiting training when the weights change by less than some threshold (i.e the solution limit has been reached), but the end goal is to produce a full library with many kinds of next step. I think the next of these will be a multilayer perceptron which initial research suggests is much more complicated! Enjoy the classes and let me know if you do anything interesting with them.

package

{

import flash.display.Sprite;

import org.experimentalized.networks.SingleLayerPerceptron;

import org.experimentalized.networks.objects.Trainer;

import org.experimentalized.networks.objects.TrainerSet;

public class Main extends Sprite

{

public function Main()

{

//NAND example

trace("Initiating new perceptron:");

var perceptron:SingleLayerPerceptron = new SingleLayerPerceptron(0.1);

var training_set:TrainerSet = new TrainerSet([ new Trainer([0,0],1),

new Trainer([1,0],1),

new Trainer([0,1],1),

new Trainer([1,1],0)

]);

trace("Training 100 times using training set...");

perceptron.TrainSet(training_set,100);

trace("Trained");

//Test some evaluations

trace("Evaluating NAND[0,0] :",perceptron.Evaluate([1,0]));

trace("Evaluating NAND[1,0] :",perceptron.Evaluate([1,0]));

trace("Evaluating NAND[0,1] :",perceptron.Evaluate([0,1]));

trace("Evaluating NAND[1,1] :",perceptron.Evaluate([1,1]));

}

}

}

Colour Wheel - HSV Colour Space

I've just been playing around with colour wheels and colour representation in flash. I had a think about whether it was possible to represent every single colour in the RGB spectrum on a single 2D image. Closer inspection reveals that it isn't possible because RGB is made up of three components - red, green and blue - and therefore needs three dimensions. This is why if you look up different colour representations on Wikipedia you might see them represented by cones, or cubes. I decided to work in the HSV colour space which is  formed of Hue (rotation), Saturation (The amount of colour compared to black/white) and Value (Essentially the brightness). The resulting colour wheel can be seen below:

Just click on the image to launch the app. By rolling over the wheel the colour in the box will show the selected colour. Use the up/down arrow keys to alter the "V" value of the wheel. This should allow you to view a complete (discrete) representation of every colour in the RGB spectrum. 

If you're interested in source code just click here to download the FlashBuilder project and AS files. Enjoy!

There's nothing boring about Tax!

Well the title might be a bit misleading, but I'll let you be the judge of that. A close friend and I were in the pub last week discussing the benefits of a tax system which, instead of being tiered, would be characterised by a continuous curve. My friend being an Economist, and me a Physicist, and both of us slightly tipsy thought that we might be able to change the world with our new theory - so I went home that night and wrote a small program using FlashMath to try and quantify some of our ideas. The program essentially integrated over a Cauchy-Schwartz curve and compared collected tax with that of the standard tiered system. I had planned on expanding on this by optimising the curve using measures such as utility (an economic measure of happiness of a population) but for now here is the source code:

package org.experimentalized
{
 package
{
 import flash.display.MovieClip;
 import com.experimentalized.math.quadrature.SimpsonsQuadrature;

 public class Tax extends MovieClip
 {
  private var gamma:Number = 7000;
  private var x0:Number = 22000;
  private var population:Number = 65000000;
  
  private var gamma2:Number = 12000;
  private var x1:Number = 30000;
  private var beta:Number = 0.52;
  
  public function Tax()
  {   
   trace(calculate_total_teired_tax_collected(1000000));
   trace(calculate_total_new_tax_collected(1000000));
  }
  
  public function calculate_total_new_tax_collected(Limit:Number):Number
  {
   var S:SimpsonsQuadrature = new SimpsonsQuadrature(new_tax_proportion,0,Limit,100);
   return S.integrate();
  }
  
  public function new_tax_proportion(X:Number):Number
  {
   return population*calculate_new_tax_collected(X)*evaluate_income_percentile(X);
  }
  
  public function calculate_new_tax_collected(income:Number):Number
  {
   var S:SimpsonsQuadrature = new SimpsonsQuadrature(lorenzian_cdf,0,income+0.001,100);
   return S.integrate();
  }
  
  public function lorenzian_cdf(X:Number):Number
  {
   return (1/Math.PI)*Math.atan((X-x1)/gamma2)*beta + beta/2;
  }
  
  public function calculate_total_teired_tax_collected(Limit:Number):Number
  {
   var S:SimpsonsQuadrature = new SimpsonsQuadrature(teired_tax_proportion,0,Limit,100);
   return S.integrate();
  }
   
  public function teired_tax_proportion(X:Number):Number
  {
   return population*calculate_teired_tax_collected(X)*evaluate_income_percentile(X);
  }
  
  public function calculate_teired_tax_collected(income:Number):Number
  {
   var taxable:Number = income - 7475;
   if(income>100000) taxable = income;
   
   var basic_rate:Number = taxable;
   if(basic_rate>35000) basic_rate = 35000;
   
   taxable = taxable - basic_rate;
   
   var higher_rate:Number = taxable;
   if(higher_rate>115000) higher_rate = 115000;
   
   taxable = taxable - higher_rate;
   
   var additional_rate:Number = taxable;   
   
   if(basic_rate<0) basic_rate = 0;
   if(higher_rate<0) higher_rate = 0;
   if(additional_rate<0) additional_rate = 0;
   
   return basic_rate * 0.2 + higher_rate * 0.4 + additional_rate * 0.5;
  }
    
  public function evaluate_income_percentile(X:Number):Number
  {
   //Model as lorenzian
   return (1/Math.PI)*(gamma/((X-x0)*(X-x0) + gamma*gamma))   
  }
 }
}

It isn't commented in the slightest but see what you can make out of it!

iPhone Location Data

For those of you who haven't yet heard, Apple are tracking your every move. As somebody who has nothing to hide ( hopefully ), I thought I'd share an experiment I'd carried out using some of my own location data and OpenHeatMap. Apple store triangulated data based on nearby cell masts and not based on GPS which is a slight shame because the accuracy isn't great. Here is a greatly speeded up video of my actions since December 2010.

Aerodynamics for Convex Hulls

I had quite a nice thought experiment and thought I'd document it for you guys to have a look at. I wanted to come up with a quick way to (semi) realistically simulate lift and drag on any shape. Such an engine could be used for flash games for example a hang glider game. Realism like this (if carefully implemented) can add a lot to a game.

I plan on implementing this in flash sometime after my exams are over, but some of you might be interested in my approach, so feel free to beat me to coding it!

Here is what I've got so far, you can click on a page to expand it in a lightbox (or click here to download the pdf):

Fun with Inverse Kinematics

In part to try out AXDT, a flash plugin for Eclipse (which you can find here), I decided to create a simple experiment involving Inverse Kinematics. This is not a very useful implementation but the results are quite fun. Essentially, the angles of subsections of  each kinematic chain are altered until they face the mouse. I quickly discovered this is not the best solution for IK, but just for fun I made a kind of windswept, fur or grass effect using about 200 IK chains. 

As usual click the top screenshot to launch the experiment and click here for source:

New Source Code Page

Hey guys,

Due to many requests being for source code I have added a source section to the blog. When my exams are over I'll work through adding source code to older projects but for now I've included a few from some new stuff I've been working on. Just click the tab at the top from anywhere to access it.

Happy flashing!

Sam

Update: Colourful Turtle

In my last experiment I recreated a turtle vector drawing program, with a simple interface. I'll post a small update to that which adds some color to the output. Here are some screenshots, as usual click the first one to launch the image, and source code can be found at the bottom, enjoy and let me know if you make anything pretty!

Source:

package org.experimentalized
{
 import flash.display.Sprite;
 import flash.events.Event;
 import flash.events.KeyboardEvent;
 import flash.display.Graphics;
 import flash.text.TextField;
 import flash.text.TextFieldType;
 import flash.geom.Rectangle;
 import flash.utils.getTimer;

 [SWF(width='550',height='550',backgroundColor='#000000',frameRate='30')]

 public class Turtle extends Sprite
 {
  private var commandstring:String = new String("mv:30,lt:90,mv:10,rt:30");
  private var S:Sprite = new Sprite();
  private var Angle:Number = new Number(0);

  private var X:Number = 0;
  private var Y:Number = 0;

  private var R:int = 0;
  private var G:int = 0;
  private var B:int = 0;


  private var input:TextField = new TextField();

  
  private function newrun(e:KeyboardEvent):void
  {
    commandstring = input.text;
    S.graphics.clear();
    S.graphics.lineStyle(1,0xFFFFFF);
    S.graphics.moveTo(0,0);
    Angle = 0;
    X = 0;
    Y = 0;
  }

  public function Turtle():void
  {
   addEventListener(KeyboardEvent.KEY_DOWN, newrun);
  
   addChild(S);

   input.width = 530;
   input.height = 20;
                 input.x = 10;
                 input.y = 10;
                 input.background = true;
                 input.backgroundColor = 0xFFFFFF;
   input.type = TextFieldType.INPUT;
   
   input.text = commandstring;


   addChild(input);

   S.x = 275;
   S.y = 200;
      
   this.addEventListener(Event.ENTER_FRAME, handler);
  }
  
  private function handler(e:Event):void
  {
   R = int(128 * (1 + Math.cos(getTimer()/1000-0.8981)));
   G = int(128 * (1 + Math.sin(getTimer()/7473-0.3812)));
   B = int(128 * (1 + Math.sin(getTimer()/1273+0.1787)));
   
   var C:uint = (R<<16|G<<8|B);   
   
   S.graphics.lineStyle(1,C);
   var commands:Array = new Array();
   commandstring = commandstring.split(" ").join("");
   commands = commandstring.split(",");
   var i:int;

   for(i=0;i<commands.length;i++)
   {
    if(commands[i].charAt(0)=="m")
    {
     var dist:Number = Number(commands[i].split(" ").join("").split(":")[1]);
     X = X + dist*Math.cos(Angle);
     Y = Y + dist*Math.sin(Angle);
     S.graphics.lineTo(X,Y);
    }
    else if(commands[i].charAt(0)=="l")
    {
     var changeleft:Number = Number(commands[i].split(" ").join("").split(":")[1]);
     Angle -= Math.PI*changeleft/180;
     if(Angle<0) Angle += 2*Math.PI;
    }
    else if(commands[i].charAt(0)=="r")
    {
     var changeright:Number = Number(commands[i].split(" ").join("").split(":")[1]);
     Angle += Math.PI*changeright/180;
     if(Angle>Math.PI*2) Angle -= 2*Math.PI;
    }
   }
  

   //Resize to fit on the square after each iteration (purely aesthetic choice...)
   
   var Rect:Rectangle = S.getRect(S.parent);
   
   S.x += 275 - (Rect.x+Rect.width/2);
   S.y += 275 - (Rect.y+Rect.height/2);
   S.width = 550;
   S.height = 550;
  }
 }
}

Turtle: 30 minute experiment

I had recently been reminiscing about IT lessons in year 3 where we were allowed to play around with those Turtle vector drawing programs. I thought I'd write one and share the results as a quick break from revision. Although it can't do much at the moment (the only commands are forward, left and right) with a bit more work I'm sure it could be made into quite an exciting program. You'll find the source code below and as usual just click the top image to launch the experiment.

Source code:

package org.experimentalized
{
 import flash.display.Sprite;
 import flash.events.Event;
 import flash.events.KeyboardEvent;
 import flash.display.BitmapData;
 import flash.display.Bitmap;
 import flash.display.Graphics;
 import flash.text.TextField;
 import flash.text.TextFieldType;

 [SWF(width='550',height='400',backgroundColor='#000000',frameRate='30')]

 public class Turtle extends Sprite
 {
  

  private var commandstring:String = new String("mv:30,lt:90,mv:10,rt:30");
  private var S:Sprite = new Sprite();
  private var Angle:Number = new Number(0);

  private var X:Number = 0;
  private var Y:Number = 0;

  private var input:TextField = new TextField();

  
  private function newrun(e:KeyboardEvent):void
  {
    commandstring = input.text;
    S.graphics.clear();
    S.graphics.lineStyle(1,0xFFFFFF);
    S.graphics.moveTo(0,0);
    Angle = 0;
    X = 0;
    Y = 0;
  }

  public function Turtle():void
  {
   addEventListener(KeyboardEvent.KEY_DOWN, newrun);
  
   addChild(S);

   input.width = 530;
   input.height = 20;
                 input.x = 10;
                 input.y = 10;
                 input.background = true;
                 input.backgroundColor = 0xFFFFFF;
   input.type = TextFieldType.INPUT;
   
   input.text = commandstring;


   addChild(input);

   S.x = 275;
   S.y = 200;
   
   S.graphics.lineStyle(1,0xFFFFFF); 
   
   this.addEventListener(Event.ENTER_FRAME, handler);
  }
  
  private function handler(e:Event):void
  {
   var commands:Array = new Array();
   commandstring = commandstring.split(" ").join("");
   commands = commandstring.split(",");
   var i:int;

   for(i=0;i<commands.length;i++)
   {
    if(commands[i].charAt(0)=="m")
    {
     var dist:Number = Number(commands[i].split(" ").join("").split(":")[1]);
     X = X + dist*Math.cos(Angle);
     Y = Y + dist*Math.sin(Angle);
     S.graphics.lineTo(X,Y);
    }
    else if(commands[i].charAt(0)=="l")
    {
     var changeleft:Number = Number(commands[i].split(" ").join("").split(":")[1]);
     Angle -= Math.PI*changeleft/180;
     if(Angle<0) Angle += 2*Math.PI;
    }
    else if(commands[i].charAt(0)=="r")
    {
     var changeright:Number = Number(commands[i].split(" ").join("").split(":")[1]);
     Angle += Math.PI*changeright/180;
     if(Angle>Math.PI*2) Angle -= 2*Math.PI;
    }
   } 
  }
 }
}

To make the output lines automatically fit the stage use the following instead:

package org.experimentalized
{
 import flash.display.Sprite;
 import flash.events.Event;
 import flash.events.KeyboardEvent;
 import flash.display.Graphics;
 import flash.text.TextField;
 import flash.text.TextFieldType;
 import flash.geom.Rectangle;

 [SWF(width='550',height='550',backgroundColor='#000000',frameRate='30')]

 public class Turtle extends Sprite
 {
  private var commandstring:String = new String("mv:30,lt:90,mv:10,rt:30");
  private var S:Sprite = new Sprite();
  private var Angle:Number = new Number(0);

  private var X:Number = 0;
  private var Y:Number = 0;

  private var input:TextField = new TextField();
  
  private function newrun(e:KeyboardEvent):void
  {
    commandstring = input.text;
    S.graphics.clear();
    S.graphics.lineStyle(1,0xFFFFFF);
    S.graphics.moveTo(0,0);
    Angle = 0;
    X = 0;
    Y = 0;
  }

  public function Turtle():void
  {
   addEventListener(KeyboardEvent.KEY_DOWN, newrun);
  
   addChild(S);

   input.width = 530;
   input.height = 20;
                 input.x = 10;
                 input.y = 10;
                 input.background = true;
                 input.backgroundColor = 0xFFFFFF;
   input.type = TextFieldType.INPUT;
   
   input.text = commandstring;


   addChild(input);

   S.x = 275;
   S.y = 200;
      
   this.addEventListener(Event.ENTER_FRAME, handler);
  }
  
  private function handler(e:Event):void
  {
   
   S.graphics.lineStyle(1,0xFFFFFF);
   var commands:Array = new Array();
   commandstring = commandstring.split(" ").join("");
   commands = commandstring.split(",");
   var i:int;

   for(i=0;i<commands.length;i++)
   {
    if(commands[i].charAt(0)=="m")
    {
     var dist:Number = Number(commands[i].split(" ").join("").split(":")[1]);
     X = X + dist*Math.cos(Angle);
     Y = Y + dist*Math.sin(Angle);
     S.graphics.lineTo(X,Y);
    }
    else if(commands[i].charAt(0)=="l")
    {
     var changeleft:Number = Number(commands[i].split(" ").join("").split(":")[1]);
     Angle -= Math.PI*changeleft/180;
     if(Angle<0) Angle += 2*Math.PI;
    }
    else if(commands[i].charAt(0)=="r")
    {
     var changeright:Number = Number(commands[i].split(" ").join("").split(":")[1]);
     Angle += Math.PI*changeright/180;
     if(Angle>Math.PI*2) Angle -= 2*Math.PI;
    }
   }
  

   //Resize to fit on the square after each iteration (purely aesthetic choice...)
   
   var Rect:Rectangle = S.getRect(S.parent);
   
   S.x += 275 - (Rect.x+Rect.width/2);
   S.y += 275 - (Rect.y+Rect.height/2);
   S.width = 550;
   S.height = 550;
  }
 }
}

Large Scale Stop Frame

In my spare time I study physics at the University of Birmingham. We're currently having some building work done on our main courtyard, so with access to one of the security cameras from our main clock tower (provided by an unnamed physicist) I decided to document the events. I wrote a program which is currently running on the physics mainframe, which essentially grabs an image from the security camera every 2 minutes during the day. I compiled these images into a video (with about 6000 frames so far) and you'll find the initial results in the vid below:



I plan on taking a few more weeks of images, the above is about 6 weeks. Enjoy.

For those of you who are interested here is the bash source code:

#!/bin/bash
#PBS -j oe
#PBS -l walltime=240:00:00

CLOSING_TIME=1
WEEKEND=0

cd /home/st-01/pictures

while [ 1 ]
do
#Determine the current time
hour=$(echo `date +%k`)

#If the current time is between 9am and 6pm not inclusive
if [ $hour -gt 9 ] && [ $hour -lt 18 ]; then
#Still taking photos - don't close up yet
CLOSING_TIME=0

#Retrieve image from the server and save it with a unique timestamp
wget http://YOURSITE.com/IMAGE.jpg
timestamp=$(echo `date +%y-%m-%d__%H:%M:%S`)
mv IMAGE.jpg ${timestamp}.jpg
else
#Otherwise must be the end of the day during the week
if [ "$CLOSING_TIME" == "0" ]; then

#ZIP UP THE IMAGES FOR THE DAY
date=$(echo `date +%y-%m-%d`)
zip ${date}.zip `ls *.jpg`
#Remove all non-zipped images
rm *.jpg
CLOSING_TIME=1
fi
fi
sleep 120
done

Multi-Color-Space-Thresholding

A few months ago I was commissioned to create some background subtraction technology in flash. The project fell through at the time. The final product wasn't as stable as I'd hoped it might be. I thought that since the project fell through I would share some of my findings to see if the collective flash community might have any ideas or suggestions as to how one might implement this kind of technology.

The idea of background subtraction is to separate two parts of an image, the foreground and a "static" background. Here is an example. Given the two images on the left, how does one extract the image on the right. You might think this is fairly easy, just look at the difference in color between the two you might say. Unfortunately it isn't as simple as this.

Firstly illumination and varying camera brightness cause a problem, not only does the foreground constantly change color, but also the background. Most web cams automatically adjust to a white point. You'll notice this if you hold a white sheet of paper against the web-cam, the background will dim. This means that in measuring the difference in RGB color for each pixel it is likely that the whole screen will be completely different from one frame to the next, not ideal if you want to separate out smaller regions!

The method I chose to incorporate was to use multiple color spaces, combined in order to create a new color space which was optimum for this kind of a background subtraction scenario. Here is the program I used to determine the optimum subtraction for each color space. Just click on the image to launch it:

Initially a background is captured, the user steps out of the frame, and a still bitmap image is taken, then for each color space a comparison is made. If the difference between given pixels is greater than a threshold for any color component (or a combination of the three) then the pixel is deemed to be in the foreground. The three color spaces used in this case where YUV, normalized RGB and normalized YUV. The latter of these I haven't seen in use before, but it gives reasonable results.

Once the image has been thresholded an alpha mask is applied to the current web-cam frame, and this image is used as the foreground. This allows some very impressive things to be done. The background can be replaced by an image, or another video (apple make use of this in their software Photo Booth), but more excitingly we now have two image planes, which means 3D!

Here is an example of background replacement:

In the image below you'll see 3D in action (excuse the poor lighting conditions), click to launch it but remember, it probably won't work fantastically. Just click to launch it:

The 3D angle can be controlled by head position (motion tracking) allowing you to look around the person in front of the camera which is quite an interesting effect.

The main issues with the project were that lighting conditions are constantly changing, and can vary hugely from web-cam to web-cam. There are other more computationally intensive approaches which flash may just about be able to handle, but unfortunately my brain cannot. Enjoy the experiment, and let me know how it works with your web-cams!

HTML5 Mandelbrot Explorer

A simple Mandelbrot explorer for HTML5, in javascript. I plan on optimizing and extending the functionality of the example when I have a bit less uni work to keep up with.

First Canvas Experiment: BitmapData

Lots of people have been talking about html 5 and canvas for a while now. As an actionscript purist I had decided not to venture into the world of incompatibility quite yet, but having seen some of the cool chrome experiments again recently, I decided to give writing some canvas toys a little go.

My first experiment is a simple one, just some randomly updating pixels on a screen.

Your browser doesn't support html5
Here is the .js code:

function setPixel(bitmapData, x, y, r, g, b, a) {
index = (x + y * bitmapData.width) * 4;
bitmapData.data[index+0] = r;
bitmapData.data[index+1] = g;
bitmapData.data[index+2] = b;
bitmapData.data[index+3] = a;
}

element = document.getElementById("bitmap");
canvas = element.getContext("2d");
width = parseInt(element.getAttribute("width"));
height = parseInt(element.getAttribute("height"));

setInterval('update()',30);

function update()
{
bitmap = canvas.createImageData(width,height);
for (i = 0; i < 10000; i++) {
     x = parseInt(Math.random() * width);
     y = parseInt(Math.random() * height);
     r = parseInt(Math.random() * 256);
     g = parseInt(Math.random() * 256);
     b = parseInt(Math.random() * 256);
     setPixel(bitmap, x, y, r, g, b, 0xff);
 }
 canvas.putImageData(bitmap, 0, 0);
}

Transparency and Reflection

I haven't posted anything new on my blog in quite a while. Most of my time has been spent working on writing a fully function Actionscript plugin for Gedit, which will (with any luck) improve the development experience for users of Linux.

Anyway I thought I'd post some eye candy, just a simple transition between two ASTrace renders. I be plan on working on a pixel bender filter to carry out these effects in real time. A kind of one-track raytracer which optimizes a single scene with a central sphere. This should be possible with two displacement filters, one for reflection and one for refraction. For now, here is a prototype without full 3D rotation (Click to launch then move the mouse left or right to change reflectivity):