Multi-Color-Space-Thresholding

A few months ago I was commissioned to create some background subtraction technology in flash. The project fell through at the time. The final product wasn't as stable as I'd hoped it might be. I thought that since the project fell through I would share some of my findings to see if the collective flash community might have any ideas or suggestions as to how one might implement this kind of technology.

The idea of background subtraction is to separate two parts of an image, the foreground and a "static" background. Here is an example. Given the two images on the left, how does one extract the image on the right. You might think this is fairly easy, just look at the difference in color between the two you might say. Unfortunately it isn't as simple as this.

Firstly illumination and varying camera brightness cause a problem, not only does the foreground constantly change color, but also the background. Most web cams automatically adjust to a white point. You'll notice this if you hold a white sheet of paper against the web-cam, the background will dim. This means that in measuring the difference in RGB color for each pixel it is likely that the whole screen will be completely different from one frame to the next, not ideal if you want to separate out smaller regions!

The method I chose to incorporate was to use multiple color spaces, combined in order to create a new color space which was optimum for this kind of a background subtraction scenario. Here is the program I used to determine the optimum subtraction for each color space. Just click on the image to launch it:

Initially a background is captured, the user steps out of the frame, and a still bitmap image is taken, then for each color space a comparison is made. If the difference between given pixels is greater than a threshold for any color component (or a combination of the three) then the pixel is deemed to be in the foreground. The three color spaces used in this case where YUV, normalized RGB and normalized YUV. The latter of these I haven't seen in use before, but it gives reasonable results.

Once the image has been thresholded an alpha mask is applied to the current web-cam frame, and this image is used as the foreground. This allows some very impressive things to be done. The background can be replaced by an image, or another video (apple make use of this in their software Photo Booth), but more excitingly we now have two image planes, which means 3D!

Here is an example of background replacement:

In the image below you'll see 3D in action (excuse the poor lighting conditions), click to launch it but remember, it probably won't work fantastically. Just click to launch it:

The 3D angle can be controlled by head position (motion tracking) allowing you to look around the person in front of the camera which is quite an interesting effect.

The main issues with the project were that lighting conditions are constantly changing, and can vary hugely from web-cam to web-cam. There are other more computationally intensive approaches which flash may just about be able to handle, but unfortunately my brain cannot. Enjoy the experiment, and let me know how it works with your web-cams!

Transparency and Reflection

I haven't posted anything new on my blog in quite a while. Most of my time has been spent working on writing a fully function Actionscript plugin for Gedit, which will (with any luck) improve the development experience for users of Linux.

Anyway I thought I'd post some eye candy, just a simple transition between two ASTrace renders. I be plan on working on a pixel bender filter to carry out these effects in real time. A kind of one-track raytracer which optimizes a single scene with a central sphere. This should be possible with two displacement filters, one for reflection and one for refraction. For now, here is a prototype without full 3D rotation (Click to launch then move the mouse left or right to change reflectivity):

Soft Shadows in ASTrace

Here is a nice example of fully antialiased soft shadows. The source light uses about 20 seed points spread out using a pseudo-random function, to produce reasonable effects.

Improved Supersampling and Antialiasing

I decided to improve the supersampling/antialiasing methods in my raytracer. Previously I applied a blur filter to the image after upsizing and then returned the image to its original size. This is not a very realistic or accurate antialiasing method so I decided to add some real supersampling to the raytracer. The two images below both have the same resolution, but the image on the right appears more realistic with fewer jagged edges. This is achieved by firing multiple rays for each pixel, and based on their relative position, calculate their contribution to the colour of that pixel. The obvious downside to this method is a largely increased rendering time, proportional to the number of rays per pixel. This means that 16 rays per pixel takes 16 times longer than a single ray. Quality definitely comes at a computational price. I think randomising the rays may improve the antialiased image quality, instead of firing them across a uniform grid and I will look into that.

The next updates are likely to be cloud maps and a sun object (infinite directional lighting).

Perlin Texturing in ASTrace

I've added simplex noise volumetric texturing to ASTrace. I found this nice little Simplex noise class by Sjeiti and implemented it in the library. Simplex is slightly faster than Perlin noise in 3 and 4 dimensions so it is ideal to use as a volumetric texture for primitives in the tracer, here are a few examples:

As an extension I decided to create an animation using 4-dimensional simplex noise, just click on the image to launch it. I am planning on implementing a cloud sphere, and this would be ideal for animations of clouds:

Material Properties

Choosing the right material to represent a particular object is important when producing realistic 3D graphics. Factors like refractive index, specularity, and reflectivity all affect how we perceive an object both in the real world and on screen, so ideally the real world case should be simulated as closely as possible in its computer based counterpart. The three images below show how differing physical features completely change the appearance of a scene. In the top image the sphere is 100% reflective, in the second it is partly transmissive and partly reflective and in the last image, the sphere is fully transmissive with a refractive index of around 1.08. I am pleased with the results I am getting from the raytracer, but adding more features increases the render time dramatically, especially for large numbers of objects. I am looking into porting the library to C++ for use in alchemy, which would show a dramatic increase in speed, possibly rendering scenes like the one below in real time. Till then here is the library.

ASTrace Pre Release

Although the library is nowhere near complete, I thought I would release a beta to see what you think. At this stage feedback would be really useful, including any suggestions you might have about extensions to the project. I should note that there is no documentation as of yet, this will be coming as soon as I manage to get asdoc to run properly. There should be plenty of scope for you to play around with the library, it'd be great if you could send me some of your renders.

Here is a link to the library: ASTRACE

and here are some of my favourite renders using the library:

FLAR Toolkit - First Go

I thought I'd finally get round to having a go with the FLAR toolkit (an augmented reality toolkit for flash). I made a marker which will be used in any AR experiments I do on the site, then quickly made a test program. All I can say is it works quite nicely and I'll be playing around with it in the near future. Click the second image to run the test!

Generating Normal Maps: 2D Images

The normal map used in the previous post was generated from a 3D model, so calculating normals was quite trivial. Taking any two dimensional image and calculating accurate realistic normals on the other hand is not. I have used a simple method using Sobel operators which seems to work quite well.

The Sobel operator provides X or Y gradient information about a certain image. Lets take a look at an example (Picture courtesy of my lovely girlfriend Camilla). The picture below shows a 2D image, with no 3D data whatsoever.

The sobel operator works by calculating the gradient of an image based on a range of pixels surrounding the pixel being tested.

First the image must be turned into a greyscale image, so that only brightness values are taken into account. Then a sobal operator is applied in the vertical and horizontal direction.

The results of applying the sobel operators can be seen in these two images. With the horizontal and vertical sobel operators respectively. These images correspond to the red and green channels in the final image. Since we have no sensible way of calculating a bump map for the two dimensional image the blue channel will just be set to a maximum throughout the image.

Finally we can apply this to the 2D Deferred Normal Lighting simulation I wrote about in the last post. The effects are quite impressive, creating a distinctly 3D look from a 2D image (Just click the image to launch the application):

To expand on this slightly further I thought it would be interesting to apply this to a webcam stream. This meant calculating a normal map every frame on top of the per pixel lighting so the resulting application is quite CPU intensive. I'm not quite as pleased with the effects as previously as there seems to be a lot of noise in the image. It might work better for you guys, just click the next image to launch it:

2D Deferred Normal Lighting

Continuing on the same lines as my previous post I thought I would create a few more 2 dimensional lighting simulations. With a little help from a post on normal mapping from a post on Planet Pixel Emporium I was able to create some real time 2D deferred normal lighting using actionscript. The simulation works on a per pixel basis so for large images the method would get very slow.

First lets take a look at the final result and then I will explain how it works. Click the image to the left to start up the application. You'll notice that moving your mouse around over the image moves the light source. Not only does this change information about how far the light should spread, but also the direction of the light with respect to every pixel on the screen.

This allows images to be rendered with a far more realistic depth and 3D feel (compare it to the original image on the right). For this reason the method is  used in many modern computer games.

The effect is created using a normal map. A normal map is a way of encoding data about the orientation of a plane using RGB colour. The red channel in the image represents the horizontal gradient of a face, the blue channel represents the vertical gradient, and the blue channel represents the height data (a bump map). The image below shows the normal map used for the image. I created the application by calculating the dot product of the normal at each point in the image with the light vector shining onto that pixel from the mouse's location.

The greater the dot product the more direct the light shining onto a point. In addition to this a distance falloff is used for the light to create the torch-like effect. I just use a factor of one over the square root of the distance as my falloff.

Obviously what isn't being considered in this is shadow casting so it isn't quite a real ray trace, but using bump map information it would be possible to create a real time semi 2-d ray trace.

Just for fun here is another sample, just click to launch it:

Galaxy

Just a little fun one because I've finished my exams and I haven't had time to start any big projects yet. A bit of very simple inverse square law gravitational physics to simulate the sombrero galaxy. All orbiting stars are attracted to the centre of the galaxy by an amount proportional to 1 over the square root of the distance between them.

Just click the two images below to launch the 2D and 3D simulations! Enjoy.

The three examples below show that with slightly different initial conditions, the shapes we find in a simple galactic simulation like this one, are actually surprisingly similar to those we see in real galaxies, for example in the first you can almost see the emergence of the spiral arms of our own galaxy the Milky Way.

3D Experiments - New Toy

I've been quite excited at the moment, playing around with my new toy Cinema 4d.

Here are some of of the first things I've made - click the images to run the videos. The first uses bullet physics and the second uses Cinema's hair plugin.

Enjoy!

Simple 3D Music Visualizer

Just played around a bit with the 3D fluid dynamics thing I made a few days ago. I added some computeSpectrum functionality using the Flash 10 audio features and created a very basic music visualizer. I'm not entirely happy with it but its worth a post, just click to start it up (the song might take a while to load):

The song is "Rabbit In Your Headlights" by Unkle, enjoy!

3D Waves: Projected 2D fluid simulation

I found this experiment quite interesting. In an earlier post I gave an example of a 2-dimension fluid simulation that I'd been working on using the Navier-Stokes equations. I decided to play around with this code a bit and use the 3D point engine from my last experiment in conjunction with the fluid engine. The density from the fluid simulation is simply projected onto the XZ plane with density values corresponding to the height of corresponding point particles.

The results are certainly interesting. Click the image below to start up the experiment. To create waves just move your mouse around and drag the movie to change the camera angle. I decided to add a slight fade and blur on the particles which makes it look very pretty. Enjoy!

OBJ Vertex Renderer

Just a really simple and quick experiment, to see if I could load a Wavefront .obj file into a flash movie and render the vertex co-ordinates in real time, in three dimensions. Just click the image below to launch the experiment. The object contains about 30,000 vertices, and these are simply transformed from 3D to 2D space and displayed using bitmap data. This is nowhere near as complicated as my prior attempt at a 3D engine, but I decided to make performance more of an issue this time. It does run at full frame rate quite happily on my computer. Let me know about your experiences.

The next step is an edge renderer (wireframe), then a face renderer. We'll see how high the performance can stay. Obviously the ideal case is a high quality raytrace in real time with shadows, reflections and more, but lets not kid ourselves.

Three Dimensional Projection

I came across a very interesting video recently. A design agency from the Netherlands called NuFormer Media have mapped various famous buildings in 3 dimensions. Using the three dimensional data, images can be projected onto the building, with amazing accuracy. The results are amazing and quite hypnotising, I especially like the crumbling building at about 36 seconds in. Just watch the video below!

Chrome Effect Ray Tracer

Applying a chrome filter to a finished ray trace render creates an amazing effect. Take a look for yourself. I'm in the process of developing online software where you'll be able to create simple 3D scenes and raytrace them with a number of effects; including anaglyph 3D, chrome and a few more! This is my screensaver at the moment:

Anaglyph Raytracing

I made a few changes to the raytracer I've been designing to create anaglyph images, that is two channel 3D images. The changes are very simple, literally two black and white images are drawn from slightly different angles, then one is tinted in cyan, and the other red. I don't have any 3D glasses but it should work. If anyone has some check out the test image below!

I'll try to produce some more images like this and get a pair of glasses to test with. Imagine realtime anaglyph raytracing!

Flash Ray Tracer Reflections

Finally managed to get reflections properly working in my actionscript raytracer. The mathematics behind reflections is very simple. Given an incoming light ray V, and a surface normal N, we can calculate the reflected light ray R using:

a = V.N
R = V + 2*a*N

I applied this theory to the ray tracer and the results are beautiful. Its amazing how much realism a few shadows and reflections can create! The number of reflections obviously changes the image realism. The images below show 1, 2 and 3 reflections per ray. Surprisingly there really isn't much of a reduction in performance - and for the purposes of a flash raytracer 3 reflections should be plenty!

Just need to add some functionality for box reflection and then I'll set the engine rendering some nice scenes!

One further extension I might set myself is diffuse reflections. Instead of light being reflected directly along the normal, there could be random fluctuations in the reflection vector. I think this could produce quite a nice wave effect. I'll keep you posted!

On a performance note the image with 3 reflections took just over 2 minutes to render in 1600x1600 with no real optimisation. For a resolution of 400x400 pixels it takes about 2.5 seconds. Not too bad at all in my opinion!

Update: Here are a few fuzzy shots. The first one is less fuzzy and the second one is more fuzzy. Quite an interesting effect I think:

Decided to go a bit mad and rendered a rather large image with a load of boxes and reflective spheres. Inspired by SuperJers PixelMachine, but rendered completely using flash and AS3, not OpenGL and C++. Here it is (Something has gone wrong with the reflections of the boxes - there is no shadowing - but I didn't have the heart to stop it once it was half way through!):

Here is another nice one, 2880 by 2880 pixels, back to the original room but with the reflection stuff added. Looks quite nice in the super high resolution!

Update: I've rendered few more large reflective images - there is still something slightly wrong with the reflections I think, check them out: