IDAT204 – Augmented reality proposal

I think I want to go ahead with the idea of displaying sound in my augmented reality project.

An image below shows a practical use, as an example; a fishing app for the iphone.

You would point the phone at an area and on the screen it will reveal that area using sonar waves. It will overlay these sonar waves on top of the live video feed.

Above is just an example of how it could be used to reveal the seabed in an ocean, but you could also use it for example to reveal the contents of a cave in complete darkness, or just looking at a normal place but with sound over the top, so you can see information such as distances of objects overlayed on top of the live feed for example.

IDAT204: Augemented Reality, overlaying information

I started looking into ways of overlaying information on a live feed, and found this video.

I’d like my project to display in a similar manner, displaying things like distance and decibel information, as well as the sound waves.

IDAT204: Augmented Reality

I started looking into sonar and its uses.

A common use for sonar is on ships for finding fish in the sea.

However, these devices aren’t the easiest to use, dont display in real time and dont show actual images.

An example of an image captured using traditional sonar is shown below:

Below is a 3d illustration of how the underwater scene could be revealed and what it may look like. This is what I want to reveal in my augmented reality project.

I would like to use this idea of using sonar to overlay a realtime feed. So for example, you could direct a screen over a pond and whats in the water from the live sonar feed that is put over the top of it.

This would probably be the main use of the project but it could also be used in many other waves, such as seeing inside a pitch black cave without a torch.

IDAT204: 3D Sonar

After rethinking my initial idea about re-representing sound I found some inspiration from the film Batman the Dark Knight.

As you can see from the clip sonar is used to reveal a space via sound waves, that otherwise was hidden.

Id like to recreate something similar using the idea of bouncing sound waves off an area in realtime, overlaying it on top of reality, using some kind of screen/glasses that is yet to be decided.

IDAT204 – Augmented Reality

This project is about Augmented Reality which is basically changing a real world location or object;

“A combination of a real scene viewed by a user and a virtual scene generated by a computer that augments the scene with additional information.”

At this stage I just want to throw some ideas around but initially I think it would be interesting to explore the relation of sound in our environments, and see if I can somehow visualise sound or modify sound within a space.

One idea is to set up microphones in a real world location, and use the data from this to recreate the scene in an alternate location. Perhaps using sonar microphones to bounce frequencies off a real world location, to display either over the top using video, or perhaps recreate the scene using the sonar data.

My 2nd idea is again the same set up of microphones, however this time also setting up speakers, and inverting the sound based on volume.

What I mean by this is recording a constant stream of sound from a number of microphones and relaying these using speakers set in close proximity to the microphones. The twist being, before the sound is outputted from the speakers, the audio data is interperated to see which microphones picked up higher/lower volumes, and reversing this.

So the speaker that is next to the microphone that is picking up the quietest sounds, is replaying this sound but amplified by a large amount. The same is done for the microphone picking up the loudest sounds, this is quietened significally when output through the speaker. Hopefully the outcome of this will be that the space will loose its focus in attention to certain areas due to sound, therefore augmenting the persons experience in that space.

My final idea involves a similar set up, except this time the speakers are placed in a remote location;

The idea of this is to get the feeling you are traveling through the space remotely, I could also integrate a live video feed from the remote location further enhancing this experience.

Im not too sure with these ideas but hopefully they are a good starting point and I can making some better decisions after speaking to Vlad about them.

IDAT211 – Space Simulator – Final Render (Without dome correction)

Now I have done my heads up display, this is the final render showing the video with the heads up display and windscreen overlapped.

I have also added the audio although for Youtube this had to be in stereo format converted from 5.1 so annoyingly sounds from the rear speakers and certain panning effects have been lost, but its a good draft to see it before it gets crazy distorted by the fish eye!

IDAT211 – Adding sound

After finding out about the 10.2 sound system in the Dome, I definatly wanted to use this in some way.

I thought about where sound would come from in an actual space shuttle and wanted to utalise the surround sound to recreate these sounds realistically, further adding to the perception of the video that you were actually inside a space shuttle.

For example, engine sounds would come from behind and below you, particles and wind blowing on the windshield would come from the front, so I could use the surround sound and assign each element of audio to a certain speaker(s).

As you can see below I exported the final video (without sound, as After effects does not offer surround sound mixing) into Adobe Premiere, and used the audio mixer as shown below:

You can just about make out which audio tracks are assigned to which speakers, if you look along the top you can see diagrams of speakers, with a black dot representing the sound. So the black dot placed at the bottom right of the diagram will come out of the rear right speaker only, the dot placed in the middle between the front left and right speakers will come out of, obviously the front left and right speakers etc etc!

I also used the record function, allowing me to drag the dot around while recording a path in relation to the video. For example I panned the sound of the engines from the front speakers (when it was on the ground) to the back speakers when it was flying through the air (as the sound would now be coming from behind only).

I think its a big part of my video and the addition of surround sound means I can use fully diegetic sound to enhance the believability of the film.

IDAT211 – Heads up/Cockpit display

Now that I have my final render of the camera travelling into space and it all looks pretty good, I need to add the heads up display and windshield.

I searched around for various images taken from inside cockpits to get inspiration from, and also to make it look as accurate as possible.

I found the above image that looks quite cool. You can kind of see a slight transparent windscreen over the actual outside world as well, which I want to include.

From this I created my display and made the windshield transparent so I can just overlay this over the video.

Its not fantastic as I havent spent as much time on it as I probably should of, but it will do for the minute and gives that extra dimension to the realism of the video.

I also hope to animate it (change the numbers and make it crack up when exiting the atmosphere) if I have enough time.

IDAT211 – Space Simulator – First Render

So here is the first render showing the base video. The next step is to apply the heads up display and windshield over the top, and fish-eye, then the final render can be completed.

IDAT211 – Final scene

After finding the first video render was good and could cope with what I wanted to do, it was time to get cracking.

I added more clouds, the actual buildings by the Dome (the Mary Newman hall etc) and also a tree as there is a tree slightly overhanging the dome also.

I put in the Earth and sun (with Lens flare) along with other wind and rumble effects on the aircraft.

Below is a screenshot of the complete scene from an alternative camera, obviously zoomed out a very, very long way so you can see everything.

Hopefully you will be able to see the various elements in context, the camera starting out at the bottom, and travelling upwards. I have placed a thumbnail of what the active camera sees at the moment the screenshot was taken (bottom left).