Project Natal aims to be pretty much everything I was working on with cameras and stereoscopic vision in college. The setup is eerily the same too, with dual fixed cameras negotiating the 3D scene. Youtube video here. Since I have walked this path, however short it was for myself, I do have some thoughts about this project and potential limiations:
Camera speed and synch:
Both camera's have to synch frames (ideal) or do some extra processing to negotiate and calibrate themselves for the difference. This is a bit processor intensive to have a perfect synch of the scene and to negotiate the differences if they are not hardware synched.
I am hoping that Microsoft realizes how much money thay can make from this type of system and invest some real money to help offset the costs for the consumer on the hardware. High quality (resolution/fps) cameras don't come cheap and are what is needed to have this system accurate.
Camera auto focus/ auto brightness and color calibration:
The operating space for these cameras will likely be in very bright rooms with constantly changing lighting situation- between daylight with constant change in brighness due to clouds, to directional and poorly lit indoor lighting. I have a feeling infrared light projection may be ok for motion and object detection and tracking, but change in lighting situation or ambient artifical light may cause innaccuracies.
Processor intensive, highly parallelizable:
I am guessing this system will not be on xbox 360 considering hardware limitations, and will likely instead be on some platform that can take full advantage of the latest in parallelizable technologies and stream computing- such as nvidia CUDA or OpenCL. I can see this being the ideal application for next generation hardware consisting of hundreds of processor cores chugging away at image processing.
I really really hope this hardware comes out because it is exactly the setup i had been playing with years ago and it would be a dream come true to see it in its final/finished form the way I had designed it.