(Click on image to enlarge)
Over the next few years, Apple’s next big thing will involve a series of mixed reality head-mounted devices and smart glasses. The camera system that Apple may introduce over time is an advanced camera system that is anything but conventional.
On Thursday, the US Patent and Trademark Office published a patent application from Apple for a new HMD camera system that includes a two-dimensional array of cameras that capture images of respective parts of a scene.
The cameras are positioned along a spherical surface so that the cameras have adjacent fields of view. The entrance pupils of the cameras are positioned at or near the user’s eye while the cameras also form optimized images at the sensor. Methods for reducing the number of cameras in an array, as well as methods for reducing the number of pixels read from the array and processed by the pipeline, are also disclosed.
While today’s patent report might be appreciated by all Apple fans, the patent will be especially appreciated by engineers, optical engineers, and technicians who follow all things VR headsets.
Overall, Virtual Reality (VR) allows users to experience and/or interact with an immersive artificial environment, so that the user feels as if they are physically in that environment. Simulated environments of virtual reality systems and/or mixed environments of mixed reality systems can thus be used to provide an interactive user experience for multiple applications, such as applications that add virtual content to a real-time view of the spectator’s environment, applications that generate 3D virtual worlds (Metaverse), interact with virtual training environments, play games, remotely control drones or other mechanical systems, view digital multimedia content, interact with the Internet , explore landscapes or virtual environments, etc.
Apple’s patent covers stage cameras for transparent head-mounted displays (HMDs) that can be used in mixed reality (MR) or virtual reality (VR) systems.
In conventional HMDs, one or more scene cameras can be mounted on the front of the HMD. However, the point of view (POV) of the scene cameras is noticeably offset and different from the POV of the user’s eyes. Apple’s invention corrects the point of view (POV) of the cameras to match the user’s POV by moving the cameras’ entrance pupils towards the user’s eyes.
In some embodiments, an HMD includes two-dimensional arrays of small form factor cameras (e.g., one left-eye array and a second right-eye array) that capture images of respective portions of a real world scene in front of the user. The cameras are positioned along a curve or spherical surface so that the cameras have adjacent non-overlapping fields of view (FOV).
To obtain a more accurate representation of the user’s perspective, the camera optics are configured so that the entrance pupils of the cameras in the array are positioned behind the camera image planes formed at the sensors. image, and at or near the user. the eye while the cameras also form images optimized at the sensor level. In this way, each camera array captures views of the scene from substantially the same perspective as the user’s respective eye.
In some embodiments, one sensor may be used to capture images for multiple (eg, four) cameras in the network. In these embodiments, the optics of cameras used with a sensor can be shifted or adjusted to align the image planes with the sensor surface.
In some embodiments, a curved sensor and simpler lens system can be used to provide a wider FOV for each camera and thus reduce the number of cameras in the array.
Apple’s patent FIG. 1A below illustrates a head-mounted display (HMD) that includes an array of cameras with entrance pupils at or near the user’s eye; FIG. 2 illustrates part of a network of cameras.
Apple’s patent FIG. 3 below illustrates an example of a camera with an entrance pupil at or near the user’s eye that can be used in a network as shown in Figs. 1A and 2.
(Click on image to enlarge)
Apple figures. Figures 9A and 9B below illustrate the reduction in the number of cameras by reducing the resolution and shifting the entrance pupil of the cameras in the peripheral regions.
Apple’s patent FIG. 16 above graphically illustrates the addition of a negative meniscus lens in front of a camera array to increase the cameras field of view (FOV) to the periphery; FIG. 17 graphically illustrates the addition of a negative meniscus lens in front of a camera array to gradually shift the point of view (POV) of the cameras towards the periphery.
Apple’s patent FIGS. Figures 20A-20C below graphically illustrate an example scene camera that includes a negative meniscus lens in front of the cameras in the camera array.
To dig deeper into the details of Apple’s patent application number 20220035160, click here.
Some Apple Inventors
Dan Hennigan: Optical Mechanical Engineering Manager
Saito Kenichi: Optical Engineer (previously worked at Cannon & Fuji Photo Film)
Bret Miller: Engineering Manager, Camera Incubation
Noah Bedard: Prototyping engineer
Catherine Berkner: Senior Engineering Manager – Camera Incubation