Today, the United States Patent and Trademark Office released an Apple patent application for a brand new home invention that primarily covers a new outdoor camera system that communicates with people in a home who are within the front door via a HomePod mini, Smart TV, Apple TV, iPhone, iPad and more. Apple’s inventors came primarily from their machine learning and computer vision teams.
Apple’s patent application covers techniques for providing or removing a notification when a particular person is detected in a particular location. For example, a user might want to be notified if someone they recognize is at the front door.
However, in other cases, they may not want to be disturbed if the person at the door is someone they recognize. In one example, video or still images of a person may be captured by a computing device (eg, including at least one camera). In some examples, if that person is detected as a particular known person, the computing device may decide to notify the user that the particular person is at the particular location – the front door for example. This way the user will know that someone they know / recognize is at the front door.
However, in other examples, the computing device may decide to suppress the notification if the detected person is recognized, thus avoiding unnecessary notifications (for example, the user may not care to be informed whether his spouse or roommate is detected at the front door).
A second part of the patent relates to a device like a HomePod mini that can capture images. In some examples, a resident device (for example, a home automation device such as a smart speaker, digital media player, or other device) may receive one or more cropped images from a user device (for example, a mobile phone). Image crops can be generated from a plurality of images that are managed as part of an image library (eg, a photo library) stored on the user device, and including associated contacts. to the user device.
Each image crop may have been selected for reception by the resident device based at least in part on a determined level of information gain associated with the particular image crop, which can enable the resident device to perform facial recognition of the face of the first person.
The resident device (HomePod mini) can also receive one or more images from another device that includes a second camera (e.g., a home observation camera), the second camera having a viewing area including a particular location (e.g. example, the front door of a porch house).
The image (s) may, respectively, include part of a face of a person at the front door, whose identity has not yet been determined. The resident device (HomePod mini) can then determine a score that corresponds to a level of similarity between a first set of characteristics associated with the face of the first person and a second set of characteristics associated with the face of the person at the front door. whose identity is not yet determined.
Based at least in part on the score, the resident device can then determine whether it is in fact the same person (for example, whether the person at the front door is the same as the personal contact in the room). photo library). The resident device (HomePod mini) can then provide a notification based at least in part on the determination. For example, the resident device may transmit notification to the user device that the person at the particular location is (or is not) the same as the first person, who has been identified as a contact associated with the user device.
In an illustrative example, consider a scenario where a device residing in a home environment provides notifications about the presence of a person near (or inside) the home. In this example, the resident device may be a home automation device (eg, smart speaker, smart digital media player) that is communicatively connected to a camera.
In one example, the camera can be configured to observe the area around the front door of the house (for example, to capture images of people who may knock on the door and / or ring the doorbell). Therefore, the resident device can be configured to receive and process one or more images from the observation camera.
In some embodiments, the facial feature model can be trained to detect and recognize a face in a situation where a person is wearing a face mask. For example, consider a scenario in which a person approaches the front door of a house wearing a face mask. In this scenario, the facial feature model can recognize that the person is wearing a face mask and generate a first facial imprint of the face of the person wearing the face mask.
The facial feature model can be trained to account for the presence of a face mask, and can compare the first facial print with facial prints generated from images of faces in a reference set of images taken from the photo gallery of a user device. In some embodiments, although a face can be recognized even with the presence of a face mask, the image of the face with the face mask may not be included in a notification sent to the user device.
Apple’s patent FIG. 1 below is a block diagram (# 100) that illustrates a System Notification Service operating in a Home # 101 environment. The family environment can include one or more people who have a certain affiliation (eg family members, roommates, etc.). In this example, user # 106 and user # 110 may represent affiliated users and may be associated with user devices # 108 and # 112, respectively. In addition, in the home environment, there may be have a resident device # 102 (for example, a tablet, smart home controller, smart digital media player, home automation device, or smart TV device).
(Click on the image to enlarge)
Apple’s patent FIG. 3 below is another simplified diagram showing an example of a process executed by a system. Process # 300 is an example of a high level process for a system (eg, resident device # 102 of Figure 1) determining the presence of a person at a location such as a front door. In some embodiments, a first set of # 313 features may correspond to a facial imprint (or multiple facial imprints) generated from the plurality of image reframes # 311. Person # 317 can match person # 120 in Fig. 1, a camera # 315 may correspond to the observation camera # 122 of FIG. 1, an image # 319 (for example, which may be one of a plurality of images (or frames)) may correspond to an image captured by camera # 315, and a second set of features # 321 may correspond to a facial print generated from image # 319.
This is a completely new concept for Apple and as such the details are plentiful. Examine Apple’s Patent Application Number 20210383100 for more.
Since this is a patent application, the timing of marketing such a product is unknown at this time.
Some of the inventors of Apple
Hendrik Dahlkamp: Responsible for machine learning; HomeKit Secure Video
Vinay Sharma: AI / ML, Computer Vision, Deep Learning. Understanding of the human and the object
Nitin Gupta: Machine learning engineer
Floris Chabert: Senior Research Engineer
Jonghoon J.: Deep learning for computer vision
Camera and visitor user interfaces
A second patent on this subject was issued today, titled “Camera and Visitor User Interface. The Apple Patent Figures 6D and 6E below illustrate a front door camera user interface that could send alerts to a user while watching Apple TV.
You can view Apple’s second patent application 20210383130 here for details.
Activity zones for camera video
A third patent on this subject was published today, titled “Activity Zones for Camera Video”.
Apple says that currently cameras can be used to monitor certain aspects of buildings. In some cases, cameras can be integrated into a smart device. For example, cameras can be included in an electronically controlled doorbell to provide video associated with an entity that activated the doorbell. When the camera detects motion, it can send a notification to a user that motion has been detected. When cameras are positioned in view of public spaces (e.g. sidewalks, public streets, etc.) or neighboring buildings (e.g., shared aisles, shared aisles, windows of adjacent buildings, etc.), users can be inundated with notifications associated with movement in public places or neighboring buildings that are irrelevant to the user.
 A user device can establish an activity zone or an exclusion zone in a video representing a physical view captured by the camera. An automatic surveillance system can analyze the video from the camera against the activity zone or the exclusion zone and trigger only the notifications associated with a zone in order to limit the notifications sent to the user device to those notifications relevant to the zone. ‘user. With activity or exclusion zones, video from a camera can be automatically monitored to reduce the volume of notifications sent to the user device.
Apple’s patent FIG. 5 below illustrates an example of object detection operation using exclusion zones.
Examine the patent application 20210383554 for more details.