OpenAI today announced that it’s finally giving ChatGPT the ability to observe screens and live video to provide comments and feedback, a day after Google LLC launched similar features in Gemini 2.0.

The features, which essentially give ChatGPT eyes, were first teased in May at the launch of the multimodal GPT-4o model, with fans waiting for it since that time as OpenAI has rolled out other features.

With the new features, unveiled in a live stream today, ChatGPT can recognize objects via a camera on a smartphone or webcam, and react to what is on a user’s computer screen, as well as anything open on a user’s computer or phone.

In a demo from OpenAI, researchers turn on the screen-sharing feature to allow ChatGPT to provide real-time feedback during a graphic design project. ChatGPT identified on-screen elements such as colors, layouts and text, offering actionable suggestions to enhance the design.

The live video feature works similarly. When users give access to the camera on their device, ChatGPT processes the video feeds in real-time to identify and analyze objects, movements and contexts. The information is then used to provide relevant insights or instructions, such as identifying a piece of hardware in a tech setup or giving step-by-step guidance during a repair.

Just in time for the holidays, video and screensharing are now starting to roll out in Advanced Voice in the ChatGPT mobile app. pic.twitter.com/HFHX2E33S8

— OpenAI (@OpenAI) December 12, 2024

Additionally, OpenAI introduced a Santa voice feature, which allows users to interact with ChatGPT in a festive tone, complete with holiday-themed responses. The seasonal addition has been designed to add a layer of engagement and fun, particularly for families and children.

During the demonstration, ChatGPT’s Santa voice delivered personalized holiday stories and answered questions about Christmas traditions.

Say ho ho ho to Santa in Voice Mode 🎅

Santa is rolling out today to everyone across all ChatGPT platforms and is available until the end of the month…then he will retire back to the North Pole. pic.twitter.com/NVS9bRok4r

— OpenAI (@OpenAI) December 12, 2024

Though the features sound great, anything that uses live video or has live access to screens is going to raise security concerns.

To address those concerns, OpenAI noted that the new functions use guardrails to protect user data. For live video and screen-sharing, all data is processed locally whenever possible to ensure that sensitive information remains private. Additionally, OpenAI added, any data shared with the cloud is encrypted and not stored beyond the session’s duration.

The new features are being rolled out to ChatGPT Plus and Enterprise users starting from today, with wider availability planned for early next year. However, free users will not be getting access to the live video and screen-sharing capabilities since OpenAI wants to focus on making money from its premium offerings.

Image: SiliconANGLE/Ideogram