I had preview access to this one for a few weeks. It's very good. I had one conversation that lasted a full hour while I was walking the dog, got some good brainstorming done against one of my projects.
The best feature is that it can delegate questions out to GPT-5.5 in the background, so you're no longer restricted to a voice model that's several years behind the frontier.
I did report a fun bug with it though: it was interrupting me and laughing at my (not really intended as) jokes while I was still talking! They seem to have clamped that behavior down thankfully, it felt a bit rude and condescending.
Seems a bit knee jerk. I go on walks and bike rides all the time. A couple of times I’ve used voice mode and it’s been interesting. I could have listened to music or a podcast, listen to YouTube or just unplug. But every walk is different.
If you’re uncomfortable with this new world, and I’m sure I am even as I participate, you could tell us more about that?
Is it responsive to personality settings? I actively don't want fake AI girlfriend, but I do get a ton of value out of voice mode. Looking forward to trying this but hoping it's not a creepy overdone mess (like Sesame). Expectations are they'll keep doubling down on fake AI girlfriend approach because the thing I want probably wouldn't drive engagement anywhere nearly as well
I have my Chat personality settings stripped right down to no-fluff. I'd want voice to be more akin to the Star Trek computer, and less akin to as you said an AI friend, but previously it was tuned too personable/friend-like.
Star Trek computer voice model is something I have yet to encounter, and I've looked repeatedly :) It's not about a specific voice, it's the fact they managed to capture "I am a utility" perfectly in the voice. Our modern friends do not want to be thought of as a utility, but to engender trust and agency all of their own and that's a huge problem for me.
> "The best feature is that it can delegate questions out to GPT-5.5 in the background, so you're no longer restricted to a voice model that's several years behind the frontier."
Wow. That's exactly what I hoped they would do.
This issue has held me back from using ChatGPT's voice mode as much as I otherwise would have, because I also use it for brainstorming while commuting, exercising, etc., and don't want it to feel stuck in the past.
Is it a dumb-down version of GPT like the current voice model? At least in french, I find the current GPT voice mode to be useless, to the point I only use the dication mode. I would ask a question and it would answer something along "That's a interesting question. I can help you with that. Anything you want to know about X?" I would ask again and it would answer the same kind of non answer.
> the answer is no it uses the latest gpt models now.
Actually it says it _can_ delegate to the latest models. Seems reasonable to ask how the voice model does when it doesn't delegate (or while waiting for the delegated answer).
About 2 years ago I used to have conversations with GPT while walking the dog. It really emphasized the need to think before you speak, but you had to think fast before it hung up on you.
This is the opposite direction AI should be going. Human relationships are the most valuable thing we have, and so, naturally, technology seeks to intermediate and now replace them.
Most of what AI does is already in wrong direction. Not just human-to-human interaction, it took away thinking, creative work, sensory perception (glasses) and responses. People call it as helping humans, but I call it as sucking away the "human-ness" from humans.
After the damage is done, the mega corps would simply shrug and will say "Well, we were just responding to our business competition" The business knows no human-ness, because it is not a human. Businesses and machines are creatures that see humans as their fodder. And humans created these, assuming it is progress, to have businesses and machines. We call it progress because it required our mind power and it helped us to dominate other species. Dolphins are laughing at us.
I don't understand this line of reasoning. How are you hindered from doing any of those things? What part of "AI can now do X" makes it so you can't also do X?
We're yet to see how this plays out, but a competing business model for creative work is emerging, where it's delegated to chatbots. Naturally, this would result in less creative work for humans.
I'm no longer writing code from scratch, as I used to do before. So, very soon, it will be "AI can now do X" makes it so I can't also do X? Same with many creative works. Radio music already sounds so plastic. I lost interest in crafting my text drafts because I can just dump some ugly text and get it refined by AI.
Hasnt this already been thoroughly discussed? Your ability to do X degrades as you offload it more frequently, eventually to the point that you can no longer even vet the quality of the output.
I think the parent is saying now that that is attempting to be applied to "creativity" directly, as opposed to something like a shift of medium, that it threatens many peoples' ability to maintain creative capabilities.
Anecdotally I've already experienced this at work where post-AI we had a junior completely stagnate and a senior with over a decade of experience in the bay atrophy to the point that he had to be let go.
Yes, every minute you spend texting or talking to a chatbot is a minute that you'd have spent talking to another human beings. Literally the only important thing in life, the basis of all value, the formation of self-identity, comes from communication with other human beings.
Let's not overdramatize, though. I'm not in need, or even in mood to talk to fellow humans every minute, so time spent with a clanker is not necessary taken from my human relations budget
Isn’t voice I difference in degree rather than in kind? I definitely talk to AI as if it were human (one might say the UX of AI is to emulate a human). And a large portion of my interaction with humans is via text, for example, this post!
Right, I think the challenge is that LLMs are essentially language-based, which is itself a very convenient interface. Covering the maximally humanistic default interface with something more mechanical is like tying your own shoelaces together, but it would protect us from the psychological hijacking we're so prone to when interacting with these machines.
I see it more as a way technology could be abused rather than an inherent flaw in the technology itself. If you start to replace human interaction with chatbot interaction, that's bad, but there's nothing wrong with using a human-like chatbot in moderation. So many other types of technology are fine in moderation but can be abused in a human-interaction-replacing way: television, social media, video games, etc.
Where do you draw the line on how good human-machine interfaces should be? I'm sure this model could be a convenience for many, and while it may be social for some, I am not sure it would substitute existing human interaction for those users.
Besides, I do not think there is anything inherently immoral with not being social, or not having the ability to be. Consider for example people who do not naturally have the social network to interact with people they want to (e.g. some gifted children).
I am not convinced this model has enough empathy to satisfy most users on an emotional level. A bond is not merely an exchange of words, but prolonged and deep contemplation of the other being. We cannot introspect into these machines, and they certainly cannot yet do the same to us, yet.
How would talking to an AI as if it were not human sound? You can probably set your system prompt to insert “beep boop” between sentences and make it refer to itself as “Cybertron9000 Personal Computing Device” if that’s what you like. Is that an improvement? Or are you against voice computer interfaces altogether?
I don't know the answer. But a robotic voice is probably not a bad idea — just having a reminder that the thing you're talking to is not actually anything like you. If you want to go full send, you could have the LLM generate a clickable interface on demand so you could interact with it as a machine. Voice/computer interfaces are obviously useful, especially for disabled folks. But the ones that existed in the past didn't pretend to laugh at your non jokes or imitate vocal fry.
> This is the opposite direction AI should be going.
There is no moral obligation, in any domain, to refuse to make a product that adults, with full informed consent, find useful and purchase. Who are you to say you know better than the market?
Market conditions are not a moral standard either, nor do they represent any cohesive one in particular. Not sure why you're contrasting their opinion with this, it's literally no better.
What I’m missing from this announcement is the capability to use connectors and tools. I don’t really get it - NONE of the frontier assistants can use tools / connectors while in voice mode - Claude, ChatGPT, Gemini, Grok. It seems so obvious: I want to be able to research stuff, pull up documents, jot down notes and do productive work while I’m talking to it, and not end voice mode whenever I need to connect to an app or service.
It’s weird. The old Claude voice mode WAS able to use tools but when they revamped it, it lost that capability and is now pinned to Haiku :(
So, yay for finally a voice mode that’s powered by a frontier model and hopefully as good as Grok voice, but sad to still not see tool use while in voice mode.
(I haven’t tried it yet, only read the announcement)
A big part of this announcement does seem to be _delegation_ in the background; they give the example of web search but that could be any tool. I haven't tried it yet either but sounds like they've found a reasonable UX that mixes that sort of high-and-variable latency tool calling (potentially with agentic loops) with a continuously speaking live voice.
I could see it relating to tools having unpredictable latency but if they already do background hand off to 5.5 then it seems like they could just enable it within that context.
I’ve been using gpt-realtime-1 for my personal assistant that runs my company and plans my day. And it works pretty well, even makes tool calls and all that.
But the multi modal stuff has resulted in a lot of debugging with weird events and message and audio sequences having race conditions, but overall it is pretty awesome.
Looking forward to moving to this model later today and will chime back in with results.
Because they take too long to run, and have an unpredictable latency and success rate. Seeing loading spinners and error messages in a visual interface is fine, but it would firmly put a natural language conversation in uncanny valley territory.
Regular chat already supports voice input, so might as well use that.
GPT-Live-1 is the first version of a new generation of models, and we believe the full-duplex architecture + delegation enables entirely new ways of human-AI interaction.
Hey! Bit of an unusual question maybe: if this stuff further exarcerbates the loneliness epidemic and atomization of society, will you be able to live with yourself you think? If you hear about teenagers only spending time with your chatbot in 5 years, will you feel some amount of personal responsibility or not? Always curious to hear you guys' perspective on that kind of stuff!
Can I connect it to my skills/tools? Example case, I have a knowledge base and event log in my company. I need a brainstorm companion, which will have full access to this knowledge, can converse about it and can invoke skills/tools available in the repo.
> GPT-Live-1 is the first version of a new generation of models, and we believe the full-duplex architecture + delegation enables entirely new ways of human-AI interaction.
Intuitively, decoupling semantic and audio-timeslice-space generations with coupled but distinct histories is right model architecture, not just for these sorts of assistants, but for domains like robotics too.
Are there any open source full duplex models that are out besides PersonaPlex? There was a chinese open one, maybe Fun Audio chat or something, that said it was going to release a full duplex version but I am not sure if it did.
My dream would be open source full duplex with function calling or some kind of rudimentary text output. PersonaPlex is still interesting although it was looking like we would need to fine tune it to handle outgoing or avoid going off the rails easily.
Working on this at https://duplexio.ai which will be open weights and free for non-commercial use. If you want to work on this send an email to anders@duplexio.ai
I want to know this too, as I’m hoping to fabricobble up a “smart speaker” that communicates with my local AI assistant. Right now we do everything via iMessage but it would be nice to be able to tell it to add things to my grocery list by voice while my hands are busy in the kitchen. Also would love if anyone has any advice on what microphone & speaker to pick out, was planning on just reusing a raspberry pi I’ve got around for the brain part.
if you don't find that then you could fake it with personaplex possibly but making another ASR/STT model just listen continuously and transcribe then send to an LLM with function calling. at least that would allow one direction easily.
I'm so mad that this might make me re-subscribe to ChatGPT. I wouldn't have believed how much I use the voice feature before LLMs and ChatGPT currently has the best voice interface. I think Grok's interface is the next best, then Claude.
Absolutely the same. Now that Fable is back, the Claude voice interface is... worth dealing with. The app mostly seems to have trouble recovering from networking issues which is jarring in a deep conversation.
But I don't think Fable or even Opus are ever used as the backend in voice mode. It has to respond in real time so I think in voice mode it's always using Sonnet.
I was missing the part where the grandmas are saying "I have absolutely no idea what I just said" :D
---
Besides that really nice demo I will give this a try, I tried some voice models before and the issue is I will ask a question, get a answer withing the next ~1 sentence and then the usual LLM bs follows which I just wanted to skip at that point
Does this model do better ignoring side conversations? That's the biggest hindrance to using ChatGPT's carplay feature is someone will say something, stopping ChatGPT from speaking or taking it in a different direction.
yes, it now has much better ability to understand the conversation and decide whether it should respond (and you can also tell it when it should respond)
Gemini live has been able to do this for over a year now. I can just activate it on my phone and it really works surprisingly well, especially the interruption. I've tested it with my 95 year old Dutch grandmother and it switched seamlessly between English and Dutch with her and handled her poor hearing very well, including her asking for repetition.
I'm a little surprised by how much OAI is playing catch up here.
One thing Simon pointed out: Usually live voice models are not as capable as the frontier ones. What makes this one special is that it can delegate the task to a frontier model while talking to you.
Gemini live is also pretty decent at computer vision stuff too. I've used it while working on my bike & car a few times, with my phone camera and it'll circle/highlight specific screws and parts. You can set your phone up on a tripod and it'll walk you through complete repairs for things.
I used it for double checking some stuff while helping my friend build his PC! I had it in tight spaces and it helped me verify some details around which nvme slot to use first.
This looks very cool. An AI that can listen and speak and handle tasks without breaking the flow of conversation would solve some big annoyances with current tools.
The concern is though as these get better will people struggle to distinguish these with real human connections?
people have been mistaking AI conversations with reality since the very first text-based models came into the public view with ChatGPT. i'm sure with each incremental improvement to outputs like this, though, more people will get convinced of its "humanity"
Every time things like this come up, I can't help but think of the ending of Inception.
It's less that you're convinced it's real and more that you no longer care if it is. "Feels real enough" is good enough.
I'm a technical user first, so I'm not sure if models have improved for RP the way they improved for applied STEM tasks and technical brainstorming. But if there is an improvement curve there, I wouldn't be surprised if this only grows in popularity.
this is excellent, I've been meaning to update my phone-dialer.apk "fake phone conversation" for when I'm trying to get out of a social sitation (not joking)
For the first actor - why does her accent change the longer she talks. It's like they had an "Estelle Costanza" dial that they started at zero and slowly rolled up to 8 or 9.
The demo video shows quite how rough around the edges this is....
Doesn't quite stop fast enough when you interrupt it. Can't find info quick enough so you have to change topic and then have it give you results later, etc.
This is a move in the right direction, but there is lots of engineering still to be done!
You're right, and to me it's refreshing to see a promo video that shows how the real product works, rather than a sanitized over-produced edit that takes out all the flaws.
I'm currently watching Better Call Saul for the first time with my wife. The fact that they used old ladies for the Ad, and that it is evident they are reading from a prompt (fake-ish feeling) gives me strong James McGill vibes haha. Hopefully the actresses were paid handsomely.
This solves my biggest annoyance with the current advanced voice: its speech getting interrupted by me setting yup or even background noise if loud enough
I've had some funny interactions with this issue. I sometimes use the voice mode when walking my dogs. I can confirm that ChatGPT responds positively to being told it's a "Good girl!"
Hoping to use this for natural conversation language learning. Previous iterations of the app kept correcting my words/grammar before it got to the model, causing issues with identifying mistakes in speech
I was going to post a comment on a related topic (I couldn't find in the announcement if this is English only or not), but would you mind expanding? I was thinking about doing something very similar, and yeah, if the model isn't hearing the mistakes I'm making, that would dramatically decrease it's usefulness.
Hopefully this also means it does better with "interruptions". I have used ChatGPT Voice in the car before and sometimes car/road noises will cause it to stop responding in the middle.
I used it to help me figure out how to turn off a feature in the rental car I was in (adaptive cruise control, I love it but snow blocked the sensor and I wanted just normal cruise control but couldn't figure it out while driving).
This kind of voice chat is awesome and I'll be even more excited when open models have this functionality. I'd love something like this paired with Home Assistant (assuming we ever get decent hardware).
You have always been able to pick between voices of many kinds though? Do you find them all sultry? From the British woman to the 17th century pirate soundalike?
If there was a voice that stripped away all the affectation I would be more likely to use it. It pretending to be human is extremely off-putting for me.
The new architecture makes sense, it seems many of the remaining problems like noise and interruptions are at the sound processing and integration level rather than at an architectural or model level now which makes for an exciting new era.
There doesn’t seem to be any indication whether this is available in the chat-got app nor is there any indication in the app that anything has changed. Anyone know how to actually try this?
GPT‑Live is rolling out now to ChatGPT users globally across iOS, Android, and ChatGPT.com. GPT‑Live‑1 will become the default model powering ChatGPT Voice for Go, Plus, and Pro users, and GPT‑Live‑1 mini will become the default for Free users.
We're beginning the rollout now, and will roll out in the next few days to ChatGPT users globally. Make sure to update to the latest version of the app!
I like this and felt like some of it was much more fluid; but was I alone in feeling like the interjected "uh-huh" or "yeah?" moments felt a little jarring?
Almost felt a bit *uncanny valley* for what "natural" conversation is supposed to be like. If the "uh huh" isn't timed correctly, it'll feel like a zoom call with lag.
Every second of the interaction is uncomfortable to me, but I also have extreme difficulty with video calls with humans. The latency completely breaks my mind.
Oh wow, I'd like this. Our current voice interactions with ChatGPT are on a 4o era model; really terrible. oAI has always been pretty cagey on the architecture of their end to end multimodal models. And RL has basically made them worse since launch. (Check the launch videos where the model sings, is more realtime, has accents, etc). I'd love to try a next gen version.
It would be great if we could have AI that wasn't trying to emulate a human. When it expresses emotion, we should see that as a bug that needs to be fixed.
The potential conversational dynamics of people telling each other "quiet!" after they pick up the habit from talking with AI will be interesting. It could lead to people being more assertive and thoughtful, or it could be contentious and rude.
Awesome that they've improved that aspect of voice chat, though.
GPT-Live does not support video at this point, but we're working hard to introduce it soon. In the meantime, our previous Advanced Voice Mode will continue to be available and supports video.
This had to land before there new device could be launched, i.e. human to AI full duplex interaction, Apple should be worried. They fumbled so hard on AI.
I am wondering for whom this new device even would be. Because phones work quite well for this use case and much more and everyone already has a phone.
I do not fully understand the complexity behind achieving full-duplex but I hope this sets the bar for Anthropic to follow. Turn-based simplex is yesterday.
It was probably the original intent of OpenAI working with Apple ... but clearly there was a reason why Apple ditched OpenAI for Google/Gemini models instead.
You can choose among 9 voices in the app, all newly refreshed for GPT-Live. If you meant whether it can detect multiple people, it can (like in the livestream), but not always perfect. Would love to hear your feedback once you try it.
Is it possible to create a "companion" of sorts with this model, using, say, an RPi and a speaker + microphone? Not for advanced scientific brainstorming, but for seniors who are often alone in their homes.
Try being a 90-year old with minimal social contact and nobody to talk to, wasting away in front of a TV blasting NewsMax or Fox News ... does that sound like Heaven or Hell?
While this is likely very useful to an enormous number of people, I suspect it will be even more useful for the elderly (if somehow it can be made accessible to them).
IIUC the literature, there is serious loss of functionality associated with lack of verbal interaction. People can say "they should just talk to more people" or "more people should make time for them" but the fact of the matter is that it doesn't happen, and if this helps terrific.
Absolutely can't wait to try this for language practice. The advanced voice mode is great but ultimately just doesn't work that well and doesn't have the feel of a natural conversation.
Basically have an older lady (not their target audience) blatantly reading a teleprompter.
Why are they going after this audience? Retired people have no use for delegated tasks or information. They also are the least likely to use it and not get frustrated.
I for one am greatly looking forward to the day these kind of voice models can be run locally. It seems like the gap between open-weight and frontier is way larger for voice models than coding/language models.
watched the live translation video very impressive
Seems like a shift from previous voice models where it sequentially processes voice to text then feeds it to LLM and then back which cant escape the clunky lag
not sure how pipecat stands now, gpt live seems like it takes audio tokens and does inference on it directly
They most definitely did not solve real time translation yet. The French in the video is barely understandable, both the translation and the pronunciation are the quality of an American who hasn't used French since high-school.
I had preview access to this one for a few weeks. It's very good. I had one conversation that lasted a full hour while I was walking the dog, got some good brainstorming done against one of my projects.
The best feature is that it can delegate questions out to GPT-5.5 in the background, so you're no longer restricted to a voice model that's several years behind the frontier.
I did report a fun bug with it though: it was interrupting me and laughing at my (not really intended as) jokes while I was still talking! They seem to have clamped that behavior down thankfully, it felt a bit rude and condescending.
I highly recommend simply enjoying the walk.
Seems a bit knee jerk. I go on walks and bike rides all the time. A couple of times I’ve used voice mode and it’s been interesting. I could have listened to music or a podcast, listen to YouTube or just unplug. But every walk is different.
If you’re uncomfortable with this new world, and I’m sure I am even as I participate, you could tell us more about that?
I recommend against providing unsolicited life advice
Isn’t that a life advice?
That's anti life-advice
Is it responsive to personality settings? I actively don't want fake AI girlfriend, but I do get a ton of value out of voice mode. Looking forward to trying this but hoping it's not a creepy overdone mess (like Sesame). Expectations are they'll keep doubling down on fake AI girlfriend approach because the thing I want probably wouldn't drive engagement anywhere nearly as well
I too found this with their previous attempts.
I have my Chat personality settings stripped right down to no-fluff. I'd want voice to be more akin to the Star Trek computer, and less akin to as you said an AI friend, but previously it was tuned too personable/friend-like.
Star Trek computer voice model is something I have yet to encounter, and I've looked repeatedly :) It's not about a specific voice, it's the fact they managed to capture "I am a utility" perfectly in the voice. Our modern friends do not want to be thought of as a utility, but to engender trust and agency all of their own and that's a huge problem for me.
> "The best feature is that it can delegate questions out to GPT-5.5 in the background, so you're no longer restricted to a voice model that's several years behind the frontier."
Wow. That's exactly what I hoped they would do.
This issue has held me back from using ChatGPT's voice mode as much as I otherwise would have, because I also use it for brainstorming while commuting, exercising, etc., and don't want it to feel stuck in the past.
Is it a dumb-down version of GPT like the current voice model? At least in french, I find the current GPT voice mode to be useless, to the point I only use the dication mode. I would ask a question and it would answer something along "That's a interesting question. I can help you with that. Anything you want to know about X?" I would ask again and it would answer the same kind of non answer.
Click through to the link, the answer is no it uses the latest gpt models now.
> the answer is no it uses the latest gpt models now.
Actually it says it _can_ delegate to the latest models. Seems reasonable to ask how the voice model does when it doesn't delegate (or while waiting for the delegated answer).
But does it generate good pelicans?
Have you compared it to grok in teslas?
Wait, you can talk to Teslas now? How did I miss thiS? Can I get a red led bar and basically have a KITT?
Cool, was this conversation through the chatgpt app?
Yes, ChatGPT on iPhone.
Thank you for testing and the feedback Simon!
About 2 years ago I used to have conversations with GPT while walking the dog. It really emphasized the need to think before you speak, but you had to think fast before it hung up on you.
Sam, is that you?
This is the opposite direction AI should be going. Human relationships are the most valuable thing we have, and so, naturally, technology seeks to intermediate and now replace them.
I'm not Catholic, but this podcast presents a very interesting argument against talking to AI as if they were human: https://newpolity.com/podcasts-hub/debate-chatbots
Most of what AI does is already in wrong direction. Not just human-to-human interaction, it took away thinking, creative work, sensory perception (glasses) and responses. People call it as helping humans, but I call it as sucking away the "human-ness" from humans.
After the damage is done, the mega corps would simply shrug and will say "Well, we were just responding to our business competition" The business knows no human-ness, because it is not a human. Businesses and machines are creatures that see humans as their fodder. And humans created these, assuming it is progress, to have businesses and machines. We call it progress because it required our mind power and it helped us to dominate other species. Dolphins are laughing at us.
I don't understand this line of reasoning. How are you hindered from doing any of those things? What part of "AI can now do X" makes it so you can't also do X?
We're yet to see how this plays out, but a competing business model for creative work is emerging, where it's delegated to chatbots. Naturally, this would result in less creative work for humans.
I'm no longer writing code from scratch, as I used to do before. So, very soon, it will be "AI can now do X" makes it so I can't also do X? Same with many creative works. Radio music already sounds so plastic. I lost interest in crafting my text drafts because I can just dump some ugly text and get it refined by AI.
Hasnt this already been thoroughly discussed? Your ability to do X degrades as you offload it more frequently, eventually to the point that you can no longer even vet the quality of the output.
I think the parent is saying now that that is attempting to be applied to "creativity" directly, as opposed to something like a shift of medium, that it threatens many peoples' ability to maintain creative capabilities.
Anecdotally I've already experienced this at work where post-AI we had a junior completely stagnate and a senior with over a decade of experience in the bay atrophy to the point that he had to be let go.
[delayed]
Yes, every minute you spend texting or talking to a chatbot is a minute that you'd have spent talking to another human beings. Literally the only important thing in life, the basis of all value, the formation of self-identity, comes from communication with other human beings.
Let's not overdramatize, though. I'm not in need, or even in mood to talk to fellow humans every minute, so time spent with a clanker is not necessary taken from my human relations budget
This is such a poor mischaracterization of OP that I actually started agreeing with OP more.
> every minute you spend texting or talking to a chatbot is a minute that you'd have spent talking to another human beings
Very blatantly and obviously not though???
Such radical carbon chauvinism is ontologically evil. May those who hold this view reincarnate as durian fruits or cockroaches.
I'm sure you'd have no problem in solitary confinement. Why not make life easy?
Just saying: those are also carbon-based lifeforms :D
The implication that durians and roaches are somehow ontologically subaltern is a worse chauvinism.
Every new release I'm convinced we are just realizing the Knowledge Navigator.
https://www.youtube.com/watch?v=umJsITGzXd0
Isn’t voice I difference in degree rather than in kind? I definitely talk to AI as if it were human (one might say the UX of AI is to emulate a human). And a large portion of my interaction with humans is via text, for example, this post!
Right, I think the challenge is that LLMs are essentially language-based, which is itself a very convenient interface. Covering the maximally humanistic default interface with something more mechanical is like tying your own shoelaces together, but it would protect us from the psychological hijacking we're so prone to when interacting with these machines.
Again, we're paying for the fact that Sam Altman didn't understand the movie her
I see it more as a way technology could be abused rather than an inherent flaw in the technology itself. If you start to replace human interaction with chatbot interaction, that's bad, but there's nothing wrong with using a human-like chatbot in moderation. So many other types of technology are fine in moderation but can be abused in a human-interaction-replacing way: television, social media, video games, etc.
Yes, but design nudges use in one direction or another. Or, as McLuhan said, "the medium is the message."
Just give it a scifi robot voice and pretend you're on a spaceship
I'm sorry, Dave. I'm afraid I can't do that.
This is actually not a bad option
But a lot of humans can't engage in meaningful technical conversations... So why not use a tool for that?
Where do you draw the line on how good human-machine interfaces should be? I'm sure this model could be a convenience for many, and while it may be social for some, I am not sure it would substitute existing human interaction for those users.
Besides, I do not think there is anything inherently immoral with not being social, or not having the ability to be. Consider for example people who do not naturally have the social network to interact with people they want to (e.g. some gifted children).
I am not convinced this model has enough empathy to satisfy most users on an emotional level. A bond is not merely an exchange of words, but prolonged and deep contemplation of the other being. We cannot introspect into these machines, and they certainly cannot yet do the same to us, yet.
How would talking to an AI as if it were not human sound? You can probably set your system prompt to insert “beep boop” between sentences and make it refer to itself as “Cybertron9000 Personal Computing Device” if that’s what you like. Is that an improvement? Or are you against voice computer interfaces altogether?
The Star Trek computer doesn't feel uncannily human, that would be a good starting point.
I don't know the answer. But a robotic voice is probably not a bad idea — just having a reminder that the thing you're talking to is not actually anything like you. If you want to go full send, you could have the LLM generate a clickable interface on demand so you could interact with it as a machine. Voice/computer interfaces are obviously useful, especially for disabled folks. But the ones that existed in the past didn't pretend to laugh at your non jokes or imitate vocal fry.
> This is the opposite direction AI should be going.
There is no moral obligation, in any domain, to refuse to make a product that adults, with full informed consent, find useful and purchase. Who are you to say you know better than the market?
I disagree. I think you absolutely have a moral obligation to consider the impacts of your product.
> Who are you to say you know better than the market?
You really don't think the scientists & engineers making these tools know some things better than the market?
Market conditions are not a moral standard either, nor do they represent any cohesive one in particular. Not sure why you're contrasting their opinion with this, it's literally no better.
What I’m missing from this announcement is the capability to use connectors and tools. I don’t really get it - NONE of the frontier assistants can use tools / connectors while in voice mode - Claude, ChatGPT, Gemini, Grok. It seems so obvious: I want to be able to research stuff, pull up documents, jot down notes and do productive work while I’m talking to it, and not end voice mode whenever I need to connect to an app or service.
It’s weird. The old Claude voice mode WAS able to use tools but when they revamped it, it lost that capability and is now pinned to Haiku :(
So, yay for finally a voice mode that’s powered by a frontier model and hopefully as good as Grok voice, but sad to still not see tool use while in voice mode.
(I haven’t tried it yet, only read the announcement)
A big part of this announcement does seem to be _delegation_ in the background; they give the example of web search but that could be any tool. I haven't tried it yet either but sounds like they've found a reasonable UX that mixes that sort of high-and-variable latency tool calling (potentially with agentic loops) with a continuously speaking live voice.
I could see it relating to tools having unpredictable latency but if they already do background hand off to 5.5 then it seems like they could just enable it within that context.
I’ve been using gpt-realtime-1 for my personal assistant that runs my company and plans my day. And it works pretty well, even makes tool calls and all that.
But the multi modal stuff has resulted in a lot of debugging with weird events and message and audio sequences having race conditions, but overall it is pretty awesome.
Looking forward to moving to this model later today and will chime back in with results.
Because they take too long to run, and have an unpredictable latency and success rate. Seeing loading spinners and error messages in a visual interface is fine, but it would firmly put a natural language conversation in uncanny valley territory.
Regular chat already supports voice input, so might as well use that.
Last I tried this is exposed via their sdk and can be built.
(Atty from OpenAI here)
GPT-Live-1 is the first version of a new generation of models, and we believe the full-duplex architecture + delegation enables entirely new ways of human-AI interaction.
Would love to hear your feedback!
Hey! Bit of an unusual question maybe: if this stuff further exarcerbates the loneliness epidemic and atomization of society, will you be able to live with yourself you think? If you hear about teenagers only spending time with your chatbot in 5 years, will you feel some amount of personal responsibility or not? Always curious to hear you guys' perspective on that kind of stuff!
Can I connect it to my skills/tools? Example case, I have a knowledge base and event log in my company. I need a brainstorm companion, which will have full access to this knowledge, can converse about it and can invoke skills/tools available in the repo.
- As models get better, have you considered some kind of filter or particular cadence to serve as a reminder that the user is not talking to a human?
- The videos felt scripted and dishonest
Can it sing? Is it an end to end multimodal model?
Can it delegate to just one agent at a time or can it spawn multiple subagents for different tasks?
Does video/image input still work with these duplex models?
When is it rolling out? Currently on ChatGPT Pro but not seeing it yet?
What made you to try again?
> GPT-Live-1 is the first version of a new generation of models, and we believe the full-duplex architecture + delegation enables entirely new ways of human-AI interaction.
Awesome. Are you guys able to share anything about the model architecture? I've been interested lately in split-transformer RVQ-based conversational agents, e.g. via stuff like https://arxiv.org/abs/2412.10208 (ResGen) and https://arxiv.org/abs/2603.18090 (MOSS-ITT) and of course Moshi (https://arxiv.org/pdf/2410.00037).
Intuitively, decoupling semantic and audio-timeslice-space generations with coupled but distinct histories is right model architecture, not just for these sorts of assistants, but for domains like robotics too.
Are you still using LiveKit for the back-and-forth architecture
Write up about the architecture is here https://openai.com/index/delivering-low-latency-voice-ai-at-...
[delayed]
I'm crying, that guy who tries to get it to count to 100 may actually have a chance
I looks like they took inspiration from Thinking Machines - http://thinkingmachines.ai/blog/interaction-models/
Are there any open source full duplex models that are out besides PersonaPlex? There was a chinese open one, maybe Fun Audio chat or something, that said it was going to release a full duplex version but I am not sure if it did.
My dream would be open source full duplex with function calling or some kind of rudimentary text output. PersonaPlex is still interesting although it was looking like we would need to fine tune it to handle outgoing or avoid going off the rails easily.
Working on this at https://duplexio.ai which will be open weights and free for non-commercial use. If you want to work on this send an email to anders@duplexio.ai
I want to know this too, as I’m hoping to fabricobble up a “smart speaker” that communicates with my local AI assistant. Right now we do everything via iMessage but it would be nice to be able to tell it to add things to my grocery list by voice while my hands are busy in the kitchen. Also would love if anyone has any advice on what microphone & speaker to pick out, was planning on just reusing a raspberry pi I’ve got around for the brain part.
If you're in the Home Assistant ecosystem, I'm intrigued by their voice hw
if you don't find that then you could fake it with personaplex possibly but making another ASR/STT model just listen continuously and transcribe then send to an LLM with function calling. at least that would allow one direction easily.
I'm so mad that this might make me re-subscribe to ChatGPT. I wouldn't have believed how much I use the voice feature before LLMs and ChatGPT currently has the best voice interface. I think Grok's interface is the next best, then Claude.
Same. I might switch back to ChatGPT from Gemini because I use the voice feature all the time.
One of my favorite use cases is talking with it while driving on random topics and learning about them.
Absolutely the same. Now that Fable is back, the Claude voice interface is... worth dealing with. The app mostly seems to have trouble recovering from networking issues which is jarring in a deep conversation.
But I don't think Fable or even Opus are ever used as the backend in voice mode. It has to respond in real time so I think in voice mode it's always using Sonnet.
That makes sense and had never occurred to me!
I just asked and Claude says it's Haiku.
I was missing the part where the grandmas are saying "I have absolutely no idea what I just said" :D --- Besides that really nice demo I will give this a try, I tried some voice models before and the issue is I will ask a question, get a answer withing the next ~1 sentence and then the usual LLM bs follows which I just wanted to skip at that point
Does this model do better ignoring side conversations? That's the biggest hindrance to using ChatGPT's carplay feature is someone will say something, stopping ChatGPT from speaking or taking it in a different direction.
(Atty from OpenAI here)
Yes! GPT-Live is much better at ignoring background noise, including other people speaking. Not perfect, but you should feel a big difference.
yes, it now has much better ability to understand the conversation and decide whether it should respond (and you can also tell it when it should respond)
Hello Justin! Mr webrtc himself and the infamous AIM 5.0
Gemini live has been able to do this for over a year now. I can just activate it on my phone and it really works surprisingly well, especially the interruption. I've tested it with my 95 year old Dutch grandmother and it switched seamlessly between English and Dutch with her and handled her poor hearing very well, including her asking for repetition.
I'm a little surprised by how much OAI is playing catch up here.
ChatGPT has had this for over a year too. This is a better model.
One thing Simon pointed out: Usually live voice models are not as capable as the frontier ones. What makes this one special is that it can delegate the task to a frontier model while talking to you.
Gemini live is also pretty decent at computer vision stuff too. I've used it while working on my bike & car a few times, with my phone camera and it'll circle/highlight specific screws and parts. You can set your phone up on a tripod and it'll walk you through complete repairs for things.
I used it for double checking some stuff while helping my friend build his PC! I had it in tight spaces and it helped me verify some details around which nvme slot to use first.
Is gemini live the same app I have on my oneplus 15? I don't think that's full duplex?
it is not full duplex. I think this GPT-Live thing is the first full duplex speech to speech model
ehh, no? definitely not.
https://kyutai.org/blog/2024-07-03-meet-moshi/
What do you mean? OpenAI has had a real-time voice model since August of last year. This is a new model with better performance.
This looks very cool. An AI that can listen and speak and handle tasks without breaking the flow of conversation would solve some big annoyances with current tools.
The concern is though as these get better will people struggle to distinguish these with real human connections?
people have been mistaking AI conversations with reality since the very first text-based models came into the public view with ChatGPT. i'm sure with each incremental improvement to outputs like this, though, more people will get convinced of its "humanity"
(see https://www.reddit.com/r/MyBoyfriendIsAI/)
Every time things like this come up, I can't help but think of the ending of Inception.
It's less that you're convinced it's real and more that you no longer care if it is. "Feels real enough" is good enough.
I'm a technical user first, so I'm not sure if models have improved for RP the way they improved for applied STEM tasks and technical brainstorming. But if there is an improvement curve there, I wouldn't be surprised if this only grows in popularity.
https://en.wikipedia.org/wiki/ELIZA
this is excellent, I've been meaning to update my phone-dialer.apk "fake phone conversation" for when I'm trying to get out of a social sitation (not joking)
Some brilliant marketing really, the grandma test.
For the first actor - why does her accent change the longer she talks. It's like they had an "Estelle Costanza" dial that they started at zero and slowly rolled up to 8 or 9.
The demo video shows quite how rough around the edges this is....
Doesn't quite stop fast enough when you interrupt it. Can't find info quick enough so you have to change topic and then have it give you results later, etc.
This is a move in the right direction, but there is lots of engineering still to be done!
You're right, and to me it's refreshing to see a promo video that shows how the real product works, rather than a sanitized over-produced edit that takes out all the flaws.
I'm currently watching Better Call Saul for the first time with my wife. The fact that they used old ladies for the Ad, and that it is evident they are reading from a prompt (fake-ish feeling) gives me strong James McGill vibes haha. Hopefully the actresses were paid handsomely.
This solves my biggest annoyance with the current advanced voice: its speech getting interrupted by me setting yup or even background noise if loud enough
Now it consistently interrupts you
Yeah, looking at the waveforms from those sample conversations it looks like it's happening at the exact same interval every time
I've had some funny interactions with this issue. I sometimes use the voice mode when walking my dogs. I can confirm that ChatGPT responds positively to being told it's a "Good girl!"
Hoping to use this for natural conversation language learning. Previous iterations of the app kept correcting my words/grammar before it got to the model, causing issues with identifying mistakes in speech
I was going to post a comment on a related topic (I couldn't find in the announcement if this is English only or not), but would you mind expanding? I was thinking about doing something very similar, and yeah, if the model isn't hearing the mistakes I'm making, that would dramatically decrease it's usefulness.
Hopefully this also means it does better with "interruptions". I have used ChatGPT Voice in the car before and sometimes car/road noises will cause it to stop responding in the middle.
I used it to help me figure out how to turn off a feature in the rental car I was in (adaptive cruise control, I love it but snow blocked the sensor and I wanted just normal cruise control but couldn't figure it out while driving).
This kind of voice chat is awesome and I'll be even more excited when open models have this functionality. I'd love something like this paired with Home Assistant (assuming we ever get decent hardware).
(Atty from OpenAI here)
Yes, GPT-Live is much better at ignoring background noise! I use it every day in the car via our CarPlay integration.
I was hopeful that they avoided the well known sultry voice this go around, but alas. There is little hope for these companies.
The full duplex is awesome, and the feedback that it is getting what you're saying is ok, but in some of the demos was a little overkill.
I'll agree that using the "Golden Girls" was at least more entertaining than the usual pitch.
You have always been able to pick between voices of many kinds though? Do you find them all sultry? From the British woman to the 17th century pirate soundalike?
If there was a voice that stripped away all the affectation I would be more likely to use it. It pretending to be human is extremely off-putting for me.
tell it "speak like a robot without affectation or emotion" in your custom instructions
"I was hopeful that they avoided the well known sultry voice this go around, but alas"
Why do you care? You can select other voices. Why do you need to control others?
What is at the root of your need for domination?
They are stealing/imitating someone elses brand.
Amping up an emotional connection is great for business.
Why do you think THEY need to dominate via an emotional connection?
This feels so dehumanising.
"i'm starving."
sounds cynical in my ears. energy demand of these toys will cause many problems, people elsewhere starving being one of them.
The new architecture makes sense, it seems many of the remaining problems like noise and interruptions are at the sound processing and integration level rather than at an architectural or model level now which makes for an exciting new era.
When GPT-Live in Codex, so i can walk the dog while shipping?
There doesn’t seem to be any indication whether this is available in the chat-got app nor is there any indication in the app that anything has changed. Anyone know how to actually try this?
GPT‑Live is rolling out now to ChatGPT users globally across iOS, Android, and ChatGPT.com. GPT‑Live‑1 will become the default model powering ChatGPT Voice for Go, Plus, and Pro users, and GPT‑Live‑1 mini will become the default for Free users.
(Atty from OpenAI here)
We're beginning the rollout now, and will roll out in the next few days to ChatGPT users globally. Make sure to update to the latest version of the app!
Read the post
So more AI psychoses coming.
I'm at the point where pricing is the first thing I look for in announcements like this.
I like this and felt like some of it was much more fluid; but was I alone in feeling like the interjected "uh-huh" or "yeah?" moments felt a little jarring?
Almost felt a bit *uncanny valley* for what "natural" conversation is supposed to be like. If the "uh huh" isn't timed correctly, it'll feel like a zoom call with lag.
Every second of the interaction is uncomfortable to me, but I also have extreme difficulty with video calls with humans. The latency completely breaks my mind.
I agree. Many times those little interjections don't feel natural. It's impressive, but there's still a lot of room for improvement.
Can a model like this critique your accent/pronunciation? That would be cool.
Oh wow, I'd like this. Our current voice interactions with ChatGPT are on a 4o era model; really terrible. oAI has always been pretty cagey on the architecture of their end to end multimodal models. And RL has basically made them worse since launch. (Check the launch videos where the model sings, is more realtime, has accents, etc). I'd love to try a next gen version.
It would be great if we could have AI that wasn't trying to emulate a human. When it expresses emotion, we should see that as a bug that needs to be fixed.
The potential conversational dynamics of people telling each other "quiet!" after they pick up the habit from talking with AI will be interesting. It could lead to people being more assertive and thoughtful, or it could be contentious and rude.
Awesome that they've improved that aspect of voice chat, though.
Very cool. I thought the agent came in a little to hot at 1:03. I wonder how it decides when to jump in.
I'm very eager to test this for brainstorming!
One thing I noticed is that we lost vision feature for some reason on the live chat?
This was an extremely useful feature. Not sure if it’s a regional thing or that they just removed that from the current live chat.
I imagine it will be even more useful with this new version.
(Atty from OpenAI here)
GPT-Live does not support video at this point, but we're working hard to introduce it soon. In the meantime, our previous Advanced Voice Mode will continue to be available and supports video.
This had to land before there new device could be launched, i.e. human to AI full duplex interaction, Apple should be worried. They fumbled so hard on AI.
I am wondering for whom this new device even would be. Because phones work quite well for this use case and much more and everyone already has a phone.
Can a phone go get you a coffee?
are these human actors or are is the whole demo AI generated?
It sounds like they've switched to a "native audio" model which if I understand right is what Gemini has had for quite a while?
I do not fully understand the complexity behind achieving full-duplex but I hope this sets the bar for Anthropic to follow. Turn-based simplex is yesterday.
This is what Siri should have been.
It was probably the original intent of OpenAI working with Apple ... but clearly there was a reason why Apple ditched OpenAI for Google/Gemini models instead.
>and linked parents may be notified in higher-risk situations involving signs of potential self-harm or suicidal intent.
This is an abuse of user trust and violates people's privacy.
Great video by the way. Extremely good!
One piece of feedback on that, could not get audio sync in Firefox. Chrome worked fine.
Oh gosh. I was watching the video and thought it is a live stream. I just noticed when it restarted.
Very cool. Not cool bringing Brazil’s loss to Norway again. We're already devastated. No need to keep beating someone on the ground. :(
Does this support more than one user voice? Or, are there plans for this? I did not see that mentioned in the announcement.
(Atty from OpenAI here)
You can choose among 9 voices in the app, all newly refreshed for GPT-Live. If you meant whether it can detect multiple people, it can (like in the livestream), but not always perfect. Would love to hear your feedback once you try it.
Any plans to add more international voices and dialects?
Is it possible to create a "companion" of sorts with this model, using, say, an RPi and a speaker + microphone? Not for advanced scientific brainstorming, but for seniors who are often alone in their homes.
If this is my idea of hell, I can't imagine what it'd be like for a 90 year old
Try being a 90-year old with minimal social contact and nobody to talk to, wasting away in front of a TV blasting NewsMax or Fox News ... does that sound like Heaven or Hell?
Fantastic to see this. I use voice a lot. It’s not quite lived up to my expectations but I think this gets much closer.
While this is likely very useful to an enormous number of people, I suspect it will be even more useful for the elderly (if somehow it can be made accessible to them).
IIUC the literature, there is serious loss of functionality associated with lack of verbal interaction. People can say "they should just talk to more people" or "more people should make time for them" but the fact of the matter is that it doesn't happen, and if this helps terrific.
Absolutely can't wait to try this for language practice. The advanced voice mode is great but ultimately just doesn't work that well and doesn't have the feel of a natural conversation.
I reaaaaaally hope we have an option to disable those random ums and ahhs that interrupt for no reason. :|
I was surprised all that made it into the demo video. Did not seem cool. Is that "active listening" or something?
Feel like the intro video is very odd.
Basically have an older lady (not their target audience) blatantly reading a teleprompter.
Why are they going after this audience? Retired people have no use for delegated tasks or information. They also are the least likely to use it and not get frustrated.
to show that anybody could use it - even older people who are less tech experts.
they are trying to expand beyond their tech audience.
GPT-Live being developed in california and being an over-active listener...
Is this website heavily vibe coded. I tried to select text and things went black.
https://imgur.com/a/ABGWRTO
Definitely in the right direction in terms of architecture. However those "hmmm" "uh huh" interjected in the demo are pretty awful.
Any pricing announced yet?
I for one am greatly looking forward to the day these kind of voice models can be run locally. It seems like the gap between open-weight and frontier is way larger for voice models than coding/language models.
I want this in my ear when I'm talking to people, so I can carry a real conversation.
how about api access?
(Atty from OpenAI here)
Coming soon! Sign up to be notified here: https://openai.com/form/gpt-live-1-in-the-api/
watched the live translation video very impressive
Seems like a shift from previous voice models where it sequentially processes voice to text then feeds it to LLM and then back which cant escape the clunky lag
not sure how pipecat stands now, gpt live seems like it takes audio tokens and does inference on it directly
>GPT‑Live can show it’s paying attention with phrases like “mhmm” or “yeah” [...]
Nooooooo!
With this, human translators have been totally and absolutely a solved problem with this version of real time translation.
This time is the most natural version that exists and it is a natural as a conversation.
To Downvoters: Why aren't you feeling the AGI?
They most definitely did not solve real time translation yet. The French in the video is barely understandable, both the translation and the pronunciation are the quality of an American who hasn't used French since high-school.
“I don’t translate, I interpret” - Ahmed the best interpreter there ever was
Why aren't you feeling the operating loss? :(