This appeared last week:
What Apple Intelligence means for you
Though the “where” and “when” of Apple’s new AI system are still a mystery, we do know lot about the “who”, “what” and “why”.
John Davidson Columnist
On
Monday, US time, Apple
announced dozens of new AI features for its phones, tablets, virtual-reality
headsets and PCs, including a long-awaited overhaul of its Siri voice
assistant, new ways to handle incoming emails and write outgoing ones, and new
ways to generate, find and manage photos on Apple devices.
Many of the headline announcements, including the Siri overhaul and the deal Apple struck with OpenAI, which may see OpenAI’s ChatGPT occasionally popping up in response to Siri requests, fall under an AI system Apple calls “Apple Intelligence”.
Technically, Apple has only announced Apple Intelligence for the USA (or, to be even more pedantic, for “US English”), promising to start rolling out some AI features in the American autumn as part of the next major software update to the iPhone, iPad and Mac.
Apple has yet to reveal when, or even if, Apple Intelligence will come to Australia. The only public (or indeed private) statement is that “additional languages will come over the course of the next year” after the US English launch. So it could be 2025 before it rolls out here.
While it’s possible Australian users will be able to get earlier access to the features simply by going into the settings menus of their device and switching its language to US English, the geographic limit appears to be about more than just language (see private cloud compute below).
Early indications are Apple may enforce the limit through other means, too, such as checking the home address of the Apple iCloud account users will need to access the services. All is not lost, however.
Apple Intelligence versus machine learning
Apple Intelligence refers primarily to the set of services that will require up-to-date hardware to run: the stuff that will need a Mac or iPad running an Apple Silicon chipset, or an iPhone greater than, or equal to, the iPhone 15 Pro.
Many of the interesting new AI features don’t fall under that umbrella. The enhancements to Apple’s browser Safari, for instance, which will allow web users to summarise and even index the content of websites, come under a different category that Apple generally refers to as “machine learning” rather than “Apple Intelligence”.
(In numerous background briefings I’ve had in the Apple Park headquarters in Cupertino this week, that’s the pattern that has emerged. If Apple is calling it “machine learning”, you won’t need a new iPhone for it, and it won’t be initially limited to the USA. Technically, Apple Intelligence could accurately be described as machine learning, too, but as you’ll see in a moment, it’s a much bigger and more ambitious framework than adding a bit of ML to a single app.)
The new calculator for the iPad falls under “machine learning” and should be available when iPadOS 18 is launched in September or October
Another example of this machine learning/Apple Intelligence divide is the Photos app that appears on iPhones, iPads, Macs and, from July 12, on Apple’s Vision Pro.
It’s receiving a significant update, with new features such as the ability to organise photos according to the grouping of people and pets in them. All the photos with only you and your dog will be separated out into one group, while photos with you, your dog and your mother will have another group.
That’s a machine learning feature that every Apple user will get.
But click on one of those photos to edit it, and only customers who qualify for Apple Intelligence will see a button for a new “Clean Up” tool that scans an image, finds people who don’t belong in the photo, and deletes them.
In addition to the machine learning and Apple Intelligence features, Apple has also announced scores of other new features in devices that have nothing to do with AI or ML.
iPhone users will be able to type in messages and schedule their delivery up to two weeks in advance, so they can send birthday wishes when they remember to do so, rather than risk forgetting to do so on the day itself.
(A handy tip coming from Apple insiders who have been testing that feature for months is to schedule the message for a random time – 8.07am rather than 8.00am, for example – so the recipient doesn’t realise they’re not top of mind.)
It’s fair to say, however, that the most exciting features fall under Apple Intelligence, and that the wait for them to arrive could feel like a long one.
How it will work
At the core of Apple Intelligence is a database called the semantic index.
Whenever an iPhone, iPad or Mac gets a new message or takes a new photo, or whenever the user creates a new document, the same mechanism that indexes the file so it can be found by Apple’s existing Spotlight search system will also send it off for processing by an AI indexer, which will extract the meaning of that document and store the meaning in the semantic index.
Where Spotlight might index, say, a new iPhone photo by the date and time it was taken, the semantic indexer might identify the people in the photo, match them up to people in the phone’s contacts database, and index them by the relationship they have to the iPhone’s owner.
Semantic indexes are protected by the same hardware-based “secure enclave” system that Apple uses to secure ultra-sensitive information such as passwords, and Apple insists they will never leave the device they’re created on.
Even when an Apple customer has multiple devices sharing data, the semantic index will never be shared between them. It will be created and maintained separately on each device.
Another database sitting on an Apple Intelligence device will contain a list of what Apple calls “app intents”.
It’s like an app store, except instead of holding lists of apps, it holds a list of the functions that the installed apps on the device can perform in an automated way.
If, say, the iPad version of Adobe’s Photoshop was able to take an image saved in the PNG format, and automatically export it as a JPEG file, Adobe would be able to inform the iPad of that function by calling up the app intent system when Photoshop was first installed on the iPad.
In addition to all of that, every Apple Intelligence device will contain dozens of different AI language models for analysing and generating text, as well as “diffusion” models for analysing and generating images.
Compared to the large language and diffusion models used in the cloud by chatbots such as OpenAI’s ChatGPT and Google’s Gemini, Apple’s models will be smaller and more numerous, each of them fine-tuned to perform a specialised task using as little processing power as possible.
(Samsung’s latest Galaxy phones and Google’s latest Pixel phones employ the same architecture: they contain numerous small models, which swap in and out of the phone’s AI processor depending on the task.)
Now, sitting above all of this will be a software layer known as an “orchestrator”, and it’s here that the fun begins.
The orchestrator takes incoming requests from the user, decides what data from the semantic index is needed to fulfil those requests, which model is best suited for the task, looks into the list of intents to see what apps might be able to perform functions useful to that task, and sets the wheels in motion getting the job done.
Using Apple Intelligence, it might be possible, for instance, for a user to ask Siri to find a picture of an uncle (identified using the semantic index), put a party hat on his head (using a diffusion model), convert that picture to a jpeg file (using the Photoshop function listed in the app intent database), and email it him (address from the semantic index, email function found in the app intent database) on his birthday (from the calendar, via the semantic index).
Private Cloud Compute
Some jobs, however, will require language or diffusion models that are simply too large, or require too much processing power, to be stored on the iPhone, iPad or Mac.
To run such models, Apple is filling data warehouses around the world with custom AI servers it’s building using the Apple Silicon chips that go into Macs and iPads, and running a highly secure version of the iPhone’s operating system, iOS.
That rollout is starting in the US, and it’s part of the reason Apple Intelligence will be only in US English when it first appears.
These “Private Cloud Compute” data warehouses are integral to Apple Intelligence, since they’re the only way Apple can send data from the user’s very private semantic index to the cloud, without breaking Apple’s promise that Apple Intelligence is private and secure.
When the orchestrator on a device running Apple Intelligence decides a job is too big for the models stored on the device itself, it may decide to bundle up the job, together with context data from the semantic index, and upload it to Private Cloud Compute.
There, the model will process the request, return the results, and then delete any record of the request.
(Unusually for a cloud server, PCC computers have no storage devices whatsoever, so not only can they not save semantic index data, they can’t even save the error logs and other logs that administrators usually rely on to manage server farms. It’s an ambitious strategy designed to assuage concerns about AI privacy.)
OpenAI
There is a final scenario, however, that has led to some controversy with the likes of Elon Musk.
If
the orchestrator decides that the job is too big even for Private Cloud
Compute, it will put some warning messages on the screen telling the user
they’re about to leave Apple airspace
and fly off into the unknown.
If the user agrees to throw caution to the wind, the orchestrator will bundle up the job and send it to a third-party system for processing.
On Monday, the only third-party Apple had inked a deal with was OpenAI, though officials said they expected to get Google on board, too, allowing jobs to be sent to Gemini.
It’s likely, however, that OpenAI and Google are both stopgap measures, until Apple’s own Personal Cloud Compute systems have models that can handle generic generative AI tasks themselves.
The real point of giving the orchestrator the last-resort option of going to a third party is to handle highly specialised AI tasks, such as medical models that take images of skin shot on an iPhone, and look for signs of skin cancer.
The very moment Apple gets its own Personal Cloud Compute system to a level good enough to handle more generic generative AI queries, it’s nearly certain it will decide it’s cheaper, more private and better PR for the orchestrator to take that option instead of using OpenAI or Google.
The only thing uncertain is how long it will take Apple to get its models to that level. But before we even get to worry about that uncertainty, we have another one to deal with.
Just when is any of it coming here?
John Davidson attended Apple’s Worldwide Developer Conference as a guest of Apple.
Here is the link:
https://www.afr.com/technology/what-apple-intelligence-means-for-you-20240613-p5jlie
I have to say I have read this through a couple of times and I am not sure just where it is all heading. I suspect it will be a while before that is clear as well as when it is going to reach good old OZ
I suspect this is more of the revolution I have been chatting about in the last few months.
These newer technologies are seeming to have a life all of their own as they advance and I have to say I am still not sure what the actual destination is!
As with all things time will tell I guess….
David.
No comments:
Post a Comment