Android’s Computer Control characteristic is a greater model of the Rabbit R1

For a quick interval final yr, it appeared that AI-powered devices just like the Rabbit R1 have been going to be the subsequent massive factor. People have been fascinated by the concept of changing their smartphones with tiny wearable containers they may speak to, however sadly, these devices did not dwell as much as the hype. They failed for a wide range of causes — like worth, redundancy, and utility — however the seeds they planted for a way forward for absolutely automated apps caught round.

These seeds grew into the “agentic AI” development we see at the moment. Companies are racing to construct AI merchandise that may carry out duties in your behalf, like serving to you code a brand new undertaking, reserving an appointment, or ordering gadgets on-line. As one of many leaders within the AI race, Google can be working by itself AI brokers, with Gemini in Chrome being its most notable providing.

Gemini in Chrome, in fact, can solely carry out actions inside the browser and never in different purposes. If you need to automate Android apps, your choices are restricted to third-party instruments like Tasker, which regularly have a steep studying curve. Even then, these instruments have to be meticulously configured to take particular actions in predetermined apps. Unlike newer AI brokers, these current automation instruments can’t carry out generalized duties from a single, pure language immediate.

That’s why Project Astra, Google’s experimental common AI undertaking, is so thrilling. At Google I/O, the corporate confirmed off a model of Astra that may management your Android telephone. In the demo, the assistant discovered a doc on-line, scrolled via it to seek out particular data, after which searched YouTube for associated movies — all utterly hands-free. To accomplish this, Astra recorded the display screen for evaluation after which despatched faucet or swipe inputs to launch apps or scroll via pages.

Google’s demo confirmed the immense potential for an AI agent that may carry out duties in Android apps, but it surely additionally revealed that the corporate nonetheless has lots of work to do. For starters, the elements of the video that includes the AI agent have been sped up 2x, suggesting it’s fairly gradual. This wasn’t a problem within the state of affairs concocted for the demo, the place the consumer clearly had their arms full, however will probably be an issue in the actual world. A gradual agent means your telephone might be occupied whereas it really works, and customary interruptions like a notification, an incoming name, or an alarm might disrupt the method by interfering with its display screen evaluation or inputs.

The objective of the I/O demo was merely to indicate off Project Astra’s capabilities slightly than element how an on-device AI agent would truly work. Google needed to hack collectively a prototype that took benefit of current Android APIs in unintended methods — the MediaProjection API for display screen recording and the Accessibility API for display screen enter — which resulted within the points talked about above.

Over the previous few months, nonetheless, Google has been engaged on a brand new, standardized framework for AI brokers to regulate Android apps. This framework, known as Computer Control, is designed to allow automated management of Android apps within the background, sidestepping these issues. Although Google in all probability gained’t announce Computer Control till subsequent yr’s Android 17 launch, we managed to uncover lots of details about it by digging via Android code. Here’s what we all know thus far.

You’re studying the Authority Insights Newsletter, a weekly publication that reveals some new side of Android that hasn’t been reported on anyplace else. If you’re searching for the most recent scoops, the most popular leaks, and breaking information on Google’s Android working system and different cell tech matters, then we’ve received you lined.

Subscribe right here to get this publish delivered to your electronic mail inbox each Saturday.

How Google’s Computer Control characteristic permits automated management of Android apps

With the discharge of Android 13 in 2022, Google launched a brand new system-level service known as the Virtual Device Manager (VDM). This service permits the creation of digital shows which can be separate from the first, seen show the consumer sees. Apps will be launched on these digital shows after which streamed to a distant machine. In flip, that distant machine can ship again enter occasions like clicks or keyboard presses for the system to inject into the app.

VDM already kinds the spine of the App Streaming characteristic in Chrome OS, which helps you to stream an app out of your Android telephone to your Chromebook. It additionally permits the Connected Camera characteristic on Pixel units, as Google quietly upgraded the service in final yr’s launch to assist digital cameras. The service performs a key position in Google’s long-term efforts to enhance cross-device compatibility, and it appears the corporate now goals to leverage it to energy its new Computer Control characteristic.

Google has added code for beginning a “Computer Control Session” to facilitate the automated management of Android apps. Each session entails a single trusted digital show that hosts the app being automated, in addition to digital enter units for sending contact and key occasions.

Client apps utilizing the Computer Control framework should specify the digital show’s properties, together with its identify, top, width, and density. They should additionally specify whether or not the show ought to stay unlocked and interactive even when the host machine is locked — a vital characteristic for really unattended management. (However, the machine should first be unlocked for an automation session to be initiated.)

Furthermore, shopper apps should set an output floor for the digital show’s content material. This permits them to entry the uncooked show frames, which might then be streamed to a distant linked machine for evaluation.

Another essential part of this framework is its means to reflect the trusted digital show onto a separate, interactive digital show. This interactive show can have completely different dimensions than the unique, and the system will routinely map enter occasions between them. This structure permits customers to see and manually work together with the app being automated, crucially with out interfering with the automation course of itself.

This separation is vital. If the unique trusted digital show have been mirrored on to a consumer — on a linked PC, for instance — then resizing the window might break the automation, as altering show dimensions can set off a configuration change that forces apps to restart. By making a second, interactive show that mirrors the trusted one, customers can remotely view and ship inputs to the app with out disrupting the method.

Given the delicate nature of this framework, entry is restricted to extremely privileged, trusted purposes. To use it, an app should first maintain the brand new ACCESS_COMPUTER_CONTROL permission. This permission can solely be held by apps signed with a digital certificates that has been explicitly allowlisted within the OS. Apps holding this permission should then ask the consumer to explicitly approve their use of the Computer Control characteristic; this approval will be granted for a single session or for all future classes.

Once an app’s request to start out a Computer Control session has been granted, the framework can prohibit it from launching or interacting with apps apart from the one being automated. This protects from automation classes being exploited to entry different delicate apps on the machine.

In observe, this implies solely choose apps trusted by Google or your machine’s producer could have entry to the Computer Control framework. Regular apps gained’t be capable to discreetly launch and management different apps within the background with out your information.

Computer Control — A brand new stage of automation for Android apps?

While we will conclude that the Computer Control framework is designed to let trusted purchasers analyze display screen information and automate duties in your behalf, there’s nonetheless lots we don’t know. For instance, it’s unclear how precisely these purchasers will management apps.

Does the “computer” in “Computer Control” actually seek advice from a PC, suggesting Google plans to stream apps to a distant PC or server for automation? This method could be much like how the Rabbit R1 works, although in that case, the apps ran totally on Rabbit’s servers. Or does “computer” imply a robotic in a extra normal sense, the place an on-device AI mannequin analyzes the display screen and performs actions domestically? The former is extra doubtless given the structure of the Computer Control framework and the place the code resides, however the latter can be actually doable when utilizing a multimodal mannequin like Gemini Nano. This method would even be extra personal, however it might consequently pressure the machine’s reminiscence and battery extra.

Whatever the case, we’re excited to see Google constructing a correct framework for true agentic AI on Android. The Computer Control framework opens the door to totally automating your apps, which is an enormous deal because it cannot solely save time but additionally dramatically enhance accessibility. Of course, AI brokers gained’t all the time get issues proper. This is probably going why Google included the flexibility to reflect the automation onto an interactive show, permitting customers to oversee the method and make changes as wanted.

Again, our understanding of this characteristic comes from code we examined within the newest Android construct (i.e., Android 16 QPR2 Beta 2). We might have missed some particulars, and it’s unclear when Google plans to launch it. If we study extra about Computer Control, we’ll be sure you let you recognize — so contemplate subscribing to the Authority Insights Newsletter so that you don’t miss a beat.

Want extra?

Authority Insights is greater than a publication — it’s the hub for all our greatest content material. If you care about Android, you gained’t need to miss any of our other exclusive reports.

Don’t have time to learn all of them? Subscribe to our Authority Insights Podcast to listen to me and my co-host, C. Scott Brown, break down our high tales of the week.