Microsoft has introduced a new capability called "computer use" within its Copilot Studio. This feature empowers AI agents
to interact with both websites and desktop applications in a manner akin to a human user.
According to Microsoft, Copilot Studio agents can now execute actions such as clicking buttons, navigating menus, and entering text into fields displayed on the screen. This "computer use" functionality allows these agents to engage with any system that presents a graphical user interface (GUI).
Charles Lamanna, a corporate vice president at Microsoft, explained in a blog post that this advancement enables agents to handle tasks even when a direct Application Programming Interface (API) for system connection is unavailable. He emphasized that if a person can operate an application, the agent can now do so as well.
Microsoft asserts that this system is designed to be resilient to changes in user interfaces and can adapt dynamically. This adaptability reduces the necessity for frequent reprogramming. Lamanna further elaborated that "computer use" automatically adjusts to modifications in apps and websites, employing built-in reasoning to resolve issues independently and maintain uninterrupted workflow. This feature mimics actual user behavior, opening up possibilities for automating workflows in applications and websites that were previously inaccessible to bots. Earlier in April, Microsoft rolled out a similar feature named Actions for its consumer-focused Copilot.
Similar functionalities have been launched by Anthropic's Claude, also under the name "computer use," and by OpenAI, which introduced OperatorAI in January.
While the ability for AI agents to independently use computers may enhance productivity, cybersecurity experts have also raised concerns about their potential misuse in cyberattacks. A threat-hunting team at Symantec recently demonstrated how OperatorAI could be employed to create a convincing phishing attack.
Found this article interesting? Follow us on X(Twitter) ,Threads and FaceBook to read more exclusive content we post.
