Microsoft’s AI systems function as a "black box"
Despite Microsoft’s assurances, cybersecurity researcher Michael Bargury has demonstrated that Copilot Studio, a tool enabling businesses to build their own AI assistants, can be exploited to extract sensitive enterprise data. Speaking at the Black Hat conference, Bargury emphasized that while Microsoft is actively addressing vulnerabilities, the rapid enterprise adoption of AI, particularly under pressure from major vendors, presents serious security concerns. He pointed out that Microsoft’s AI systems function as a "black box", meaning users have little visibility into how decisions are made and data is processed, making security oversight difficult. Through reverse engineering, his team identified ten security mechanisms but also discovered 15 different ways to exploit Copilot, exposing its susceptibility to data leaks and unauthorized access.
One alarming proof-of-concept attack Bargury demonstrated involved manipulating financial transactions in Microsoft 365 Copilot. By sending a malicious email, an attacker could remotely hijack Copilot, trick it into searching for sensitive data, embedding that data into a URL, and then baiting a victim into clicking the link, effectively exfiltrating data. While enterprises may consider opting out of Copilot, Bargury warns that avoiding AI-powered tools entirely is nearly impossible, as AI will soon be deeply embedded into business software and services. This underscores the need for enhanced transparency, security measures, and enterprise awareness to mitigate AI-related risks.
