Einstein Copilot Internal Deployment at Salesforce

Title

Einstein Copilot Internal Deployment at Salesforce

Content

Salesforce conducted a large-scale internal deployment of Einstein Copilot, their conversational AI assistant built on large language models (LLMs), within their own enterprise CRM environment starting February 2024. The deployment was a significant dogfooding exercise where Salesforce used their own product internally before broader customer availability. Despite the exceptional complexity of Salesforces internal org — with thousands of objects, hundreds of thousands of fields, and billions of records — deploying Einstein Copilot technically took less than two hours. However, comprehensive testing against this massive dataset took considerably longer. The rollout strategy followed a phased approach: starting with just 100 sellers as a controlled pilot group for direct feedback and rapid iteration, before progressively expanding to thousands of users. For testing, the team prepared approximately 30 potential user queries for each standard Copilot action, manually evaluating whether each query was successfully executed. They established an 80% success rate target for queries related to supported use cases, and tested more than 250 queries total, creating a foundational regression test set. For the 20% of failing queries, they made deliberate decisions about whether to document as known limitations or build custom actions. Custom actions tailored to specific business needs — such as giving sellers a daily account update or helping prioritize tasks — were quickly realized to be essential for delivering extra value to sales teams. The deployment already saves countless hours of work daily across thousands of employees. Key lessons included starting with standard actions before adding custom capabilities, implementing comprehensive testing protocols, and establishing clear feedback loops.