Atlassian reports that after a year-long, large-scale online evaluation across more than 1,900 repositories, its Rovo Dev Code Reviewer cut median PR cycle time by 30.8% and reduced the number of human-written review comments by 35.6%. Of the comments Rovo Dev generated, 38.70% led directly to code changes in the subsequent commit, compared with 44.45% for human-written comments. Readability, bugs and maintainability were the most resolved comment types, and qualitative feedback from engineers was overwhelmingly positive, with the reviewer praised for finding nuanced bugs and subtle errors such as duplicate method names.
Atlassian's post says code review is a critical pillar of high-quality software engineering, but with the rapid growth of its products and the complexity of its codebase, manual reviews can become a significant bottleneck. It lists time spent by engineers, delays in shipping features and bug fixes, potential for human error and context switching as the challenges. The goal was an assistant to handle the repetitive, context-independent parts of code review so that engineers can focus on high-level logic, architecture and complex design problems that require human expertise.
After extensive internal testing and dogfooding, Atlassian deployed the reviewer at scale, and it went generally available in October 2025 after positive feedback from open beta customers. The online evaluation was accepted by the 48th IEEE/ACM International Conference on Software Engineering (ICSE 2026), a collaboration between Atlassian's DevAI engineering team, its data science team and associate professor Kla Tantithamthavorn of Monash University.
The reviewer is built on Anthropic's Claude 3.5 Sonnet and integrated into Bitbucket. It uses zero-shot structured prompting with a persona, chain-of-thought and review guidelines, fed with the pull request title and description, the related Jira issue summary and description, and the complete code diff. A second stage is an LLM-as-a-Judge check based on gpt-4o-mini that filters out hallucinated or factually incorrect comments. A third stage is a ModernBERT-based actionability check, fine-tuned on a proprietary, internally sourced dogfooding dataset of over 50,000 high-quality Rovo Dev-generated comments to classify whether comments led to a code resolution.
Rovo Dev works as a human-in-the-loop first-pass reviewer: it posts comments with suggestions and bugs, the final decision to accept or decline rests with the human reviewer, and suggestions can be accepted with a single click in Bitbucket. Human feedback is used to refine the AI over time. Atlassian says it is continuing to refine Rovo Dev by investigating advanced context enrichment techniques and improving its agentic capability.
Details