AI polish makes it harder to spot problems. But there’s a quick fix.
Atlassian’s Teamwork Lab found AI-polished drafts obscure fundamental flaws, reducing critical feedback. A simple 'Early Draft' label restores scrutiny without harming perceived effort.
Generative AI can quickly transform rough notes into polished documents, but this 'AI polish' may mask critical errors by presenting work as final. Atlassian’s Teamwork Lab tested whether polished drafts reduce reviewers’ ability to spot foundational flaws in a proposal to reduce IT tickets. The study assigned 903 reviewers three versions: unpolished, AI-polished, and AI-polished with an 'Early Draft' label.
The proposal intentionally contained two flaws: no adoption strategy for a self-help hub and success metrics tied to website traffic rather than ticket reduction. Reviewers evaluating the AI-polished version were significantly less likely to identify these errors compared to those reviewing the unpolished draft.
However, adding an 'Early Draft' label to AI-polished documents largely negated the polish effect, restoring flaw detection to levels comparable to unpolished drafts. Crucially, reviewers still perceived the work as high effort, addressing concerns that labeling might undermine credibility.
The study also found older employees and managers were less susceptible to AI polish, spotting flaws regardless of presentation. Researchers recommend using AI tools judiciously and labeling early drafts to balance efficiency with meaningful feedback, preventing oversight of critical issues.