Nebius Benchmarks Its Sentinel Compliance Audit Agent Across Four Configurations on an FDA Guidance Task

Nebius built Sentinel, a regulatory compliance audit agent, on its Nebius Agents Blueprint and described the work in an engineering post by Tikhon Roshchupkin and Aleksandr Patrushev. The agent takes a change in regulatory guidance, finds every Standard Operating Procedure it touches across a corpus of 200 SOPs spanning 10 business units, classifies the severity of each gap against 36 regulatory frameworks such as HIPAA, SOC 2, GDPR, the EU AI Act and the NIST AI RMF, and files a Jira ticket for every confirmed finding. Nebius ran the same live FDA audit task through four configurations to show which bottleneck each step removed. The prototype used GPT-5.5 with Pinecone for static retrieval and took 29.9 minutes at a cost of $469.99, but could not ground its answers in current guidance. The grounded configuration added Tavily for live web search, which raised freshness and accuracy but cost $657.19. The optimised configuration kept the prompts and retrieval and replaced GPT-5.5 with DeepSeek-V4-Pro running on Nebius Token Factory, which cut the cost to $34.63, roughly 19 times cheaper, with data never leaving the environment; on Token Factory the model swap is one line of configuration, although this run took 53.9 minutes. The production configuration ran on Nemotron-Ultra and added LangSmith for tracing every tool call, retrieval and decision, and Snowglobe from Guardrails AI for adversarial simulation before launch. It completed the task in 12.6 minutes for $23.59, the fastest and cheapest run, and produced 10 high-severity Jira tickets grouped into five remediation themes. LangSmith traces revealed that 27% of sub-agent traces (80 of 299) had silently returned empty answers that the wrapping tool reported as clean results. Quality was measured on a fixed benchmark of 120 tasks with known ground truth. A single-shot RAG call on the same model and the same Pinecone index gave up on complex queries that the agentic setup answered. The production configuration reached 1.00 recall with the best precision of the field, 0.91, for $7.93 across all 120 tasks.