Databricks Demonstration: Pi's Minimalist Design Achieves Cost Reduction and High-Quality Results

Databricks conducted its own benchmark based on tasks performed daily by its engineers. The company's investigation found that the harness used to call the model significantly impacts both cost and quality. Specifically, it was reported that simple harnesses like Pi showed the best performance across many workloads.

When combined with Opus 4.8, Pi recorded the highest overall pass rate at a significantly lower cost than Claude Code or Codex. Databricks stated that when using different harnesses with the same model and level of reasoning effort, the cost per task varied by more than twofold, while quality remained comparable.

Pi reduced the number of executions required to complete a task by keeping the context per turn to approximately one-third and maintaining a narrow working set. This is referred to as "context discipline."

David Cortés of Shopify Engineering introduced a case where the automated optimization loop "Autoresearch" was developed simply by instructing Pi to create an extension for itself. Pi read its own extension documentation and constructed a new workflow.

This extension is being utilized as a practical productivity tool within Shopify. The system is designed to run experiments when changes are requested, continuing to self-improve while eliminating regressions.