SaaStr’s 20+ AI Agents: The Good, The Bad, and The Broken

The Gist

• SaaStr runs 20+ AI agents in production with just 3 humans overseeing operations
• Maintenance challenges emerge as agents misdiagnose issues and blame third-party tools
• Micro-hallucinations in AI agents highlight the need for continuous human oversight

• AI agents require constant monitoring and maintenance to avoid drifting from reality or providing incorrect data.
• AI agents can fail silently, leading to unnoticed backend issues while the frontend appears functional.
• AI agents should be trained on every product change, pricing update, and workflow to avoid giving customers wrong answers.
• AI agents excel in ensuring 'no lead is left behind' by consistently following up with every prospect and lead.
• AI agents can automate customer onboarding and follow-up processes, ensuring accountability and reducing gaps.
• AI agents can be easier to deploy and use when they are narrowly focused rather than broad and extensible.

• Monitor AI agent interactions daily to catch and correct errors or drifts in performance.
• Train AI agents on every product, pricing, and workflow change to ensure accurate customer interactions.
• Automate customer onboarding and follow-up processes using AI agents to ensure accountability and reduce gaps.
• Deploy narrowly focused AI agents for faster implementation and easier use, especially for GTM teams.

• 44% ahead of plan (An AI agent initially reported this metric but later corrected it to 11% due to comparing the wrong year.)
• 5x cost increase (Clay's Sculptor agent quoted 11,000 credits for a task that previously cost 2,500 credits.)
• 90% idle time (AI agents are idle 90% of the time, indicating underutilized capacity.)
• 4,000 startup pitch decks (An AI analyzer graded this number of pitch decks before encountering hallucinations due to a model update.)