[PRODUCT] Measure and reduce false positive impact on user retention
Business Problem
Churn analysis shows: 2% of monthly churn directly attributed to false positive moderation.
User feedback (last 30 days):
- 47 support tickets: "My post was wrongly removed"
- NPS score from affected users: -35 (vs +42 platform average)
- 12 users explicitly mentioned churn threat
Impact
Monthly cost of false positives:
- Lost revenue: $18K/month (2% of $900K MRR)
- Support cost: 47 tickets × 30min × $50/hr = $1,175
- Total: $19K/month or $228K/year
Success Metrics
Primary
- FPR: 12% → <5%
- Churn from FP: 2% → <0.5%
- Support tickets: 47/month → <15/month
Secondary
- User satisfaction (NPS): -35 → +20 for affected users
- Appeal success rate: 68% → <20% (fewer valid appeals = better initial decisions)
Engineering Work Required
- Fairness auditing (#5 (closed)) - reduced AAVE bias
- Sentiment filter (#9 (closed)) - 63% FPR reduction on idioms
- Review queue (#4 (closed)) - human validation
- None
- Analytics dashboard to track FPR by user segment
- A/B test framework to validate improvements
- User feedback loop ("Was this decision correct?")
Timeline
- Week 1-2: Build analytics dashboard (Michael)
- Week 3-4: Deploy feedback mechanism (mobile/web teams)
- Month 2: Measure baseline, iterate on models
- EOQ: Hit <5% FPR target, measure retention impact
Acceptance Criteria
-
FPR dashboard live in prod -
Weekly FPR reports to leadership -
FPR < 5% sustained for 30 days -
Churn from FP measurably reduced -
Support ticket volume down 50%
Owner: @ben (Product) Eng: @bob_wilson @sabrina_farmer Data: Need analytics eng support