AI PM Cheatsheet

Your quick reference guide for AI Product Management

šŸ“Š Key Metrics & KPIs

Essential metrics every AI PM should track

šŸŽÆModel Performance Metrics

  • Accuracy Critical
    (TP + TN) / (TP + TN + FP + FN)
  • Precision
    TP / (TP + FP) - Minimize false positives
  • Recall
    TP / (TP + FN) - Minimize false negatives
  • F1 Score
    2 Ɨ (Precision Ɨ Recall) / (Precision + Recall)
  • AUC-ROC
    Area under ROC curve (0.5-1.0)

šŸ’°Business Metrics

  • ROI Critical
    (Gain - Cost) / Cost Ɨ 100%
  • Time to Value
    Time from project start to first business impact
  • Adoption Rate
    Active Users / Total Users Ɨ 100%
  • Churn Rate
    Lost Customers / Total Customers
  • NPS Score
    % Promoters - % Detractors

āš”System Performance

  • Latency Important
    P50, P95, P99 response times
  • Throughput
    Requests per second (RPS)
  • Error Rate
    Failed requests / Total requests
  • Availability
    Uptime / Total time (Target: 99.9%+)
  • Resource Utilization
    CPU, Memory, GPU usage %

šŸ“ˆData Quality Metrics

  • Completeness
    Non-null values / Total values
  • Consistency
    Matching records across sources
  • Data Drift Important
    KL divergence, PSI score
  • Label Quality
    Inter-annotator agreement rate
  • Coverage
    Data points per segment

šŸ”„ AI Project Lifecycle

Key stages and deliverables

1ļøāƒ£Discovery Phase

Problem Definition

Define business problem, success metrics, constraints

Feasibility Study

Assess data availability, technical feasibility, ROI

Stakeholder Alignment

Get buy-in, define roles, set expectations

2ļøāƒ£Development Phase

Data Collection

Gather, clean, label training data

Model Development

Train, validate, iterate on models

Integration

Build APIs, integrate with systems

3ļøāƒ£Deployment Phase

A/B Testing

Run experiments, measure impact

Gradual Rollout

Progressive deployment, monitor metrics

Full Launch

Complete rollout, documentation

4ļøāƒ£Monitoring Phase

Performance Tracking

Monitor KPIs, detect degradation

Model Retraining

Update with new data, fix drift

Continuous Improvement

Iterate based on feedback

šŸ“ Essential Formulas

Key calculations for AI PMs

Statistical Significance (A/B Testing)

Z = (p1 - p2) / āˆš(p Ɨ (1-p) Ɨ (1/n1 + 1/n2)) where: p = pooled proportion p1, p2 = conversion rates n1, n2 = sample sizes

Use Z > 1.96 for 95% confidence

Sample Size Calculation

n = (ZĀ²Ī±/2 Ɨ p Ɨ (1-p)) / EĀ² where: Z = Z-score (1.96 for 95% CI) p = expected proportion E = margin of error

Minimum users needed per variant

Model Accuracy Trade-offs

Precision ā†‘ = Fewer False Positives Recall ā†‘ = Fewer False Negatives F1 = Harmonic mean of both

Balance based on use case

Cost per Prediction

Cost = (Infrastructure + Data + Development) / (Predictions Ɨ Time Period)

Track for budget optimization

šŸ’” Pro Tips

Best practices from experienced AI PMs

āš ļø Data Quality > Model Complexity

80% of AI project success depends on data quality. Invest heavily in data infrastructure and governance.

šŸŽÆ Start with MVP

Begin with simple rules or heuristics. Prove value before investing in complex ML models.

šŸ“Š Define Success Early

Establish clear, measurable success criteria before starting development. Include both technical and business metrics.

šŸ”„ Plan for Retraining

Models degrade over time. Budget for continuous retraining and monitoring from day one.

šŸ‘„ Involve Stakeholders

Regular demos and updates prevent surprises. Keep stakeholders engaged throughout the project.

āš–ļø Ethics First

Consider bias, fairness, and privacy implications early. Build responsible AI from the ground up.

āš ļø Risk Management Checklist

Common risks and mitigation strategies

šŸ”“Critical Risks

  • Data Privacy Breach
    ā†’ Implement encryption, access controls, audit logs
  • Model Bias
    ā†’ Regular bias testing, diverse training data
  • Production Failure
    ā†’ Rollback plans, monitoring, gradual rollout
  • Regulatory Non-compliance
    ā†’ Legal review, documentation, audit trails

šŸŸ”Common Pitfalls

  • Overfitting
    ā†’ Cross-validation, regularization, more data
  • Data Drift
    ā†’ Monitor distributions, automated retraining
  • Scope Creep
    ā†’ Clear requirements, phase-based delivery
  • Technical Debt
    ā†’ Regular refactoring, documentation

šŸ’¬ Stakeholder Communication Templates

Key messages for different audiences

šŸ‘”Executive Updates

  • Focus on Impact
    Revenue impact, cost savings, strategic value
  • Timeline & Milestones
    Key dates, dependencies, risks
  • Resource Needs
    Budget, headcount, infrastructure

šŸ‘Øā€šŸ’»Engineering Teams

  • Technical Requirements
    APIs, data formats, performance needs
  • Integration Points
    Systems, dependencies, interfaces
  • Success Metrics
    Latency, accuracy, scalability targets

šŸ‘„End Users

  • Value Proposition
    How it helps them, time saved
  • How It Works
    Simple explanation, no jargon
  • Privacy & Trust
    Data usage, control, transparency

šŸ“ŠData Scientists

  • Business Context
    Problem, constraints, success criteria
  • Data Availability
    Sources, quality, volume, labels
  • Production Requirements
    Latency, throughput, maintenance