Learn & AI PM Curriculum

WEEK 01

AI Fundamentals & LLM Architecture for PMs

Mark Done

Understand how Transformer neural networks work under the hood, tokenization mechanisms, context windows, and cost vs. latency trade-offs for product features.

🎯 Core Learning Objectives

Understand the Transformer architecture: Self-attention mechanism, input embeddings, and next-token prediction loops.
Master tokenization math (1 token ≈ 0.75 words / 4 characters in English).
Evaluate Time-to-First-Token (TTFT) and throughput (tokens/sec) budget for user-facing features.
Calculate API costs for input vs. output tokens across GPT-4o, Claude 3.5 Sonnet, and Llama 3.

📖 Key Terminology

TTFT (Time To First Token)

Duration from user query submission to the first streamed character response. Target <800ms.

Context Window

Maximum combined limit of input prompt + output tokens model can process in one turn.

Temperature & Top-P

Hyperparameters controlling randomness/creativity vs. deterministic response output.

🛠️ Hands-On PM Deliverable

Create a Latency & Cost Estimation Budget sheet for a feature expecting 50,000 daily active queries.

WEEK 02

Prompt Engineering, System Prompts & Context Management

Mark Done

Master zero-shot, few-shot prompting, system instruction design, structured JSON output schemas, and security against prompt injection.

🎯 Core Learning Objectives

Write production-grade System Prompts specifying persona, task boundaries, tone, and fallbacks.
Implement Chain-of-Thought (CoT) prompting to improve reasoning on complex multi-step tasks.
Enforce structured JSON output formatting with Pydantic or JSON Mode for API integration.
Identify and mitigate Indirect Prompt Injection and jailbreaking vulnerabilities.

📖 Key Terminology

Few-Shot Prompting

Providing 2-5 input/output examples within the prompt to guide output structure and style.

Prompt Injection

Security exploit where untrusted user input overrides developer system instructions.

🛠️ Hands-On PM Deliverable

Draft a production-ready System Prompt for a Customer Support Classifier with JSON output schema.

WEEK 03

AI Agents & Agentic Workflows (ReAct, Tools & Planning)

Mark Done

Explore autonomous agents, the Reason + Act (ReAct) execution loop, tool-calling APIs, multi-agent orchestrations, and long-term memory.

🎯 Core Learning Objectives

Understand the 4 core components of an Agent: Perception, Planning, Memory, and Tool Execution.
Implement the ReAct loop: Thought → Action → Observation → Final Answer.
Design Function Calling schemas (OpenAI / Anthropic tool-use definitions).
Architect Multi-Agent patterns: Supervisor-Worker, Router, and Evaluator-Optimizer loops.

📖 Key Terminology

ReAct Pattern

Reasoning + Acting execution cycle allowing LLMs to query external APIs dynamically.

Tool Calling

Capability of an LLM to output formatted parameters to execute backend APIs or database queries.

🛠️ Hands-On PM Deliverable

Design an Agentic Workflow Diagram for an automated Travel Booking Assistant with fallback retry rules.

WEEK 04

RAG (Retrieval-Augmented Generation) & Vector Databases

Mark Done

Ground AI models with company knowledge bases, document chunking strategies, vector embeddings, semantic search, and hybrid retrieval.

🎯 Core Learning Objectives

End-to-End RAG Pipeline: Ingestion → Parsing → Chunking → Embedding → Indexing → Retrieval.
Evaluate Chunking Strategies: Fixed-size vs. Sentence-based vs. Parent-Document chunking.
Vector DB Architecture: Pinecone, Qdrant, Milvus, and PGVector similarity scoring (Cosine / Dot Product).
Advanced RAG: Hybrid Search (BM25 + Dense Vectors) and Cohere Reranking.

📖 Key Terminology

Embeddings

Dense numerical vector representations of text capturing semantic meaning in multi-dimensional space.

Cosine Similarity

Mathematical distance metric used to find the most relevant document chunks for a query.

🛠️ Hands-On PM Deliverable

Define RAG Data Governance & Chunking Specs for a company Knowledge Base Search tool.

WEEK 05

Fine-Tuning vs. RAG & AI Model Evaluations (Evals)

Mark Done

Choose when to use Fine-Tuning vs. RAG, and set up continuous evaluation pipelines (Evals) to measure response accuracy and prevent regressions.

🎯 Core Learning Objectives

Decision Framework: When to Fine-Tune (Tone/Style/Domain format) vs. RAG (Dynamic Knowledge/Facts).
Building Evals: Creating a Golden Test Dataset (50-200 ground truth test cases).
LLM-as-a-Judge: Automated evaluation scoring using Claude/GPT-4 for Faithfulness, Relevance, and Toxicity.
Regression Prevention: CI/CD integration for model prompt updates.

📖 Key Terminology

LLM-as-a-Judge

Using a stronger LLM (e.g. GPT-4o) to grade candidate model responses on accuracy criteria.

LoRA (Low-Rank Adaptation)

Efficient fine-tuning technique that freezes base weights and trains lightweight rank matrices.

🛠️ Hands-On PM Deliverable

Build a 30-Prompt Model Benchmark Evaluation Suite with scoring criteria for accuracy & hallucination rates.

WEEK 06

Building Agentic Products: UX Patterns, Fallbacks & Safety

Mark Done

Master user experience design for probabilistic AI features, handling streaming responses, human-in-the-loop approvals, and guardrails.

🎯 Core Learning Objectives

AI UX Patterns: Streaming text, thought visibility, source attribution, and inline editing.
Safety Guardrails: Implementing NeMo Guardrails, Llama Guard, and PII anonymization layers.
Human-in-the-Loop (HITL): Approval gates for high-risk actions (sending emails, making payments).
Graceful Fallbacks: Degrading gracefully when latency spikes or model rate limits occur.

📖 Key Terminology

Human-in-the-Loop (HITL)

UX pattern requiring explicit human verification before an agent executes critical actions.

PII Anonymization

Stripping Personally Identifiable Information before sending user data to third-party LLMs.

🛠️ Hands-On PM Deliverable

Design Wireframes & Error Recovery Flows for an Agentic Email Writer with Human Approval steps.

WEEK 07

Capstone Project & AI Product Lifecycle Management

Mark Done

Synthesize everything learned into a complete PRD (Product Requirements Document) for an Enterprise Agentic AI product.

🎯 Core Learning Objectives

AI Product PRD: Problem definition, target persona, agent architecture, metrics, and risk assessment.
Key Metrics: CSAT, Resolution Rate, Cost per User, Fallback Rate, and Model Drift.
GTM & Monitoring: Setting up telemetry dashboards (LangSmith, Helicone, Datadog LLM).
Continuous Lifecycle: Iterative prompt updates, model fine-tuning cycles, and user feedback loops.

📖 Key Terminology

Model Drift

Degradation in model response performance over time due to changing real-world data distribution.

LLM Telemetry

Monitoring tool tracking token usage, latency distribution, cost per user, and error rates.

🛠️ Final Capstone Deliverable

Publish a complete Technical PRD for an Enterprise AI Agent with latency budget, safety rules, and ROI analysis.

AI Product Management Learning Curriculum

AI Fundamentals & LLM Architecture for PMs

Prompt Engineering, System Prompts & Context Management

AI Agents & Agentic Workflows (ReAct, Tools & Planning)

RAG (Retrieval-Augmented Generation) & Vector Databases

Fine-Tuning vs. RAG & AI Model Evaluations (Evals)

Building Agentic Products: UX Patterns, Fallbacks & Safety

Capstone Project & AI Product Lifecycle Management

⚡ Interactive PM Tool: Token Cost & Latency Estimator