LEARN WITH DIRECTION
Small steps. Deeper understanding.
A sequence beats a random walk. Choose a path and work through it at your own pace.
01 / Your priority path
Start this planOpenAI ML coding prep
Fifty focused implementations: matrices and gradients, vectorized neighbors, entropy, attention, training bugs, and data evaluation. Independent preparation, not confirmed interview questions.
0 of 50 completed0%
01Matrix-vector multiplication and shape contractsUnrated02Multiclass cross-entropy and manual linear-layer gradientsUnrated03A custom autograd operationUnrated04Central-difference gradient checkingUnrated05MLP BackwardHard06Find the missing gradientEasy07Repair a small classifier training stepUnrated08Pairwise squared distancesUnrated09Nearest neighbors with deterministic tiesUnrated10cumsum · each output chooses a prefixMedium11Entropy and exact discrete KL divergenceUnrated12Online mean and varianceUnrated13Scaled attention with fully masked rowsUnrated14Multi-head causal self-attention from scratchUnrated15Causal attention with padding and empty rowsUnrated16KV-cached causal attention and offset masksUnrated17A two-layer NumPy MLP and its complete backward passUnrated18A classifier training step that actually changes parametersUnrated19Overfit one batchEasy20Audit labels, duplicates and split leakageUnrated21Audit duplicates, group leakage and classifier metricsUnrated22Binary confusion metrics with explicit edge policiesUnrated23Fit preprocessing on training data onlyUnrated24Choose a threshold using validation dataUnrated25Stable log-sum-expUnrated26Stable softmax over the correct axisUnrated27Softmax backward without a quadratic JacobianUnrated28Binary cross-entropy from logitsUnrated29Linear regression loss and gradientsUnrated30One k-means assignment/update stepUnrated31Masked mean with an empty-row policyUnrated32Scatter-add with repeated indicesUnrated33Embedding lookup and repeated-token gradientsUnrated34Pad a variable-length batch without guessing validity from token IDUnrated35Masked token loss with correct weightingUnrated36Inverted dropout and train/eval behaviorUnrated37Batch normalization with running statisticsUnrated38Layer normalization from primitive operationsUnrated39SGD with momentumUnrated40Adam and decoupled AdamWUnrated41Correct accumulation for unequal microbatchesUnrated42Global gradient norm clippingUnrated43A complete tiny decoder-only TransformerUnrated44Temperature, top-k and nucleus sampling probabilitiesUnrated45Reproducible minibatchesUnrated46ROC-AUC with tied scoresUnrated47Expected calibration errorUnrated48Covariance with samples as rowsUnrated49Categorical sampling by inverse CDFUnrated50Gaussian maximum-likelihood estimatesUnrated
02 / Start here
Start this planTensor fluency
Twenty-one small puzzles for a big shift in how you think about arrays. Work with shapes, broadcasting, indexing, and vectorization.
0 of 21 completed0%
01ones · make a shape, fill it with oneEasy02sum · a reduction is a dot productEasy03outer · make each value meet every other valueEasy04diag · index arrays are pairedEasy05eye · equality draws a diagonalEasy06triu · an inequality draws a triangleEasy07diff · subtract the preceding inputEasy08vstack · choose a source by output rowEasy09roll · wrap a coordinate with moduloEasy10flip · reverse the coordinate systemEasy11sequence_mask · each row has its own horizonEasy12linspace · move indices onto a number lineEasy13heaviside · translate the branch literallyEasy14repeat · multiply every row by oneEasy15cumsum · each output chooses a prefixMedium16pad_to · a rectangle can act like an identityMedium17bincount · categories become one-hot rowsMedium18scatter_add · a histogram with arbitrary payloadsMedium19flatten · decode a linear addressMedium20bucketize · count the boundaries you have crossedMedium21compress · turn the mask into destination addressesHard
03 / Build the fundamentals
Start this planPyTorch foundations
Build the fundamentals of a working training loop: tensors, gradients, modules, optimizers, and evaluation.
0 of 20 completed0%
01Linear Regression Using Gradient DescentEasy02SoftmaxEasy03Cross-Entropy LossEasy04Gradient ClippingEasy05Gradient AccumulationEasy06Compute Multi-class Cross-Entropy LossEasy07Gradient CheckpointingEasy08Implement Early Stopping Based on Validation LossEasy09Implement Mean Squared Error LossEasy10Implement Ridge Regression Loss FunctionEasy11Implement Softmax FunctionEasy12Implement the Hard Sigmoid Activation FunctionEasy13Sigmoid Activation Function UnderstandingEasy14SoftmaxEasy15Softmax Activation Function Implementation Easy16Overfit one batchEasy17Find the missing gradientEasy18Calculate Covariance MatrixEasy19DropoutEasy20ExponentialLR Learning Rate SchedulerEasy
04 / Go deeper
Start this planInside the transformer
Move from shape manipulation to attention and the building blocks of modern language models. Implement each piece before putting it together.
0 of 20 completed0%
01Scaled attention with fully masked rowsUnrated02Multi-head causal self-attention from scratchUnrated03Sinusoidal position embeddings, including odd widthsUnrated04Rotary position embeddingsUnrated05SwiGLU feed-forward blockUnrated06KV-cached causal attention and offset masksUnrated07Grouped-query attentionUnrated08A complete tiny decoder-only TransformerUnrated09Packed-sequence attention boundariesUnrated10Temperature, top-k and nucleus sampling probabilitiesUnrated11Beam search with finished hypothesesUnrated12Apply byte-pair merges without overlapping replacementsUnrated13Online softmax-weighted accumulationUnrated14Causal attention with padding and empty rowsUnrated15Build a Complete LLM Inference EngineUnrated16Implement 2D Positional EmbeddingsUnrated17Implement KV Cache for Autoregressive GenerationUnrated18Implement LoRA on a Linear LayerUnrated19Implement Sliding Window AttentionUnrated20Implement a Swin Transformer Block (Shifted Window Attention)Unrated
05 / The complete archived collection
Start this planTensorGym, recovered
All 30 original descriptions, with the 20 recovered starters and solutions. Original test data is preserved; incomplete or inconsistent cases are labeled.
0 of 30 completed0%
01Warmup: Tensor additionEasy02Tensor flatteningEasy03Tensor Reshaping and TranspositionEasy04Sum of Every Second RowEasy05Row with Highest MeanEasy06Find the Top Scoring StudentEasy07Normalize Scores and Count High Achievers per SubjectEasy08Implement Softmax FunctionEasy09Implement Attention MechanismMedium10Implement Batch NormalizationMedium11Compute Variance ManuallyEasy12k-Nearest Neighbors (k-NN) ClassificationHard13Implement K-means ClusteringHard14Simple MLP Forward PassEasy15Tensor broadcastingEasy16Implementing Custom ReLU Activation FunctionMedium17Implement Mean Squared Error LossEasy18Implement LoRA in a Linear LayerHard19Drop Column with the most zerosEasy20Cube RotationsMedium21Standardize Features (Z-score)Easy22Add Bias Column for Linear ModelsEasy23Single Gradient Descent Step (Linear Regression)Medium24Logistic Regression: Probabilities and LabelsMedium25Masked Scaled Dot-Product AttentionHard26Scale and Shift a TensorEasy27Column SumsEasy28Row-wise Max Values and IndicesEasy29Product of Main DiagonalEasy30Convert Vector to Column VectorEasy
06 / Strengthen your intuition
Start this planML from first principles
Rebuild your mathematical intuition through classical algorithms, probability, and evaluation. Focus on why each implementation works.
0 of 24 completed0%
01Stable least squaresUnrated02Linear regression loss and gradientsUnrated03Binary cross-entropy from logitsUnrated04Multiclass cross-entropy and manual linear-layer gradientsUnrated05One k-means assignment/update stepUnrated06PCA by SVDUnrated07Online mean and varianceUnrated08Reproducible minibatchesUnrated09Audit labels, duplicates and split leakageUnrated10Fit preprocessing on training data onlyUnrated11Binary confusion metrics with explicit edge policiesUnrated12ROC-AUC with tied scoresUnrated13Choose a threshold using validation dataUnrated14Expected calibration errorUnrated15Paired cluster bootstrap for an evaluation differenceUnrated16Pad a variable-length batch without guessing validity from token IDUnrated17Audit duplicates, group leakage and classifier metricsUnrated18Implement Temperature SamplingUnrated19Implement Top-k SamplingUnrated20Implement Top-p (Nucleus) SamplingUnrated21How do you assess noisy labels?Unrated22How would you find evaluation leakage?Unrated23A new method wins by one percentage point. What next?Unrated24What is a good baseline?Unrated