What if the activations are still too big?

1. Inputs to each Transformer block can't be recomputed quickly
2. Still, we can offload to CPU

Prefetch before we reach the layer, similar to weights for FSDP

First implemented in:

Unsloth Gradient Checkpointing - 4x longer context windows

unsloth.ai/blog/long-context