Tokens, context and why AI forgets things

AI models forget Information because they have a limited "context window" a maximum number of tokens (words/parts of words) they can process at once. When a conversation exceeds this limit, the model drops the oldest information to make room for new input, akin to a First-in-first-out (FIFO) system

1. Tokens: These are the basic units of data (parts of words, punctuation) that AI models use to process text. A single English word is typically slightly more than one token, and a paragraph can contain 100 to 150 tokens.
2. Context Window: This represents the AI's "short-term working memory". It includes everything in the current chat: your prompts, instructions, and previous AI responses. If the prompt + history exceeds this limit, the earliest parts of the conversation are "forgotten".
3. Why AI Forgets:
   1. Capacity Limit: When the maximum number of tokens is reached, the model simply cannot hold more information.
   2. FIFO (First-In-First-Out): The oldest messages are pushed out first to accommodate new ones.
   3. Lost in the Middle: Models sometimes struggle to retrieve information buried in the middle of long, complex, or heavily loaded contexts.
   4. Focus Shift: Models are optimized for the current task, causing them to prioritize recent information over older, potentially crucial, details.