Project

llm-tokens-atlas

Dataset and analysis project for understanding LLM token behavior, context use, and cost implications.

GitHub Repository All Projects

Focus

  • LLM analysis
  • Data
  • Cost research

GitHub Description

Token dataset and analysis work for LLM context and cost research.

  • Organizes token-related data for research and comparison.
  • Supports better intuition around context size, tokenization, and cost.

Package Links

This project is distributed as a dataset and Python package rather than npm packages.

Preview / Demo

Dataset preview

Load the public dataset with datasets.load_dataset('faraa2m/llm-tokens-atlas') and inspect empirical tokenization deltas across providers and prompt formats.

Open Hugging Face dataset