speedydivid / TokenForge

Context window optimization for AI coding agents. Sandboxes tool output (98% reduction), persists session memory via SQLite + FTS5, and enforces routing across 17 platforms through MCP + hooks. Works with Claude Code, Gemini CLI, VS Code Copilot, OpenCode, Codex CLI.

juspay / hyperswitch

Open source, composable payments platform | PCI compliant | SaaS and Self-host options | Enables connectivity to multiple payment, payout, fraud, vault and tokenization providers | Uplifts authorization with intelligent routing and revenue recovery | Reduce payment processing costs with cost observability | Reduces payment ops with reconciliation

explosion / spaCy

💫 Industrial-strength Natural Language Processing (NLP) in Python

toon-format / toon

🎒 Token-Oriented Object Notation (TOON) – compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.

stanfordnlp / CoreNLP

CoreNLP: A Java suite of core NLP tools for tokenization, sentence segmentation, NER, parsing, coreference, sentiment analysis, etc.

stanfordnlp / stanza

Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages

amitshekhariitbhu / llm-internals

Learn LLM internals step by step - from tokenization to attention to inference optimization.

smg-project / smg

Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, TokenSpeed, SGLang, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, chat history, tokenization caching, Responses API, embeddings, WASM plugins, MCP, and multi-tenant auth.

liangpan99 / TokenHSI

[CVPR 2025 Oral] TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenization

loupe-php / loupe

A full text search engine with tokenization, stemming, typo tolerance, filters and geo support based on only PHP and SQLite.

zeemscript / TrustMint

Open-source toolkit for compliant real-world asset tokenization on Stellar.KYC registry, compliance engine, and asset templates for invoices, property shares, and carbon credits.

devlab-group / rwa-platform-evm

A self-hosted, single-tenant platform for tokenizing a real-world asset class as a permissioned ERC-20, with an auditor-attested supply lifecycle and on-chain cash redemption.

ziplab / ZipTok3D

ZipTok3D: High-Fidelity 3D Tokenization with Compact Token Prefixes

michaelc-yu / cs336-llm-systems

Large language model systems implementation and optimization: Transformers, BPE tokenization, FlashAttention, Triton, distributed training, GPU profiling, and GRPO reasoning RL

1amageek / swift-gigatoken

670 MB/s Swift tokenizer inspired by Marcel Roed’s gigatoken. Zero-copy, NEON/CRC32, WebAssembly, and Embedded Swift.