uclanlp / OpenVLHarness
Official Code for OpenVLHarness: Real-World Visual Agentic Tool‑Use with Multimodal Memory
uclanlp / OpenVLHarness
huggingface / transformers
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
bojieli / ai-agent-book
《深入理解 AI Agent：设计原理与工程实践》（李博杰 著）开源主仓库：全书正文、编译版 PDF 与按章配套代码
sgl-project / sglang
SGLang is a high-performance serving framework for large language models and multimodal models.
deepset-ai / haystack
Open-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search, and conversational systems.
microsoft / unilm
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
NVIDIA-NeMo / Speech
A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)
pipecat-ai / pipecat
Open Source framework for voice agents, multimodal apps, and realtime AI. Maintained by Daily and the community.
modelscope / ms-swift
Use PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600+ LLMs (Qwen3.8, DeepSeek-V4, GLM-5.1, InternLM3, Llama4, ...) and 300+ MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM5.3, Gemma4, Llava, Phi4, ...) (AAAI 2025).
multimodal-art-projection / YuE
YuE2: frontier music generation with symbolic planning, zero-shot covers, and agentic music editing.
datawhalechina / all-in-rag
🔍大模型应用开发实战一：RAG 技术全栈指南，在线阅读地址：https://datawhalechina.github.io/all-in-rag/
CVHub520 / X-AnyLabeling
X-AnyLabeling: A lightweight, efficient, and unified cross-platform desktop application for annotating text, image, video, point cloud and multimodal data, combining versatile built-in tools with state-of-the-art AI models and flexible multi-format export.
StarTrail-org / PixelRAG
https://arxiv.org/abs/2606.28344. The end of web parsing. The beginning of scalable pixel-native search. link: https://pixelrag.ai/
OthersideAI / self-operating-computer
A framework to enable a multimodal model to operate a computer.
yzhao062 / pyod
A Python library for anomaly detection across tabular, time series, graph, text, image, and audio data. 60+ detectors, benchmark-backed ADEngine orchestration, and an agentic workflow for AI agents.
xorbitsai / inference
Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.
bentoml / BentoML
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
AI4Finance-Foundation / FinRobot
FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Models
Blaizzy / mlx-audio
A text-to-speech (TTS), speech-to-text (STT) and speech-to-speech (STS) library built on Apple's MLX framework, providing efficient speech analysis on Apple Silicon.
vllm-project / vllm-omni
A framework for efficient model inference with omni-modality models
OpenBMB / UltraRAG
A Low-Code MCP Framework for Building Complex and Innovative RAG Pipelines
facebookresearch / mmf
A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)
InternLM / xtuner
A Next-Generation Training Engine Built for Ultra-Large MoE Models
kyegomez / tree-of-thoughts
Plug in and Play Implementation of Tree of Thoughts: Deliberate Problem Solving with Large Language Models that Elevates Model Reasoning by atleast 70%
EvolvingLMMs-Lab / lmms-eval
One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks
morphik-org / morphik-core
Open-source multimodal retrieval engine (Morphik Core)
embeddings-benchmark / mteb
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
nicedreamzapp / claude-code-local
Run Claude Code 100% on-device with local AI on Apple Silicon. MLX-native Anthropic-API server. 6 fighters incl. Muse-Glimmer 30B (now multimodal — reads images, abliterated), Gemma 4 31B, Qwen 3.5 122B (65 tok/s), DeepSeek V4 Flash (1M ctx). Private, offline, airgap-ready. Built for NDA / legal / healthcare workflows.
xlang-ai / OSWorld
[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
QwenLM / Qwen-MM-Plugins
Make any agent harness multimodal-native.
datachain-ai / datachain
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
autodistill / autodistill
Images to inference with no labeling (use foundation models to train supervised models).
roboflow / maestro
streamline the fine-tuning process for multimodal models: PaliGemma 2, Florence-2, and Qwen2.5-VL
om-ai-lab / OmAgent
[EMNLP-2024] Build multimodal language agents for fast prototype and production
zai-org / GLM-V
GLM-4.6V/4.5V/4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning
NVIDIA-NeMo / DataDesigner
🎨 NeMo Data Designer: Generate high-quality synthetic data from scratch or from seed data.
HUANGCHIHHUNGLeo / claude-real-video
Let Claude (or any LLM) actually watch a video — scene-aware, deduplicated frames + transcript, from a URL or local file. Runs locally, MIT.
google-gemini / genai-processors
GenAI Processors is a lightweight Python library that enables efficient, parallel content processing.
shcherbak-ai / contextgem
Extract structured data and insights from documents using LLMs and decision models
kyegomez / BitNet
Implementation of "BitNet: Scaling 1-bit Transformers for Large Language Models" in pytorch
NVIDIA-AI-Blueprints / video-search-and-summarization
NVIDIA AI Blueprint for video search and summarization (VSS) is a GPU-accelerated reference architecture for building video analytics agents with real-time verified alerts, visual Q&A, and automated reporting. The VSS Blueprint uses vision language models (VLMs) such as NVIDIA Cosmos, LLMs such as NVIDIA Nemotron, RAG, and NVIDIA NIMs.
Tencent / WeMM-Embedding
WeMM-Embedding is a family of universal multimodal embedding models by the WeChat Vision Team at Tencent, supporting multimodal understanding and retrieval.
bilibili / Index-Translate
A Multilingual, Multimodal Translation Model Family
facebookresearch / multimodal
TorchMultimodal is a PyTorch library for training state-of-the-art multimodal multi-task models at scale.
pixeltable / pixeltable
The backend agents build with - Multimodal database, orchestration, and serving in one file
waybarrios / vllm-mlx
High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.
sgl-project / sglang-omni
SGLang-Omni is a high-performance serving runtime for interactive intelligence model systems.
emcf / thepipe
Get clean data from tricky documents, powered by vision-language models ⚡
NPC-Worldwide / npcpy
The python library for research and development in NLP, multimodal LLMs, Agents, ML, Knowledge Graphs, and more.
mattmireles / gemma-tuner-multimodal
Fine-tune Gemma 4 and 3n with audio, images and text on Apple Silicon, using PyTorch and Metal Performance Shaders.
yuezhiai / jonex
All-in-One Multimodal Parsing Engine + Ontology-Powered, LLM Wiki-Driven AI-Ready Knowledge Engine
Capsize-Games / airunner
Offline inference engine for art, real-time voice conversations, LLM powered chatbots and automated workflows
verl-project / verl-omni
Multimodal RL training framework for diffusion & omni models
Anionex / agent-vision-toolkit
为纯文本模型"看图“设计更好的视觉工具箱和技能，支持多图理解，图片问答，前端UI还原、GUI 自动化等，并可选无缝接入多个主流agent，直接识别粘贴图片｜ A vision toolkit and skill designed for text-only llms — image Q&A, long-screenshot OCR, frontend UI restoration, and GUI automation, with optional seamless integration for Codex, Claude Code, Pi, Oh My Pi, and OpenCode
EvolvingLMMs-Lab / LLaVA-OneVision-2
Fully Open Framework for Democratized Multimodal Training
ALwrity / ALwrity
ALwrity - AI-first Digital Marketing Platform. AI Content Strategy and Planning, Multimodal content generation, Publishing, Analytics, AI SEO, Connect & Manage Social Accounts. Marketing OS - WIP
lhotse-speech / lhotse
Tools for handling multimodal data in machine learning projects.
OpenMOSS / MOVA
A foundation model that generates synchronized video and audio in a single model
dtsola / xiaoyaosearch
小遥搜索，听懂你的话、看懂你的图，用AI找到本地任何文件。让搜索像聊天一样简单。XiaoyaoSearch: Understands your words, reads your images, finds any local file with AI. Making search as easy as chatting.
Tencent-Hunyuan / UniRL
UniRL is a Framework for Unified Multimodal Model Reinforcement Learning