bojieli / ai-agent-book
《深入理解 AI Agent：设计原理与工程实践》（李博杰 著）开源主仓库：全书正文、编译版 PDF 与按章配套代码
bojieli / ai-agent-book
multimodal-art-projection / YuE
YuE2: frontier music generation with symbolic planning, zero-shot covers, and agentic music editing.
uclanlp / OpenVLHarness
Official Code for OpenVLHarness: Real-World Visual Agentic Tool‑Use with Multimodal Memory
Mintplex-Labs / anything-llm
Stop renting your intelligence. Own it with AnythingLLM. Everything you need for a powerful local-first agent experience
huggingface / transformers
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
datawhalechina / all-in-rag
🔍大模型应用开发实战一：RAG 技术全栈指南，在线阅读地址：https://datawhalechina.github.io/all-in-rag/
pipecat-ai / pipecat
Open Source framework for voice agents, multimodal apps, and realtime AI. Maintained by Daily and the community.
liustack / modlens
The first vision plugin for DeepSeek Harness, and the vision bridge for every text-only coding agent. Paste an image, get structured JSON evidence (OCR, layout, semantics). | 全网最强 DeepSeek Harness 外挂视觉插件，为 DeepSeek、GLM 等纯文本模型外挂视觉能力，粘贴图片即得结构化 JSON 证据（OCR、版面、语义）。
vllm-project / vllm-omni
A framework for efficient model inference with omni-modality models
screenpipe / screenpipe
YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context
sgl-project / sglang-omni
High Performance Serving Runtime For Interactive Super Intelligence System
lancedb / lancedb
Developer-friendly OSS embedded retrieval library for multimodal AI. Search More; Manage Less.
verl-project / verl-omni
Multimodal RL training framework for diffusion & omni models
CVHub520 / X-AnyLabeling
X-AnyLabeling: A lightweight, efficient, and unified cross-platform desktop application for annotating text, image, video, point cloud and multimodal data, combining versatile built-in tools with state-of-the-art AI models and flexible multi-format export.
microsoft / AesCode
Aesthetic code generation with decoupled cross-modal rewards: turning prompts into editable, verifiable HTML/CSS infographics, with image generators as aesthetic guidance.
yuezhiai / jonex
All-in-One Multimodal Parsing Engine + Ontology-Powered, LLM Wiki-Driven AI-Ready Knowledge Engine
AI4Finance-Foundation / FinRobot
FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Models
rockmatrix-ai / EdgeRock
EdgeRock: Strong Reasoning at Constant Memory. Multimodal models with O(N) generation and O(1) persistent history. Inference and evaluation code.
bytedance / UI-TARS-desktop
The Open-Source Multimodal AI Agent Stack: Connecting Cutting-Edge AI Models and Agent Infra
deepset-ai / haystack
Open-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search, and conversational systems.
modelscope / ms-swift
Use PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600+ LLMs (Qwen3.8, DeepSeek-V4, GLM-5.1, InternLM3, Llama4, ...) and 300+ MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM5.3, Gemma4, Llava, Phi4, ...) (AAAI 2025).
StarTrail-org / PixelRAG
https://arxiv.org/abs/2606.28344. The end of web parsing. The beginning of scalable pixel-native search. link: https://pixelrag.ai/
showlab / Awesome-Multimodal-Embodied-Agent
Survey on Multimodal Embodied Agents: Computer-Use → Robot-Use
TanStack / ai
🤖 Type-safe, provider-agnostic TypeScript AI SDK for streaming chat, tool calling, agents, and multimodal apps across OpenAI, Anthropic, Gemini, React, Vue, Svelte, and Solid.
Blaizzy / mlx-audio
A text-to-speech (TTS), speech-to-text (STT) and speech-to-speech (STS) library built on Apple's MLX framework, providing efficient speech analysis on Apple Silicon.
lance-format / lance
Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..
appletea233 / OneSearch-VL
A unified multimodal deep research agent for images and videos, with open models, datasets, and training and evaluation code.
rerun-io / rerun
Visualize, query, and stream to train on multimodal robotics data.
jordanrendric / claude-video-vision
Give Claude the ability to watch and understand videos — Claude Code plugin with frame extraction and multimodal audio analysis
zhongkaifu / TensorSharp
A native .NET LLM inference engine and agent runtime for GGUF models. TensorSharp provides a console application, a web-based chatbot interface, iPhone App, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It supports Windows/MacOS/iOS/Linux with full GPU capability
Code-Amadeus / Amadeus
Real-time multimodal desktop agent evolving toward a persistent AI OS interface (0.17 α).
caiweihan / jev-account-filter
Account screening, channel screening and classification with Jev for dataset curation and research. Find sign language, news, piano/instrument performance and makeup tutorial creators from titles and thumbnails. 视频平台工具：批量筛选账号与频道，寻找手语账号、新闻频道、乐器演奏与美妆博主。Douyin, Kuaishou, YouTube, Bilibili.
QwenLM / Qwen3-VL
Qwen3-VL is the multimodal large language model series developed by Qwen team, Alibaba Cloud.
yashab-cyber / opendroid
Your Open Autonomous Android Agent — A production-ready, self-planning AI assistant powered by local/remote LLMs and accessibility-driven screen automation.
YogurtJ / pivane
Self-hosted web workspace for the Pi coding agent on desktop, tablet & phone: subagents, scheduled tasks, MCP, memory and a bring-your-own image/video/speech lab. 自托管 AI 编程 Agent 网页工作台：多 Agent、定时任务、MCP、记忆与多媒体，电脑/平板/手机随时接着用。
apache / seatunnel
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
Dicklesworthstone / franken_surveillance_system
Pure-Rust, evidence-native, local-first sensor fabric and semantic control plane for owner-authorized cameras and capture drones
genkit-ai / genkit
Open-source framework for building agentic apps in JavaScript, Go, Dart, and Python, built and used in production by Google
NVIDIA-NeMo / DataDesigner
🎨 NeMo Data Designer: Generate high-quality synthetic data from scratch or from seed data.
DEEIX-AI / DEEIX-Chat
An enterprise AI workspace for model routing, multimodal chat, files, tools, billing, identity, and operations.
BradyFU / Awesome-Multimodal-Large-Language-Models
:sparkles::sparkles:Latest Advances on Multimodal Large Language Models
HUANGCHIHHUNGLeo / claude-real-video
Let Claude (or any LLM) actually watch a video — scene-aware, deduplicated frames + transcript, from a URL or local file. Runs locally, MIT.
ZSeven-W / dsh-crew
DeepSeek Harness (DSH) plugin: dispatch work to DSH agents from Claude Code / Codex — native subagent progress, in-host worker sessions with per-tier presets, and a multimodal bridge that lends the text-only harness vision and image generation.
TeleAI-UAGI / Awesome-Agent-Memory
Curated systems, benchmarks, and papers etc. on memory for LLMs/MLLMs --- long-term context, retrieval, and reasoning.
OmniJev / OneJev
🚀🚀 A multimodal System One decision model that gives calibrated answers to typed questions about screens, photos, video and text in one forward pass.
Anionex / agent-vision-toolkit
为纯文本模型"看图“设计更好的视觉工具箱和技能，支持多图理解，图片问答，前端UI还原、GUI 自动化等，并可选无缝接入多个主流agent，直接识别粘贴图片｜ A vision toolkit and skill designed for text-only llms — image Q&A, long-screenshot OCR, frontend UI restoration, and GUI automation, with optional seamless integration for Codex, Claude Code, Pi, Oh My Pi, and OpenCode
ysr666 / dsh-vision-router
Eyes for text-only DeepSeek Harness agents: built-in free vision chain (no key) + pixel-level vision tools (Q&A, grounding, crop, pixel diff, colors, OCR, SVG trace, cutout, screenshots). One-command install, no Python, image turns work like ordinary tool-calling turns.
allenv0 / SCM
Multimodal visual memory agent on macOS
rollroyces / py-idp
AI-enabled Intelligent Document Processing for Python: 12+ LLM backends, Pydantic schemas, self-hosted OCR (Nanonets-OCR2-3B), auto-schema discovery, HITL review, prod FastAPI server. AGPL-3.0 + commercial.
waybarrios / vllm-mlx
High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.
oxbshw / watch-skill
Give AI agents eyes, ears, and verifiable results. Watch Skill turns video, audio and screen activity into searchable, timestamped evidence and proves work with deterministic contracts, not model opinion. DeepWatch is the agent workspace built on DeepSeek Harness. Python + npm, MCP, CLI, REST, Web.
vortex-data / vortex
An extensible, state-of-the-art framework for columnar compression, and the fastest FOSS columnar file format. Formerly at @spiraldb, now an Incubation Stage project at LFAI&Data, part of the Linux Foundation.
xlang-ai / OSWorld
[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
worldbench / awesome-vla-for-ad
🌐 Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future
OpenMOSS / MOSS-VL
An open-weight 11B model series for long-form and real-time video understanding
HRuiCcc / RuiC-card-skill
全息闪卡通用 Agent Skill，不挑宿主、多模态模型都能用：一句话生成可拖转、会流光、带层次景深的 3D 闪卡网页，附可编辑 Blender 工程 · A host-agnostic Agent Skill: turn one sentence into an interactive 3D holographic card page with layered parallax, plus an editable Blender project.
SimpleJev / JevAny
Open infrastructure for training, evaluating, and deploying System 1 decision models across language and multimodal backbones.
NVIDIA-NeMo / Speech
A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)
gorse-io / gorse
AI powered open source recommender system engine supports classical/LLM rankers and multimodal content via embedding
xorbitsai / inference
Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.