trymirai / uzu

# uzu
A high-performance inference engine for AI models. It allows you to deploy AI directly in your app with **zero latency**, **full data privacy**, and **no inference costs**. Key features:
- Simple, high-level API
- Unified model configurations, making it easy to add support for new models
- Traceable computations to ensure correctness against the source-of-truth implementation
- Utilizes unified memory on Apple devices
- Broad model support
## Quick Start
 Rust 
Add the dependency:
 [code] 
Run the code below:
 [code] 
 Python 
Add the dependency:
 [code] 
Run the code below:
 [code] 
 Swift 
Add the dependency:
 [code] 
Run the code below:
 [code] 
 TypeScript 
Add the dependency:
 [code] 
Run the code below:
 [code] 
Everything from model downloading to inference configuration is handled automatically. Refer to the documentation for details on how to customize each step of the process.
## Examples
You can run any example via `cargo tools example` \ \ :
### Chat
In this example, we will download a model and get a reply to a specific list of messages:
 Rust 
 [code] 
 Python 
 [code] 
 Swift 
 [code] 
 TypeScript 
 [code] 
 Once loaded, the same `ChatSession` can be reused for multiple requests until you drop it. Each model may consume a significant amount of RAM, so it's important to keep only one session loaded at a time. For iOS apps, we recommend adding the Increased Memory Capability entitlement to ensure your app can allocate the required memory.
### Chat with the cloud model
In this example, we will get a reply to a specific list of messages from a cloud model:
 Rust 
 [code] 
 Python 
 [code] 
 Swift 
 [code] 
 TypeScript 
 [code] 
### Chat with shared instance
This example shows how to reuse chat instance without reloading model into memory:
 Rust 
 [code] 
 Python 
 [code] 
 Swift 
 [code] 
 TypeScript 
 [code] 
### Chat with structured output
Sometimes you want the generated output to be valid JSON with predefined fields. You can use `Grammar` to manually specify a JSON schema for the response you want to receive:
 Rust 
 [code] 
 Python 
 [code] 
 Swift 
 [code] 
 TypeScript 
 [code] 
### Tool calls
This example shows how to use external tools:
 Rust 
 [code] 
 Python 
 [code] 
 Swift 
 [code] 
 TypeScript 
 [code] 
## Development
`uzu` is a native Rust crate with bindings available for:
- `Swift` via uniffi-rs
- `Python` via pyo3
- `TypeScript` via napi-rs
It supports:
- Backends:
 - `metal`
 - `cpu`
- Targets:
 - `aarch64-apple-darwin`
 - `aarch64-apple-ios`
 - `aarch64-apple-ios-macabi`
 - `aarch64-apple-ios-sim`
 - `aarch64-pc-windows-msvc` _(in progress)_
 - `aarch64-unknown-linux-gnu` _(in progress)_
 - `wasm32-wasip1-threads` _(in progress)_
 - `x86_64-apple-darwin`
 - `x86_64-pc-windows-msvc` _(in progress)_
 - `x86_64-unknown-linux-gnu` _(in progress)_
For initial setup we recommend running cargo tools setup , which installs all necessary dependencies ( rustup , uv , pnpm , Rust targets , Metal toolchain , ...) if not al