miltos22 / llama-wackMall

git clone https://github.com/miltos22/llama-wackMall.git

Caching experts made model agnostic for llama. Drastically speeds up generation speed on MoE models that fully fit in the RAM but partially in Vram. WIP: Same speedup but for models that don't fit in the ram either.