Llama-2 13B Chat Deprecated

For the past several months I've been building Atenia Engine — a

from-scratch LLM inference runtime in Rust. The goal: run models

on hardware where other engines fail, without sacrificing

mathematical correctness.

Today it runs Llama 2 13B Chat (26 GB, BF16) on a laptop with

8 GB VRAM and 32 GB RAM. The model doesn't fit in VRAM. It barely

fits in RAM. Atenia handles it by moving tensors intelligently

between VRAM, RAM, and NVMe as execution proceeds.

**The result, measured on real hardware:**

- Load: 26 GB in ~167s (~156 MB/s from NVMe)

- Forward: 200s

- Force 50% LRU spill to NVMe (866 tensors, 13 GB): 19s

- Post-spill forward (lazy restore): **23s**

- argmax(pre-spill) == argmax(post-spill) == 1, logit 4.7747

- **[PASS] ✓ bit-exact — the spill+restore cycle is

mathematically transparent**

Total: ~7 minutes. Reproducible with one command.

**What makes it different from llama.cpp / vLLM / mistral.rs:**

Those are great projects optimized for throughput and ease of use.

Atenia's priority is different: *verifiable correctness first,

then performance*. Every model is validated against F64

mathematical ground truth — not against other frameworks.

Across four production checkpoints, Atenia F32 is between

4,096× and 9,692× closer to mathematical truth than standard

PyTorch BF16 inference.

**Performance (post-M4.8 AVX2+FMA+matrixmultiply stack):**

| Shape | Before | After | Speedup |

|---|---|---|---|

| 4×5120×13824 (MLP gate/up) | 1,954 ms | 39 ms | 49.5× |

| 1×5120×5120 (Q/K/V/O) | 175 ms | 13 ms | 13.4× |

| 1×4096×32000 (LM head) | 694 ms | 76 ms | 9.2× |

The engine was routing all MatMul through a scalar triple-loop

because of a lexicographic string comparison bug in the APX mode

dispatcher (`"4.19" >= "6.3"` evaluates false). Fixing it +

adding AVX2/FMA runtime dispatch + matrixmultiply for

cache-blocked sgemm gave the gains above. Vendor-agnostic:

works on Intel and AMD, NEON-ready for Apple Silicon (v24

roadmap).

**Reproduce it yourself:**

```bash

git clone https://github.com/AteniaEngine/ateniaengine.git

cd ateniaengine

huggingface-cli download meta-llama/Llama-2-13b-chat-hf \

--local-dir ./models/llama-2-13b-chat \

--include '*.safetensors' '*.json' 'tokenizer*'

cargo install --path .

atenia run --mode c \

--model ./models/llama-2-13b-chat \

--cache-dir ./atenia-cache

```

Hardware requirements: x86-64 with AVX2+FMA (Intel Haswell+

or AMD Excavator+), 32 GB RAM, NVMe for the cache dir.

The model download requires a free Meta license

(one-click on HuggingFace).

**What it doesn't do yet:**

- No tokenizer, no KV cache, no text generation (M5, next

milestone)

- 13B forward runs on CPU — the GPU MatMul pool (64 MB blocks)

is too small for 13B-scale tensors; non-pooled cudaMalloc

is M5+

- No quantization support (GGUF etc.)

**Stack:** pure Rust, ~1,200 #[test] functions, Apache 2.0.

Repo: https://github.com/AteniaEngine/ateniaengine

Happy to answer questions about the architecture, the

correctness methodology, or the beyond-VRAM approach.

[**Ruter AI Lab**](https://ruter.no/) released Norwegian 13b model: [RuterNorway/Llama-2-13b-chat-norwegian · Hugging Face](https://huggingface.co/RuterNorway/Llama-2-13b-chat-norwegian)

My ggml quant: [https://huggingface.co/NikolayKozloff/Llama-2-13b-chat-norwegian/resolve/main/Llama-2-13b-chat-norwegian-Q6\_K.bin](https://huggingface.co/NikolayKozloff/Llama-2-13b-chat-norwegian/resolve/main/Llama-2-13b-chat-norwegian-Q6_K.bin)

Update: Developers added GPTQ version.

[RuterNorway/Llama-2-13b-chat-norwegian-GPTQ at main (huggingface.co)](https://huggingface.co/RuterNorway/Llama-2-13b-chat-norwegian-GPTQ/tree/main)

I am about to embark on experimenting with "[RAG on Windows using TensorRT-LLM and LlamaIndex](https://github.com/NVIDIA/trt-llm-rag-windows#building-trt-engine)".

Since I have an RTX 4070, it is written in Nvidia's instructions that **I need to build the TRT Engine based on LLaMa 2 13B chat-hf and LLaMa 2 13B AWQ int4**.

I have already obtained access to the HF model.

Nvidia says, of course, that **I have to download the LLaMa 2 13B chat-hf model (**[this is the link](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf/tree/main)**)**, but on the HuggingFace page the model is divided into **three .safetensors files and three .bin files referring to the pytorch version**.

https://preview.redd.it/h0cjk8ur5m9c1.png?width=1584&format=png&auto=webp&s=12f0888949a7d20694d63a28f72121b079bec4ef

What should I do about this?

How do I "download" the LLaMa 2 13B chat-hf model as it is indicated by Nvidia?

Thank you.

Hello

I'm using Axolotl to fine-tune `meta-llama/Llama-2-13b-chat-hf.` How should I choose the value for `warmup_steps` and for `val_set_size` in the config yaml file of Axolotl? In the example config files 10 warmup steps and a val set size of 0.05 is used but others also used 100 warm up steps and 0.01 or 0.02 for val set size. I have a dataset with around 3800 samples.

Hi. A few days ago i tried to make ggml version for previously released Dutch model: [Mirage-Studio/llama-gaan-2-7b-chat-hf-dutch-epoch-5 · Hugging Face](https://huggingface.co/Mirage-Studio/llama-gaan-2-7b-chat-hf-dutch-epoch-5) but didn't succeded cause i don't know how to set pad\_token\_id to required value. (The model's developers mentioned this requirement in model card but didn't give explanations for newbies like me.)

Anyway today i'm happy because i successfully made ggml for most fresh Dutch model: [BramVanroy/Llama-2-13b-chat-dutch · Hugging Face](https://huggingface.co/BramVanroy/Llama-2-13b-chat-dutch) Here it is: [https://huggingface.co/NikolayKozloff/Llama-2-13b-chat-dutch/resolve/main/Llama-2-13b-chat-dutch-Q6\_K.bin](https://huggingface.co/NikolayKozloff/Llama-2-13b-chat-dutch/resolve/main/Llama-2-13b-chat-dutch-Q6_K.bin)

So i want to share my happiness with LocalLLaMa members and hope that ggml version will be usefull for guys who learn Dutch language. Cheers.