LLM Showdown - NVIDIA A4000 vs 4000 SFF ADA - Which REALLY Reigns

Описание: RTX A4000 vs RTX 4000 SFF ADA – does the new SFF generation beat the original full size A4000 in LLM workloads?

In this video, we put these two professional graphics cards head-to-head and break down their real-world performance, LLM inference benchmarks and memory differences.

⚡ The new RTX 4000 SFF ADA comes with 20GB of RAM and only 70W power draw compared to 140W on the RTX A4000 – but with reduced memory bandwidth. Does that efficiency tradeoff pay off in real workloads? We’ll find out.

My Links 🔗
👉🏻 Subscribe:    / @thenittygritty  
👉🏻 BlueSky: https://bsky.app/profile/techgrandpa....
👉🏻 GitHub: https://github.com/tech-grandpa

⬇️Chapters in case you want to skip ahead ⬇️
00:00 - Prelude
00:48 - Why a new graphic card?
02:33 - The contender introduced
04:18 - The setup - could you guess which one is faster?
06:57 - Which one is faster? - First results
09:48 - The problem with mainstream testing (with regards to LLM inference performance)
10:35 - Expanded test setup - more cards added
13:04 - Second set of test results
15:20 - Fictional Dream Card
17:10 - Closing words

Errata
the model used for testing was neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8