I consolidated my setup a bit. This is my local LLM hosting server. I took a gamble on a Chinese NVLink SXM2 mezzanine board from ebay, it was surprisingly plug and play for my dual 16GB v100s 😂. I’m also running a Tesla P40 that I repasted with liquid metal that sits at 24C idle 🥶.
Nice, dude. Love anyone who learns by doing.
It’s been a fascinating experience so far and I feel like I’m only touching the surface. I’m exploring some of the various memory harnesses to hook into Hermes Agent and see how it learns with extended use.
From the first photo I thought this was an earthquake simulator with models of skyscrapers
😂 I can see how it looks that way lol
What kind of model and space limitations are you under with v100s?
Some of the most interesting computer music I’ve heard in years was composed on them but idk if it’s worth getting into a whole new generation of hardware if it can only really do that.
Currently I’m running a Q6K quant of Hermes 4 14B with a 32K context window via llama.cpp that works pretty well. Generation output is a comfy ~50tok/sec. These v100s are 16GB each, but there are 32GB versions available too.
I’m running everything via NixOS and have to do package overrides to get inference engines to build with the right CUDA versions.
My goal is to get a cohesive environment set up for Hermes Agent to learn my system/lab/network and help my grow it over time.
Overall, I’m happy with them. The mezzanine board is good quality, I’m using PTM sheets under those massive heatsinks and some arctic p9 fans to keep them at around 60C under load.
That’s great! But I’m even more curious about the green device in the background of the first pic 👀 What’s that?
It’s an older version of a heatset insert fixture. I used it to put threaded brass inserts into my 3d prints instead of trying to capture a nut or screwing directly into plastic.
https://www.printables.com/model/609644-stealth-press-1-heat-set-insert-press-legacy
