With enough RAM and ideally a good GPU you can run smaller models (~8B Parameters) locally on your own device.
affenlehrer
joined 1 week ago
With enough RAM and ideally a good GPU you can run smaller models (~8B Parameters) locally on your own device.
I really recommend watching this introduction by Andrej Karpathy https://www.youtube.com/watch?v=7xTGNNLPyMI
One part that really stuck with me is that the data in the model is more like a fading memory but the stuff in the context window is more like the working memory. Since I learned that I tend to put as much information as possible into the context window before asking questions about it. This improved the results drastically and reduced hallucinations.