The embedding models at the heart of txtai can be small enough to run on intel CPUs from ten years ago. It's extremely frustrating when HN commentators don't do even the most basic research into the product that they are critiquing.

It’s frustrating when people ask for hardware requirements without stating what they are trying to do, do you have 100,000,000 books to index or do you have 5 articles? What are the context lengths you need? What about latency?

How can someone tell you what hardware you need when you give literally no information about what you’re trying to do?

There's a difference between "how many CPU-hours will my task need" and "how much memory does this program use to even start up".

Having some idea of the task will guide the choice of model, which will be an enormous factor in memory use (I.e. whether it will startup or not)

Do you need a 70b param model or a 7b model? Theres thousands and thousands of dollars hardware difference there

With no idea of the task, one can’t even ball park it

This particular tool has a page listing recommended models: https://neuml.github.io/txtai/models/

A RTX 3090 is more than enough for 7B LLMs. With 4-bit quantization, you can run inference with an even larger LLM using a 24GB GPU.

If you're using remote API services, you might be able to just use a CPU.