Don't forget about GPU, most import part for the LLMs. The more vRAM the better. However even without GPU it will be useful.
I think we should focus on scalability of the project, rather than impressing anyone with the hardware at early stages. We must be resourceful and utilize most of the things that community can get hands on. Many of us has older hardware laying around and giving it a new purpose is a good way to reduce e-waste a little bit.
We will at some point need people who are familiar with crypto/blockchain technologies and somebody on the front end.
Project will be built around virtualization, so we can easily deploy and scale. Those who have spare computers laying around (even laptop will do) can start practicing to get comfortable for the future work.
Install Proxmox on your hardware. It will allow you to run multiple virtual systems at once on your hardware.
You will have an option to either install a full VM or a container (faster, less resources, but shared kernel and potentially less secure).
ProxMox based on Debian, so will need to get comfortable with the command line.
Also Docker will be used a lot. It can be either inside any VM on Proxmox, or can run it on your main machine.
ProxMox is free and open source, and there are community helper scripts (search this one) that can automate most of the installs.
This will be basically your homelab. It's sort of a personalized cloud with it's own network.
From there practice by coming doing some small DIY for yourselves, like a personalized media server. Or a filter for your home network (you can have openWRT as a VM and redirect traffic trough it).
Proxmox Community Helper Scripts should have plenty of options to explore. Imagine this as LEGO. Each block is a ready to use minimal OS. We will be combining these kind of blocks a lot during this project. For example one block will be RAG, another will be scheduling requests, other will be managing nodes, some others will just run LLM on VM with GPU pass trough...
The beauty of it that these blocks can be later separated and run across different machines and this how we scale.
I will be sharing these "blocks" and encourage others too if you build something that might be useful for the project.
Meanwhile I will be occasionally dropping knowledge for everyone in this thread. One may pick something and lock in. AI like Grok should help you with the basics issues at early stages, but don't trust it too much.
Let's see how it goes :)
I decided one of the major questions we had to solve is assessing the CPU, GPU and RAM of all devices, so that this can be communicated and recorded by other systems. I wrote this in python, I assumed that was intended. I normally work with java script, but in this case python is too dramatically superior not to use.
At first I created a script that used too many dependencies and collected way too much information about my device. I was able to see absurd things like my IP address and even that I had an inactive VPN at the time. The version I settled on is much simpler and generates a device ID number without collecting hardware serial numbers.
Conceptionally, the block I created is run on each device to measure total, available and later on consumed resources. Other systems can request a diagnostic report for each device whenever needed. It appears this is necessary for effective management of resources. People will likely want to record their own copy of these diagnostic reports that go into more detail than the block currently does, I know I definitely will be.
I'm going to work on another block now, likely for managing multiple VMs based on the collected data.
I'm willing to share the scripts I've already worked on, not sure how I should submit them tho. Obviously, people should look at what the script does before they run it, even if they have it analyzed by an AI.
So ya this was already a lot of fun, hopefully people are interested in putting actual work into this and don't just like talking about it.
