ToZ-Wide AI Answering System

Yes I wanted to do this as well, currently learning how to. The best would be to self host it but we also need a good computer for that.
I think this is a very good base model: https://huggingface.co/soob3123/Veritas-12BExternal ↗ as it already have a ton of ancient philosophical knowledge baked in.
with 4bit quantization, in theory a mac studio with 32-64 gb unified ram could run 5-10 instances of this parallel, plus we can have a wait list.
Maybe a 8B parameter model would be enough as well if we give it a very sophisticated knowledge base, but it must not be retarded as well. It's a matter of trying and testing.

AFODO wrote:View source

if we give it a very sophisticated knowledge base

and a good agentic rag. These as well increase ram consumption but they are necessary.

Monophtalmos wrote:View source

The AI will function similarly to ChatGPT and answer questions people ask. Its responses will be generated from all the threads written on the forum (and of course other texts from the sites as well). When the AI chooses a response, it will prioritize sermons, replies from Clergy and Guardians, and then move on to moderators. Since it will also indicate which thread or text it is referencing, there won't be any "made-up information" issues.

I had this idea as well and wrote it to our High Priest and the SGs a few years ago when I first learned what ChatGPT was.

After 15 years of being here I just cannot answer many of the same basic topics over and over again, I find myself giving answers that are too short and lacking, whereas newer members will give a better reply (regarding cleaning aura, etc). So I realized that to have answers generated from all previous replies would be very convenient and ensure new members get their question thoroughly answered.

High Priestess Lydia wrote:View source

I had this idea as well and wrote it to our High Priest and the SGs a few years ago when I first learned what ChatGPT was.

After 15 years of being here I just cannot answer many of the same basic topics over and over again, I find myself giving answers that are too short and lacking, whereas newer members will give a better reply (regarding cleaning aura, etc). So I realized that to have answers generated from all previous replies would be very convenient and ensure new members get their question thoroughly answered.

Then it's God's willing, we will do it High Priestess!

I am all for it. I have strong interest in working on building custom AI's for websites, businesses etc and this is a good way for me to learn how's and do's.

High Priestess Lydia wrote:View source

After 15 years of being here I just cannot answer many of the same basic topics over and over again, I find myself giving answers that are too short and lacking, whereas newer members will give a better reply (regarding cleaning aura, etc). So I realized that to have answers generated from all previous replies would be very convenient and ensure new members get their question thoroughly answered.

Your support and my brothers/sisters’ eagerness to join this project makes me happy, High Priestess.

Covering all forum replies would indeed be the better option. However, the training process for the AI will be somewhat demanding.

Since there are many outdated or incorrect replies, we’ll need a clear and reliable framework of accurate information. The AI should also be able to evaluate the accuracy of what it generates before giving an answer.

_

If you deem it appropriate we can create a group conversation on the forum with those who want to take part in the project and discuss the details there.

This sounds great.

The only problem I can see, can it be flooded with wrong requests?

StraitShot47 wrote:View source

This sounds great.

The only problem I can see, can it be flooded with wrong requests?

For first version we can basically ban those who misuse it, but I think it's feasible to have an agent that decides if the prompt is worthy enough to send it to the llm.

Since we are moderating every reply anyways, might as well turn moderation into supervised training process.

Moderator should evaluate if the answer provided by the community member was helpful. This should generate training data.

This idea requires hefty investments (bare minimun tens of thousands of dollars. Specific enterprise processors alone cost thousands and tens of thousands..) and or limitations for usage, and it should be designed in a way that eliminates or greatly mitigates abuse. I am unsure if ToZ has the required resources available yet, but I 100% support the idea. Actually, I am one of those who looks forward to having a personal android assistant. We are making great strides toward that at the moment.

What is achievable now is RAG based on the books we have in library to query all of them at once with a locally running LLM. Mid-range consumer hardware will be fine for this purpose.

I'm super interested in this! Couldn't we like create a way for people to donate their hardware to temporarily power this AI, that way we wouldn't need to spend thousands on it right away? My PC is pretty powerful with almost 100GB.

Vicky<3 wrote:View source

I'm super interested in this! Couldn't we like create a way for people to donate their hardware to temporarily power this AI, that way we wouldn't need to spend thousands on it right away? My PC is pretty powerful with almost 100GB.

It could be similar to how people mine crypto, except instead for running the AI

SigTyr wrote:View source

What is achievable now is RAG based on the books we have in library to query all of them at once with a locally running LLM. Mid-range consumer hardware will be fine for this purpose.

Will consumer-grade hardware be able to handle dozens or even hundreds of prompts in a short period of time, if not simultaneously?

Henu the Great wrote:

Excactly as I thought. Personally, I see, if the hardware is less than Xeon/Threadripper/Epyc level, it gives the wrong impression of our capabilities. This is the Temple of Zeus, not our personal hobby project. I have a Threadripper 3945x and 128GB of DDR4 laying around, and plan to eventually build a Threadripper 5000-series system. That could be a powerful node for the task, but relying solely on that is a bit amateurish in my view.

Don't forget about GPU, most import part for the LLMs. The more vRAM the better. However even without GPU it will be useful.

I think we should focus on scalability of the project, rather than impressing anyone with the hardware at early stages. We must be resourceful and utilize most of the things that community can get hands on. Many of us has older hardware laying around and giving it a new purpose is a good way to reduce e-waste a little bit.

We will at some point need people who are familiar with crypto/blockchain technologies and somebody on the front end.

Vicky<3 wrote:

Brother, I'm SOOOO down for this. We should start working on the concept in further details immediately.

Project will be built around virtualization, so we can easily deploy and scale. Those who have spare computers laying around (even laptop will do) can start practicing to get comfortable for the future work.

Install Proxmox on your hardware. It will allow you to run multiple virtual systems at once on your hardware.

You will have an option to either install a full VM or a container (faster, less resources, but shared kernel and potentially less secure).

ProxMox based on Debian, so will need to get comfortable with the command line.

Also Docker will be used a lot. It can be either inside any VM on Proxmox, or can run it on your main machine.

ProxMox is free and open source, and there are community helper scripts (search this one) that can automate most of the installs.

This will be basically your homelab. It's sort of a personalized cloud with it's own network.

From there practice by coming doing some small DIY for yourselves, like a personalized media server. Or a filter for your home network (you can have openWRT as a VM and redirect traffic trough it).

Proxmox Community Helper Scripts should have plenty of options to explore. Imagine this as LEGO. Each block is a ready to use minimal OS. We will be combining these kind of blocks a lot during this project. For example one block will be RAG, another will be scheduling requests, other will be managing nodes, some others will just run LLM on VM with GPU pass trough...

The beauty of it that these blocks can be later separated and run across different machines and this how we scale.

I will be sharing these "blocks" and encourage others too if you build something that might be useful for the project.

Meanwhile I will be occasionally dropping knowledge for everyone in this thread. One may pick something and lock in. AI like Grok should help you with the basics issues at early stages, but don't trust it too much.

Let's see how it goes :)

We have already looked into this, as HPS Lydia has noted.

The other issue is that AI as of now can be browbeaten to eventually agree with the user, regardless of how strong the system prompt is. It's just in the training, and the smaller the local model, the worse it is in this regard. In addition, as noted, self-hosting the AI requires extreme hardware purchases. Going API route for something smarter like ChatGPT, requires us to monitor payments and plus all the traffic just ends up in OpenAI's hands anyways.

Henu the Great wrote:

Excactly as I thought. Personally, I see, if the hardware is less than Xeon/Threadripper/Epyc level, it gives the wrong impression of our capabilities. This is the Temple of Zeus, not our personal hobby project. I have a Threadripper 3945x and 128GB of DDR4 laying around, and plan to eventually build a Threadripper 5000-series system. That could be a powerful node for the task, but relying solely on that is a bit amateurish in my view.

Why can't it be something in between, for now? Launch it as an unofficial prototype project, and if it works then it can become more integrated. My own interest here is learning and understanding how to build this thing, not so much turning a profit. If we used tokens to represent hardware allocation to this AI, which makes sense and would allow us to perhaps even go as far as creating an actual bitcoin currency for the TOZ, officially or otherwise, I'd probably be donating it to HOO anyway. My hardware is quite new and I don't use overclocking, so I don't expect to need replacements for maybe another two years? I already run my rig pretty hard though, pretty much to it's limit most days at night for another side project.

SigTyr wrote:View source

Don't forget about GPU, most import part for the LLMs. The more vRAM the better. However even without GPU it will be useful.

I think we should focus on scalability of the project, rather than impressing anyone with the hardware at early stages. We must be resourceful and utilize most of the things that community can get hands on. Many of us has older hardware laying around and giving it a new purpose is a good way to reduce e-waste a little bit.

We will at some point need people who are familiar with crypto/blockchain technologies and somebody on the front end.

Project will be built around virtualization, so we can easily deploy and scale. Those who have spare computers laying around (even laptop will do) can start practicing to get comfortable for the future work.

Install Proxmox on your hardware. It will allow you to run multiple virtual systems at once on your hardware.

You will have an option to either install a full VM or a container (faster, less resources, but shared kernel and potentially less secure).

ProxMox based on Debian, so will need to get comfortable with the command line.

Also Docker will be used a lot. It can be either inside any VM on Proxmox, or can run it on your main machine.

ProxMox is free and open source, and there are community helper scripts (search this one) that can automate most of the installs.

This will be basically your homelab. It's sort of a personalized cloud with it's own network.

From there practice by coming doing some small DIY for yourselves, like a personalized media server. Or a filter for your home network (you can have openWRT as a VM and redirect traffic trough it).

Proxmox Community Helper Scripts should have plenty of options to explore. Imagine this as LEGO. Each block is a ready to use minimal OS. We will be combining these kind of blocks a lot during this project. For example one block will be RAG, another will be scheduling requests, other will be managing nodes, some others will just run LLM on VM with GPU pass trough...

The beauty of it that these blocks can be later separated and run across different machines and this how we scale.

I will be sharing these "blocks" and encourage others too if you build something that might be useful for the project.

Meanwhile I will be occasionally dropping knowledge for everyone in this thread. One may pick something and lock in. AI like Grok should help you with the basics issues at early stages, but don't trust it too much.

Let's see how it goes :)

Huh, this is new territory for me. That's good, I didn't want this to be easy. I have another week until school starts, and I reasonably believe learning this will be useful, so I will be. Thank you for explaining how to get started with the practical work to be done.

Personally, even if the TOZ never decides to use this, I know that I personally will. An AI that uses TOZ as a foundation instead of all the crap on the wider web sounds like an incredibly useful thing, like having someone who has read and remembers every single post ever written ever... I was considering doing this solo since I started through localization.

I had AI review your plan, and ya it checks out. It's complicated though, but it sounds solid and is the standard high end method of doing this. What bothers me is there's vulnerabilities in this system once it becomes functionally collaborative from sharing a kernel. For that reason, we can't just allow anybody to participate without some way to verify our work. Also, it sounds like your quality of work is going to be higher than some less experienced people like myself, which must be considered also.

If we get serious on this, we need someone to step up as a project administrator that actually has skills to prevent people from sabotaging this project [which they have reason to], on purpose or otherwise, and see this on a bigger picture in terms of whether our work actually "works" together as a greater whole. It's clear that if this was an actual TOZ project, that position would be appointed by them and likely go to ApolloAbove as head of IT (?)- but this isn't the case here clearly. I'm saying this not to nominate myself, I'm very much unqualified for that lol. Hopefully people aren't blowing hot air on how knowledgeable they are on this?

ApolloAbove [SG] wrote:View source

We have already looked into this, as HPS Lydia has noted.

The other issue is that AI as of now can be browbeaten to eventually agree with the user, regardless of how strong the system prompt is. It's just in the training, and the smaller the local model, the worse it is in this regard. In addition, as noted, self-hosting the AI requires extreme hardware purchases. Going API route for something smarter like ChatGPT, requires us to monitor payments and plus all the traffic just ends up in OpenAI's hands anyways.

With all due respect I assume that you may have looked into the idea of having an AI system in it's traditional sense, not into the community hosted and developed kind of project. Correct me if I am wrong.

For now can do just do a good RAG that will be available for everyone to use personally with mid-range hardware. (At home, offline)

The LLM's role would be essentially to summarize the text provided by this RAG. Community members would be able to test different models so we can figure out which of them if any can handle this job.
8b models, need at least 8GB VRAM. NVIDIA RTX 3060/RX 6700 XT both have 12GB, either is a decent pick.

RX 6800 has 16GB VRAM is good for the job and can run models up to 14b parameters (or smaller model, but with a bigger context window) Used ones can be obtained for less than 400$.

If we decide to use this for ToZ, then this RAG system can compliment current search function, as for now it only searches across the website, not the content of PDF library.
This system might provide additional information in a form of summary and point to a book/webpage.

At early stages, it will not have chatting capabilities at all to reduce the load on hardware, but it will help to decide if it's worth it to proceed further.

From there user input and the response can be used to produce training data (after manual validation by high ranking members) , so we can have our own LLM if required, that later on, potentially can be used on forums.

If buying GPU is not an option, consumer grade GPU can be rented too, but maybe we can count on the community later on... I am thinking of P2P GPU nodes running by the members and some sort of hub-server that will ping available nodes and submit the query to them. Basically this will solve the problem of extreme hardware purchases, but the project will grow only if this community would be genuinely interested and actively participating.