💾🗂️ Definitive system for total backup of the entire forum 🔄

[չ∂αяквℓσω]

​

Hi everyone, I've been working on this for a while, and as promised, I've finally managed to create the definitive system for backing up the entire forum, all threads/posts, etc.

In previous posts, I've already described the crucial importance of a backup, and I'm extremely happy with the success of this project.

Now, I'll show you an example and a video tutorial on how I performed the procedure:

The script in question is written in Python, but this time I also managed to convert it into a Linux executable, so you can directly run the interpreted Python code by installing the libraries with "pip3 install <library>" and "python3 the_script.py" It's highly recommended to limit excessive RAM usage with: "ulimit -v 1000000 && python3 the_script.py" (this prevents excessive RAM usage, setting it to a maximum of 1GB).

There are two scripts in question. The first, the main one, is used to download an entire forum section at will. There are also counters to let the user know when it's about to finish and how much is left. The second script is very interesting. It essentially allows the complete conversion of .html files into a single .txt file, removing unnecessary content and massively compressing the already optimized files, allowing for unprecedented space savings. And here's the best part:

The total forum download (CSS omitted since it's useless) currently weighs about 13GB, which is too large. With a new and advanced .zip compression system, I managed to compress the entire file, and it weighed a total of 2.7GB.

Yes, can you believe it? Almost a million messages, etc., weighing from 13GB to 2.7GB.

But it doesn't end there, because thanks to the second script, which runs a highly optimized conversion system, I was able to convert all the .html files into a few .txt files based on sections, and without compression, it didn't even give me 1GB. And most importantly, after compression (since the .zip system is highly optimized for compressing .txt files), the forum's total size came to only 266.6MB (what a coincidence, 66.6)!!!

Yes, we've gone from about 13GB at the start to 266.6MB on the way, and anyone can download it, anyone. It's so small now that it's as big as a large Android app. Even on a slow connection, a USB stick, anything, you can archive the entire ToZ forum. I'm not kidding. At first, I couldn't believe it either. I didn't even think I could do this. And in the end, I managed to make full forum backups accessible to everyone, and we'll guarantee that our content will be preserved forever!

I'm extremely happy because I managed to keep my promise!

Here are the download mirrors, which are the entire folder where I also have other projects. I've merged everything into one (if they get banned, I'll obviously try to update them as soon as possible):

https://www.mediafire.com/folder/y90mvxjz2b5vr/Satanas

https://mega.nz/folder/E4I1xKRJ#yP9Aj9DHKA1wsfbDhdsVcw

Hail Zeus!

 

[AFODO]

Interesting @ApolloAbove [TG]

 

[Isothemides]

You have many great ideas, it would be wonderful to contact ApolloAbove, mention your projects, and ask him if you can help with anything internal.
The way we move forward is with coordination and your skills could be much appreciated. Be Blessed Brother.

 

[NG SATchives]

Thank you I'll also use this to back up the forums on my end as well.

 

[Isothemides]

ApolloAbove

*Guardian ApolloAbove
Didn't notice that small error, my mistake!

 

[Chandra [NG]]

This is a very valuable contribution, thank you. After closer review and approval, I will likely add these backups to the Library of Thoth when the updated site is released. I truly appreciate your efforts.

 

[չ∂αяквℓσω]

You have many great ideas, it would be wonderful to contact ApolloAbove, mention your projects, and ask him if you can help with anything internal.
The way we move forward is with coordination and your skills could be much appreciated. Be Blessed Brother.

Thank you so much, it is an honor for me to help our community.

As soon as I have any other ideas, I will not hesitate to post them on the forum.

 

[չ∂αяквℓσω]

VERY IMPORTANT NEW UPDATE!!!

Finally after a long time I returned to work for ToZ, unfortunately having little time available due to my commitments regarding study and work in general I have been less and less active here.

And here we are now with a new extremely important update, I have now decided to do things differently, from today onwards all ToZ software will have a graphical interface (GUI) and will be simplified making them useful to all types of users here, from geeks to average users. And in fact I also made a decision, the software will no longer only be distributed in Python, but also in executable format for both Linux and Windows (.exe), however, not having 32 bit computers I couldn't compile the script for 32 bit systems, even if now the vast majority of computer systems today are almost always 64 bit, so I think there are no problems. The reason I decided to make the software compatible for Windows too is because unfortunately many users, even for work, are forced to use proprietary software that is not compatible with Linux (which is a huge injustice) and not even with Wine. In the future I will also make a guide on how to lock down the Windows system, deactivating telemetry and obtaining an excellent privacy system there too (regards users who for one reason or another cannot abandon Windows).

But now returning to the software, I have decided to unify all the most useful main functions of the tools created by me in a single tool, the most important part concerns backups, today it is extremely important to have a backup of everything, this also applies to the forum, in fact this is an evolution of the old backup tool created previously which also includes an advanced system for making backups of entire web portals, and yes, you can also easily make an entire backup of the ToZ, translators can use it both to make backups and to unify the graphic interface in a modern way and translate it into their language. A long time ago when it was still JoS there was a button to download the entire website, unfortunately I don't know what happened to it, perhaps to save space they may have removed it, and I think it also applies to the new library, however with the automation script there is no longer this problem, the software uses a multi-threading system to download very quickly and also has excellent protection against memory leaks and buffer overflows.

Instead, regarding the main function (the most important) is that of being able to download entire sections of the forum and make a complete backup of them, from today some very important improvements have been made, first of all now you can also download all the media such as images, reactions, emojis, videos, etc.. Then instead of the total conversion into a single .txt file, there is a conversion system into a single .html file which is also much more efficient than .txt, the interface is now much nicer, more intuitive and the media are important in base64 within the html itself, and there is also a very efficient navigation system between the various posts, if you don't feel like downloading and converting there is already a special function to download and convert specifically without doing anything else.

Among the added functions there is the one that could perhaps prove to be the most useful for administrators, in fact there is a system for converting the .html file (also compatible with media) into .json or .csv, it serves to create a support system for recovery on vbulletin-based forums, in this way those who are practical could easily adapt it to recover the entire forum (at frontend level, for the backend obviously the user databases are not touched) if bugs, very serious problems, hacker attacks and more, in this way even individual users would contribute in an extremely efficient way to guaranteeing the eternal persistence of the information present here and an unparalleled resilience (this was done by trying to make the software more accessible by creating the gui while also making it multi-platform).

I also signed the .exe file to avoid false positives with antivirus, unfortunately not having a Mac and having problems with the virtual machine I was unable to compile it for Mac, (even if it is not recommended for all of us to already use Windows, let alone an even more closed and dangerous system like Apple).

Small note: Unfortunately on Archive.org they removed the software for an unspecified reason, Archive has also been causing problems lately, but alternatively I also uploaded it to a Github repository, so you can also check the sources, I also opened a ProtonDrive account to try to create better alternatives and obviously I also uploaded it to the classic Mega and Mediafire.

Useful links:

GitHub - darkblows/arcane-archiver: Contain a simple software

Contain a simple software. Contribute to darkblows/arcane-archiver development by creating an account on GitHub.

github.com

59.41 MB folder on MEGA

3 files

mega.nz

MediaFire

MediaFire is a simple to use free service that lets you put all your photos, documents, music, and video in a single place so you can access them anywhere and share them everywhere.

www.mediafire.com

Proton Drive

Securely store, share, and access your important files and photos. Anytime, anywhere.

drive.proton.me

​

 

[չ∂αяквℓσω]

I made another video example showing how to use the software, but unfortunately JewMotion deleted it.
I uploaded another one and am now uploading a copy here as well.