Does lemmy have any communities dedicated to archiving/hoarding data?
Be smart and keep it all on thumb drives.
Okay so where do I find some cheap hard drives? Europe if possible :-)
If you do this please share your IP so I can use your backup too
You can find me at ::1
Neither are that bad honestly. I have jigdo scripts I run with every point release of Debian and have a copy of English Wikipedia on a Kiwix mirror I also host. Wikipedia is a tad over 100 GB. The source, arm64 and amd64 complete repos (DVD images) for Debian Trixie, including the network installer and a couple live boot images, are 353 GB.
Kiwix has copies of a LOT of stuff, including Wikipedia on their website. You can view their zim files with a desktop application or host your own web version. Their website is: https://kiwix.org/
If you want (or if Wikipedia is censored for you) you can also look at my mirror to see what a web hosted version looks like: https://kiwix.marcusadams.me/
Note: I use Anubis to help block scrapers. You should have no issues as a human other than you may see a little anime girl for a second on first load, but every once and a while Brave has a disagreement with her and a page won’t load correctly. I’ve only seen it in Brave, and only rarely, but I’ve seen it once or twice so thought I’d mention it.
For wikipedia you’ll want to use Kiwix. A full backup of wikipedia is only like 100GB, and I think that includes pictures too.
120GB not including Wikimedia 😉
Also, I wish they included OSM maps, not just the wiki.
You can easily download planet.osm, I think it’s a couple of TB for the compressed file.
I wonder if there’s anyways to edit these files afterwards? They tend to be read only, right? I must confess, I don’t have too much experience with this myself.
It’s probably hundreds of thousands of HTML files, no? What is the fear about being able to edit or not?
I believe kiwix uses zim files.
Okay, I’m unfamiliar with both. Well, I still don’t understand why read-only state matters; are you concerned about tampering?
Well I think it would be cool to be able to fix/edit any inaccurate articles, or pages that may have been messed with by trolls, or to update with more up to date info.
Yeah also if you make a Zim wiki or convert a website into Zim then you can run that stuff too. If you use Emacs it’s easy to convert some pages to wikitext for Zim too
Last time I updated it was closer to 120GB but if you’re not sweating 100 GB then an extra 20 isn’t going to bother anyone these days.
Also, thanks for reminding me that I need to check my dates and update.
EDIT: you can also easily configure a SBC like a Raspberry Pi (or any of the clones) that will boot, set the Wi-Fi to access point mode, and serve kiwix as a website that anyone (on the local AP wifi network) can connect to and query… And it’ll run off a USB battery pack. I have one kicking around the house somewhere
Just built one of those using Dietpi as the OS and NVME M.2 for the storage. I have many different ZIMs and running different services and only using about 270GB.
Works great for offline use. Probably should add an ISO or 2 as well.
What other services are you running?
@fmstrat@lemmy.world asked what else I was running in a sibling comment to yours and I didn’t have an answer because I’m not… yet : )
DietPi makes it dead simple to run most of these things as their “software suite” is pretty robust and simple to setup.
For “user facing” applications:
- Homer Dashboard as the landing page when going to the .local address in a browser
- Kiwix for the ZIMs
- Hedgedoc for personal note taking/wiki
- Lychee photos for a very lightweight photo album maker/viewer for keepsake photos.
For “admin side” stuff:
- Portainer to manage the containers/stacks
- Watchtower to auto-update the containers while they’re still network connected
- Transmission daemonized to download and seed the ZIMs or anything else non-pirate related
- Use jojo2357’s ZIM updater to auto-update ZIMs via cron job while they’re still network connected
- DietPi-Dashboard as an all-in-one dashboard to monitor and control the RPi from a web interface. (Yeah I know I can do everything SSH’ing in but I’m lazy.)
- File Browser just in case I want other people to have access to files but since it’s in maintenance mode and I’m unsure I want others to have access, might strip it out
I try to use containers from LinuxServer.io whenever possible. Mostly just cause it’s what I do on my main server.
I’m still looking at adding/removing things as I get more time to sit down but I’m pretty happy with it’s current state.
Do you recommend adding anything else to it?
For instance, OSM maps?
I’ve been thinking about running the Kiwix app + OSMAnd on an old Android phone and auto updating it once a year.
That’s a good question (and good idea) that I hadn’t really thought about past a collection of ZIMs. The one I built advertises it’s own AP SSID that anyone can connect to and then access the ZIMs that are served via
kiwix-serveon HTTP/80. That is, I wanted a single, low power, headless device that multiple people could use simultaneously via wifi and browser rather than a personal device.I hadn’t really thought about other helpful services past that. I mean, we’ve got a (wee) server so why not use it? I like the idea of OSM and their website is open source but has a lot of dependencies :
openstreetmap-website is a Ruby on Rails application that uses PostgreSQL as its database, and has a large number of dependencies for installation
A fully-functional openstreetmap-website installation depends on other services, including map tile servers and geocoding services, that are provided by other software. The default installation uses publicly-available services to help with development and testing.
I wonder how hard it would be to host everything it needs locally/offline… and what that would do to power consumption : )
Thanks for the idea - something to look into, for sure.
Saw your comment on mine and finally saw this one.
I’m gonna take a look at openstreetmap-tile-server and see about running that since if all has gone to shit, who knows if GPS will work. Least it’s almost like a paper map and can be auto-updated as long as we still have internet. Quick Gist someone wrote here.
Yeah, I feel the same in that it’s assuredly doable, but how hard is it?
If you’re able to dig into and make some progress, please tag me because I’m interested but don’t have much time these days.
I stumbled across this sort of fascinating area of doomsday prepping a few weeks back.
A nice addition to that, don’t just make it a USB, but a raspberry pi. So you’d have a reasonably low-powered computer you could easily take with you.
Not suggesting this one as it seems a bit expensive to me, but https://www.prepperdisk.com/products/prepper-disk-premium-over-512gb-of-survival-content?view=sl-8978CA41
Just built one of these myself. I went NVME M.2 instead of SD Card to avoid data corruption. I know SD Cards are fine if you don’t write to them a lot but if you wanna update or add your own stuff, scares me. Plus NVME is just so much faster.
How would you access the info if electricity permanently goes out?
Pretty much what Sinthesis said; USB power brick and/or solar panels. Both at the ready and tested. Also got a big ass battery backup that will charge off solar panels.
You find a generator, or solar panels, or wind mill, or water turbine, or a bicycle hooked up to a generator.
If electricity permanently goes out then we’re in a scavenger situation and it is time to start taking apart things that are no longer necessary to build the things that are.
The English Language Wikipedia probably wouldn’t be hard, or Debian Stable.
All of Debian’s packages might be a tad more expensive, though.
It depends if you want the images or previous versions of wikipedia too. The current version is about 25Gb compressed, the dump with all versions is aparently multiple terabytes. They don’t say how much media they have, but I’m guessing it’s roughly “lots”.
This might be a good place to start for Wikipedia;
https://meta.wikimedia.org/wiki/Data_dump_torrents#English_Wikipedia
And the english with no pictures is even smaller
And you can use Kiwix to setup a locally hosted wikipedia using the data dumps
Did I miss something? Whats happening to debian stable?
How would one go about making an offline copy of the repos? Asking for a friend.
man rsync man cron xdg-open \ https://lmgtfy.app/?q=debian+rsync+mirrorStart from here https://wiki.debian.org/DebianRepository/Setup
Arch: https://wiki.archlinux.org/title/DeveloperWiki:NewMirrors
The official repo is only about 80GB, I have an old copy from when I was running an airgapped system. Not sure about the AUR, it’s probably in the TBs range though.
AUR might not be as big as you think. It would wouldn’t work for offline tho since many AUR packages pull archives from websites during the build process.
Oh yeah, didn’t think about that. Having a bunch of PKGBUILD files isn’t very useful.
I guess you could compile all the things, if you had a lot of spare processing power. A once a year snapshot would probably be enough for anything short of a Mad Max future.
That future might not be far off considering what Trump did today. Balance of power is seriously about to shift.
I saw that Wikipedia was having funding problems, what happened to Debian?
They lie. Wikipedia has plenty of money. Do not give those parasites any more.
https://en.wikipedia.org/wiki/Wikimedia_Foundation#Spending_and_fundraising_practices
This post foreshadowed today’s AWS outage.
👀
I also recommend downloading “Flashpoint archive” to have flash games and animations to stay entertained.
There is a 4gb version and a 2.3TB version.
There is a 4gb version and a 2.3TB version.
That’s quite the range
When I downloaded it years ago it was 1.8TB. It’s crazy how big the archive is. The smaller one is just so it’s accessible to most people.
Is that Flash exclusive or do they accept other games from that era?
I’m not sure, but I do think it’s just flash
I downloaded wikipedia a month or two ago, I recommend it.
How big is Wikipedia?
If you don’t care about edit history, and only care about English, there are zim files w/ images for <150 GiB
I thought the whole point of torrenting was to decentralise distribution. I use torrents to get my distros.
In my own little bubble, I thought that’s how most people got their distro.
What happens when they just cut the underwater cables? Torrent over carrier pigeon for a linux distro would take ages
We need some more community wifi projects
Community Wisps are cool
Sneakernet to the rescue. Some of you are too young to know about walking around with boxes full of disks.
It was trading CD-R’s during my high school days… good times. Napster was just starting to take off by the time we had a CD-R trading network set up, Napster just increased the amount of CD’s that got passed around.
A wise man once said
Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
A good way to see what the future of places like the U.S are is to look at places like North Korea, where they do exactly this, move files around on flash media to avoid the state censors.
Pigeon latency is horrible, but the bandwidth is pretty great. You could probably load up an adult pigeon with at least 12TB of media.
https://en.wikipedia.org/wiki/IP_over_Avian_Carriers
Just gonna leave this here for whoever wants to read more on the methodology and potential risks.
Over a 30-mile (48 km) distance, a single pigeon may be able to carry tens of gigabytes of data in around an hour, which on an average bandwidth basis compared very favorably to early ADSL standards, even when accounting for lost drives.
Compared to what I use at home now, this sounds great
Tiny jump drives on pigeons is low key excellent imo
@Maroon I thought torrent technology to be a godsend for package managers.
Why none of them use it?
I mean, damn.
Torrents are often used for installers, but for packages it tends to be more trouble than what it’s worth. Is creating a torrent for a 4k library worth it?
I would add in some rom collections and book repositories as well. The whole library of Nintendo games is under a gig and would go a long way for entertaining people.
