Background: 15 years of experience in software and apparently spoiled because it was already set up correctly.
Been practicing doing my own servers, published a test site and 24 hours later, root was compromised.
Rolled back to the backup before I made it public and now I have a security checklist.
Had this years ago except it was a dumbass contractor where I worked who left a Windows server with FTP services exposed to the Internet and IIRC anonymous FTP enabled, on a Friday.
When I came in on Monday it had become a repository for warez, malware, and questionable porn. We wiped out rather than trying to recover anything.
questionable?
Yeah just like that. Ask more questions
Use gnome powder to shrink, go behind the counter, kick his ass and get your money back.
Technically it’s still a public server. Just even more so.
I’ve always felt that if you’re exposing an SSH or any kind of management port to the internet, you can avoid a lot of issues with a VPN. I’ve always setup a VPN. It prevents having to open up very much at all and then you can open configured web portal ports and the occasional front end protocol where needed.
Exactly.
All of my services are ‘local’ to the VPN. Nothing happens on the LAN except for DHCP and WireGuard traffic.
Remote access is as simple as pressing the WireGuard button.
Interesting. Do you know how it got compromised?
I published it to the internet and the next day, I couldn’t ssh into the server anymore with my user account and something was off.
Tried root + password, also failed.
Immediately facepalmed because the password was the generic 8 characters and there was no fail2ban to stop guessing.
because the password was the generic 8 characters and there was no fail2ban to stop guessing
Oof yea that’ll do it, your usually fine as long as you hardened enough to at least ward off the script kiddies. The people with actual real skill tend to go after…juicer targets lmao
Haha I’m pretty sure my little server was just part of the “let’s test our dumb script to see if it works. Oh wow it did what a moron!”
Lessons learned.
Don’t use passwords for ssh. Use keys and disable password authentication.
More importantly, don’t open up SSH to public access. Use a VPN connection to the server. This is really easy to do with Netbird, Tailscale, etc. You should only ever be able to connect to SSH privately, never over the public net.
what’s netbird
https://netbird.io/. Wireguard based software defined networking, very similar to Tailscale.
It’s perfectly safe to run SSH on port 22 towards the open Internet with public key authentication only.
https://nvd.nist.gov/vuln/detail/cve-2024-6409 RCE as root without authentication via Open SSH. If they’ve got a connection, that’s more than nothing and sometimes it’s enough.
That attack vector is exactly the same towards a VPN.
A VPN like Wireguard can run over UDP on a random port which is nearly impossible to discover for an attacker. Unlike sshd, it won’t even show up in a portscan.
This was a specific design goal of Wireguard by the way (see “5.1 Silence is a virtue” here https://www.wireguard.com/papers/wireguard.pdf)
It also acts as a catch-all for all your services, so instead of worrying about the security of all the different sshds or other services you may have exposed, you just have to keep your vpn up to date.
Are you talking a VPN running on the same box as the service? UDP VPN would help as another mentioned, but doesn’t really add isolation.
If your vpn box is standalone, then getting root is bad but just step one. They have to own the VPN to be able to even do more recon then try SSH.
Defense in depth. They didn’t immediately get server root and application access in one step. Now they have to connect to a patched, cert only, etc SSH server. Just looking for it could trip into some honeypot. They had to find the VPN host as well which wasn’t the same as the box they were targeting. That would shut down 99% of the automated/script kiddie shit finding the main service then scanning that IP.
You can’t argue that one step to own the system is more secure than two separate pieces of updated software on separate boxes.
Tailscale? Netbird? I have been using hamachi like a fucking neanderthal. I love this posts, I learn so much
Which distro allows root to login via SSH?
Not very many. None of the enterprise ones, at least.
All of them if you configure it?
Ah, timeless classic.
Any idea what ip addresses were used to compromise it?
I ran a standard raspian ssh server on my home network for several years, default user was removed and my own user was in it’s place, root was configured as standard on a raspbian, my account had a complex but fairly short password, no specific keys set.
I saw constant attacks but to my knowledge, it was never breached.
I removed it when I realized that my ISP might take a dim view of running a server on their home client net that they didn’t know about, especially since it showed up on Shodan…
Don’t do what I did, secure your systems properly!
But it was kinda cool to be able to SSH from Thailand back home to Sweden and browse my NAS, it was super slow, but damn cool…
Why would a Swedish ISP care? I’ve run servers from home since I first connected up in … 1996. I’ve had a lot of different ISPs during that time, although nowadays I always choose Bahnhof because of them fighting the good fights.
They probably don’t, unless I got compromised and bad traffic came from their network, but I was paranoid, and wanted to avoid the possibility.
Lol ssh has no reason to be port exposed in 99% of home server setups.
VPNs are extremely easy, free, and wireguard is very performant with openvpn also fine for ssh. I have yet to see any usecase for simply port forwarding ssh in a home setup. Even a public git server can be tunneled through https.
Yeah I’m honest with myself that I’m a security newb and don’t know how to even know what I’m vulnerable to yet. So I didn’t bother opening anything at all on my router. That sounded way too scary.
Tailscale really is magic. I just use Cloudflare to forward a domain I own, and I can get to my services, my NextCloud, everything, from anywhere, and I’m reasonably confident I’m not exposing any doors to the innumerable botnet swarms.
It might be a tiny bit inconvenient if I wanted to serve anything to anyone not in my Tailnet or already on my home LAN (like sending al someone a link to a NextCloud folder for instance.), but at this point, that’s quite the edge case.
I learned to set up NGINX proxy manager for a reverse proxy though, and that’s pretty great! I still harden stuff where I can as I learn, even though I’m confident nobody’s even seeing it.
Honestly, crowdsec with the nginx bouncer is all you need security-wise to start experimenting. It isn’t perfect security, but it is way more comprehensive than fail2ban for just getting started and figuring more out later.
Here is my traefik-based crowdsec docker composer:
services: crowdsec: image: crowdsecurity/crowdsec:latest container_name: crowdsec environment: GID: $PGID volumes: - $USERDIR/dockerconfig/crowdsec/acquis.yaml:/etc/crowdsec/acquis.yaml - $USERDIR/data/Volumes/crowdsec:/var/lib/crowdsec/data/ - $USERDIR/dockerconfig/crowdsec:/etc/crowdsec/ - $DOCKERDIR/traefik2/traefik.log:/var/log/traefik/traefik.log:ro networks: - web restart: unless-stopped bouncer-traefik: image: docker.io/fbonalair/traefik-crowdsec-bouncer:latest container_name: bouncer-traefik environment: CROWDSEC_BOUNCER_API_KEY: $CROWDSEC_API CROWDSEC_AGENT_HOST: crowdsec:8080 networks: - web # same network as traefik + crowdsec depends_on: - crowdsec restart: unless-stopped networks: web: external: truehttps://github.com/imthenachoman/How-To-Secure-A-Linux-Server this is a more in-depth crash course for system-level security but hasn’t been updated in a while.
wow crazy that this was the default setup. It should really force you to either disable root or set a proper password (or warn you)
Now that you mentioned it, it didn’t! I recall even docker Linux setups would yell at me.
Love Hetzner. You just give them your public key and they boot you into a rescue system from which you can install what you want how you want.
I think their auction servers are a hidden gem. I mean the prices used to be better. Now they have some kind of systrem that resets them when they get too low. But the prices are still pretty good I think. But a year or two ago I got a pretty good deal on two decently spec’d servers.
People are scared off by the fact you just get their rescue prompt on auctions boxes… Except their rescue prompt has a guided imaging setup tool to install pretty much every popular distro with configurable raid options etc.
Yeah, I basically jump from auction system to auction system every other year or so and either get a cheaper or more powerful server or both.
I monitor for good deals. Because there’s no contract it’s easy to add one, move stuff over at your leisure and kill the old one off. It’s the better way to do it for semi serious stuff.
Most distributions disable root by default
Which ones? I’m asking because that isn’t true for cent, rocky, arch.
Rocky asks during setup, I assume centOS too
we’re probably talking about different things. virtually no distribution comes with root access with a password. you have to explicitly give the root user a password. without a password no amount of brute force sshing root will work. I’m not saying the root user is entirely disabled. so either the service OP is building on is basically a goldmine for compromised machines or OP literally shot themselves in the root by giving root a password manually. something you should never do.
Many cloud providers (the cheap ones in particular) will put patches on top of the base distro, so sometimes root always gets a password. Even for Ubuntu.
There are ways around this, like proper cloud-init support, but not exactly beginner friendly.
#no thank you lol
Yeah I was confused about the comment chain. I was thinking terminal login vs ssh. You’re right in my experience…root ssh requires user intervention for RHEL and friends and arch and debian.
Mostly Ubuntu. And… I think it’s just Ubuntu.
Fedora (immutable at least) has it disabled by default I think, but it’s just one checkbox away in one of the setup menus.
Standard Fedora does as well
Ah fair enough, I know that’s the basis of a ton of distros. I lean towards RHEL so I’m not super fluent there.
This is like browsing /c/selfhosted as everyone portforwards every experimental piece of garbage across their router…
portforwards every experimental piece of garbage across their router…
Man some of those “It’s so E-Z bro” YouTubers are WAY too cavalier about doing this.
hey, thats me!
Meh. Each service in its isolated VM and subnet. Plus just generally a good firewall setup. Currently hosting ~10 services plubicly, never had any issue.
Well, if you actually do that, bully for you, that’s how that should be done if you have to expose services.
Everyone else there is probably DMZing their desktop from what I can tell.
Yeah the only thing forwarded past my router is my VPN. Assuming I did my job decently, without a valid private key it should be pretty difficult to compromise.
I don’t think I’m ever opening up anything to the internet. It’s scary out there.
I don’t trust my competence, and if I did, I dont trust my attention to detail. That’s why I outsource my security: pihole+firebog for links, ISP for my firewall, and Tailscale for tunnels. I’m not claiming any of them are the best, but they’re all better than me.
Isp for firewalls might not be better than you. Get something dedicated.
Ubiquiti or pfsense is a good start.
Lol you can actually demo a github compromise in real time to an audience.
Make a repo with an API key, publish it, and literally just watch as it takes only a few minutes before a script logs in.
I search commits for “removed env file” to hopefully catch people who don’t know how git works.
You gremlin lmao
–verbose please?
edit: never mind, found it. So there’s dumbasses storing sensitive data (keys!) inside their git folder and unable to configure .gitignore…
yeah, I just tried it there, people actually did it.My work is transferring to github from svn currently
My condolences
And this is why every time a developer asks me for shell access to any of the deployment servers, I flat out deny the request.
Good on you for learning from your mistakes, but a perfect example for why I only let sysadmins into the systems.
You’re not wrong! Devops made me lazy
Please examine where devops allowed non-system people to be the last word on altering systems. This is a risk that needs block-letter indemnification or correction.
It’s not that devops made ya lazy. I’ve been doing devops since before they coined the term, and it’s a constant effort to remind people that it doesn’t magically make things safe, but keeping it safe is still the way.
Ah not to discount devops, I mean that in a good way.
Devops made me lazy in that for the past decade, I focus on just everything inside the code base.
I literally push code into a magic black box that then triggers a rube goldberg of events. Servers get instanced. Configs just get magically set up. It’s beautiful. Just years of smart people who make it so easy that I never have to think about it.
Since I can’t pay my devops team to come to my house, I get to figure it all out!
We have it at my company its just a very small group and we have to manually enable it for production and its through tools like teleport. Staging and the like its free game there for them for debugging, same infra through. Gives us best of all worlds
One time, I didn’t realize I had allowed all users to log in via ssh, and I had a user “steam” whose password was just “steam”.
“Hey, why is this Valheim server running like shit?”
“Wtf is
xrx?”“Oh, it looks like it’s mining crypto. Cool. Welp, gotta nuke this whole box now.”
So anyway, now I use NixOS.
Good point about a default deny approach to users and ssh, so random services don’t add insecure logins.
Do not allow username/password login for ssh. Force certificate authentication only!
If it’s public facing, how about dont turn on ssh to the public, open it to select ips or ranges. Use a non standard port, use a cert or even a radius with TOTP like privacyIdea. How about a port knocker to open the non standard port as well. Autoban to lock out source ips.
That’s just off the top of my head.
There’s a lot you can do to harden a host.
dont turn on ssh to the public, open it to select ips or ranges
What if you don’t have a static IP, do you ask your ISP in what range their public addresses fall?
Sure. My ISP gave me this range for this exact reason.
Do not allow username/password login for ssh
This is disabled by default for the root user.
$ man sshd_config ... PermitRootLogin Specifies whether root can log in using ssh(1). The argument must be yes, prohibit-password, forced-commands-only, or no. The default is prohibit-password. ...
Although disabling the root user is a good part of security, leaving it enabled should not alone cause you to get compromised. If it did, you were either running a very old version of OpenSSH with a known flaw, or, your chosen root password was very simple.
The latter. It was autogenerated by the VPS hosting service and I didn’t think about it.
It should be a serious red flag that your VPS host is generating root passwords simple enough to get quickly hacked.
I’m pretty sure they assumed if you bought their service, you have the competency to properly set it up.
And I proved them wrong.
You should turn off ssh password logins on external facing servers at a minimum. Only use ssh keys, install fail2ban, disable ssh root logins, and make sure you have a firewall limiting ports to ssh and https.
This will catch most scripted login attempts.
If you want something more advanced, look into https://en.m.wikipedia.org/wiki/Security_Technical_Implementation_Guide and try to find an ansible playbook to apply them.
Just turn off password logins from anything but console. For all users. No matter where it runs.
It becomes second to nature pretty fast, but you should have a system for storing / rotating keys.
How do I whitelist password logins? I only disabled password logins in SSHd and set it to only use a key.
I also like to disable root login by setting its default shell to nologin, that way, it’s only accessible via sudo or doas. I think there’s a better way of doing it, which is how Debian does it by default when not setting a root password, but I’m not sure how to configure that manually, or even what they do.
To whitelist password logins in ssh you can match username and give them yes after you set no (for all). But i see no reason for password logons in ssh, console is safe enough (for me).
Right - so console/tty login is restricted by pam and its settings. So disabling ssh root logins means you can still log in as root there.
To lock root you can use passwd -l
If locking root I would keep root shell so i could sudo to root.
So my normal setup would be to create my admin user with sudo rights, set «PasswordAuthentication no» in sshd config and lock root with «sudo passwd -l root» Remember to add a pubkey to admin users authorizedkeys, and give it a secure but typable password
My root is now only available through sudo, and i can use password on console. Instead of locking root you can give it secure typable password. This way root can log in from console so you dont need sudo for root access from console.
It boils down to what you like and what risks you take compared to usable system. You can always recover a locked root account if you have access to single-user-mode or a live cd . Disk encryption makes livecd a difficult option.
Did all that, minus the no ssh root login (only key, obviously) plus one failed attempt, fail2ban permaban.
Have not had any issues, ever
On a new linux install or image I will always:
- Make new users(s)
- Setup new user to sudo
- Change ssh port
- Change new user to authenticate ssh via key+password
- Disable root ssh login
- Setup new user to sudo
I hope it is not a passwordless sudo, it is basically the same as root.
That’s more or less the advice I’ve gotten as well. I’ve also read good things about fail2ban which tries to ban sources of repeated authentication failures to prevent brute force password attempts. I’ve used it, but the only person who has managed to get banned is myself! I did get back in after the delay, but I’m happy to know it works.
I’m having the opposite problem right now. Tightend a VM down so hard that now I can’t get into it.
