Before I begin: let me give you my chops. I've been a Cisco Certified CCNA (at least) fellow since at least 2004. I've seen the networking and IT business evolve from the birth of the internet up until this point. I've also used Solarwinds in previous jobs... so I'm somewhat familiar with it's inner workings.
When I heard the news that Solarwinds and Orion were both compromised since April of 2020 a huge stone has sunk in my stomach and won't leave. Thus: I'm here to talk about it.
Solarwinds is used primary as a monitoring and automation system. As a monitoring system it can check internal system logs of web servers and network devices (anything that can create or write to a log file which includes network switches, routers, wi-fi access points, you name it), it can save and store those network logs, and parse those logs for things you don't want to have happen (like devices crashing or network links going down).
it can further take that output and create automated warnings to allow people like me know how bad the problem is so that we can address it in a timely fashion.
Naturally, some people have taken this a step further with automation. Is there an access point showing signs of trouble? Have Solarwinds reboot it first. Getting regular alerts about a device in timely intervals? Have a script keep records of it for a meeting next month. Constantly getting false alarms? You can suppress those.
It's that automation bit that bugs me. Depending on how integrated Solarwinds might be in a given system Solarwinds can use accounts with admin level privileges to access files and operate whole systems. If Solarwinds was indeed compromised... then ANY system that had this level of automation attached (and any "good" engineer would have done this to save time) would have been potentially remote controlled by malicious actors.
Furthermore: since this was used by every branch of the US Government along with 400+ out of the 500 Fortune 500 companies... this is basically someone potentially having the keys to the whole damn kingdom.
Before I begin: let me give you my chops. I've been a Cisco Certified CCNA (at least) fellow since at least 2004. I've seen the networking and IT business evolve from the birth of the internet up until this point. I've also used Solarwinds in previous jobs... so I'm somewhat familiar with it's inner workings.
When I heard the news that Solarwinds and Orion were both compromised since April of 2020 a huge stone has sunk in my stomach and won't leave. Thus: I'm here to talk about it.
Solarwinds is used primary as a monitoring and automation system. As a monitoring system it can check internal system logs of web servers and network devices (anything that can create or write to a log file which includes network switches, routers, wi-fi access points, you name it), it can save and store those network logs, and parse those logs for things you don't want to have happen (like devices crashing or network links going down).
it can further take that output and create automated warnings to allow people like me know how bad the problem is so that we can address it in a timely fashion.
Naturally, some people have taken this a step further with automation. Is there an access point showing signs of trouble? Have Solarwinds reboot it first. Getting regular alerts about a device in timely intervals? Have a script keep records of it for a meeting next month. Constantly getting false alarms? You can suppress those.
It's that automation bit that bugs me. Depending on how integrated Solarwinds might be in a given system Solarwinds can use accounts with admin level privileges to access files and operate whole systems. If Solarwinds was indeed compromised... then ANY system that had this level of automation attached (and any "good" engineer would have done this to save time) would have been potentially remote controlled by malicious actors.
Furthermore: since this was used by every branch of the US Government along with 400+ out of the 500 Fortune 500 companies... this is basically someone potentially having the keys to the whole damn kingdom.
See... you're an old school IT admin who worries about permission levels, just join the cloud, baby; what could go wrong? Everyone is doing it...
The sarcasm is strong with you... but yes. I'm a grizzled old school It admin who thinks that our over-reliance on Automation due to increased IT workloads (and lower quality workers due to a number of factors) may have done us in here.
A system like Solarwinds is genius when it comes to the level of things you can do and monitor but it was ultimately a single point of failure if someone got a hold of admin credentials (or worse: control over the entire company).
An open source replacement is required... one where the level of control can be compartmentalized and limited in scope. The entire model will probably have to be re-thought after this...
...assuming the industry survives...
I feel like a dinosaur at the age of 30 for not trusting a gooddamn thing to "the cloud".
While a replacement is needed, you can't really monetize open source.
The closest open source thing I've seen is Ansible.
There is no cloud. It is just someone else's computer.
Cannot echo this hard enough.
Open source isn't meant for monetization on it's own.
Open source is meant for transparency.
There are times when you want open source. Things that REQUIRE transparency should REQUIRE open source.
And if this is going to be monitoring government systems then it needs to be transparent.
As for not trusting a goddamn thing to the cloud: I was suspicious about Facebook when it came out... and look how that turned out.
As for the cloud: cloud is just a fancy way of saying "you don't own the hardware... but just trust us! Ok?" I'm not touching that with a 30ft pole after this.
Spot on!
I've noticed that the main observable difference between developers and system administrators each one's respective trust in the cloud. Most sysadmins I know want on-prem services and cloud based services can burn in a fire.
We are currently in the midst of one of the biggest events in history in my opinion. We are too close to it to see whats really happening in a historical scale, but I think people will read about these years in the history books long down the line. This is all just one more domino in the line.
I feel the same way.
This is gonna be one of those events where everything that came after it will be unrecognizable to everything that came before it... at least in the IT world.
I just wish more people paid attention.
Bruh.....
This is why I prefer on site servers and I will always Press X to Doubt the cloud.
Help I’m a Solarwinds SME....MY JOB IDPS IS FUCKED
This is what we call in the IT industry a "Resume Generating Event."
Do you know if dominion used solarwinds? I saw a screen shot that showed potentially they did
Yes.
They did.
We know they used Serv-U (which in itself wasn't part of the exploit). I don't have any information if they implemented Orion or not.
If they were using Solarwinds they were 9/10 using Orion as the backend monitoring.
I suspect you're correct, but Dominion has made a habit of doing everything incorrectly. Ironically, not implementing Orion monitoring services would be one of those incorrect things. haha
It would be easier implementing a half baked implementation of Orion than it would be to implement your own back end to put into Solarwind's front-end.
And knowing how most IT people are lazy and mostly incompetent that's probably what they did.
My thoughts precisely.
I'm pretty sure the US Marshals are about to find out...
Dominion had a super user root account for everyone. Solarwinds should be the smallest of their problems.
CodeMonkey posted a screencap:
https://twitter.com/CodeMonkeyZ/status/1338431708496945157
I'd start learning your competitors practices quickly and work on becoming a Solarwinds migration SME.
Gotta be an opportunist in this industry.
Time to write 3 letters my friend.
Petition to start Sysadmin.win .. kek.
So basically, we had a security system in place... malicious actors could have come in, set up false alarms ahead of time, so :our guys: see them, ignore them, set them to automatically ignore. Then the bad guys can go in on a backdoor without setting an alarm off to steal/manipulate/etc.
Am I understanding correctly?
(thank you for your input! This is worrying me too). Is there any way to put a fix in/migrate it to another system (even though it'd suck) and so on?
It's largely a monitoring system that helps IT folks keep an eye on the health of all the important infrastructure/systems. It's very common for these kinds of tools to use accounts that have significant privileges across the enterprise/company.
It's like giving a hacker the master key to all the servers and networking infrastructure in a business... as well as a detailed map of all the key pieces of infrastructure in existence.
What a skilled/knowledgeable individual can do with that knowledge and elevated privileges is almost limitless.
This is a frighteningly correct assessment.
This wouldn't have even triggered alarms (although false alarms could have been made).
If.i understand it properly, an analogy could be made that you hire a maintenance man to tend your property. He has access to the entire property but is only given certain tasks. If someone stole his keys, they could use that same access to steal and vandalize.
This is it... in Layman's terms.
Thank you i believe all of tne defense contractors use it too
Every agency of the US did... at least the Civilian ones.
Just think of what has been stolen
I saw Lockheed Martin... Which can't be good
See the new Chinese J-20 and how similar it is to Lockheed's F22?
I know stealing military tech has always been an issue, but this would make it so simple.
Gratuitous follow-up: the J20 is made in China so it's shit anyway.
This may be a dumb question, but I’m not very tech savvy... So what you’re saying is that a hacker could get into Solarwinds and then use that to get into any other system that uses it? For example, getting into all of Lockheed Martin, going through whatever they wanted anywhere there, and repeating that in any company or government office they wanted?
This depends on their level of automation... which basically means yes since every IT department depends on automation to deal with the ever increasing workloads.
The service accounts that would have been given to Solarwinds as a means of having access/privileges to do what was scripted would have basically given the hackers free reign.
Thank you for replying, I had no idea how massive this is.
Also could be a "legitimate" user of the system. The "hack" can be used as plausible deniability when the shit hits the fan. The bad actors then have a potential "out". "It wasn't us, the systems were hacked using this exploit!"
Most "hacks" aren't...
Not OP, but yes.
Is OP. The answer is definitely yes.
Thanks for the response. That’s terrifying.
Call me retarded, but what does OP stand for? Operations?
Original Poster.
Man, I thought so, but then I started to think it was a fancy tech acronym. Thx.
That's the lazy way of doing it...
...which basically means that's how everyone did it...
It's not a hack if it's a built in feature.
its way worse than you think. solarwinds left admin creds for their ftp of their official download server in a github repo that ended up being discovered in nov 2019 and that was accessed by god knows who aside from the guy who reported it. while that is not enough access for them to implant the backdoor within the digitally signed .dll its located in, its more than enough of a starting point to obtain said access.
https://savebreach.com/solarwinds-credentials-exposure-led-to-us-government-fireye-breach/
the solarwinds backdoor was used to gain access to fireeye's internal network where a vmware 0day was deployed for further lateral movement and compromise, they got data including fireeye's own weaponized exploits along with the weaponized exploits collected by fireeye during their many investigations of breaches across multiple organzations and entities, and this is just the beginning of whats coming to light now. expect to see a vast amount more information to come out over the next few days as more orgs verify the IOC's
Pardon my french but...
FUCK
This is indeed worse than I thought it was... and I was pretty sure it was bad.
its definitely far worse than people realize. fireeye basically cooperate with our gov and most other govs to investigate nation-state sponsored attacks and such.
some takeaways after reading the main analysis of the backdoor
the attacker most definitely had knowlege of the inner workings tailored access operations and possibly had training, as well as having performed enough operational security and information gathering to isolate and terminate their implant if they detected any traffic from known netblocks leased or utilized by intel such as the microsoft netblock 96.31.172.0/24, and the nokia netblock our NSA has used for numerous ops, 131.228.12.0/22
if its this bad how do we know the same people didnt hack the election? to me this may be a blessing in disguise giving a reason for a new one
My God this is terrible. So much information on so many levels. Secrets, developments, tech.. all compromised.
Very sick and saddened !!!
Great post first I have seen with hands on after watching this play out 24 hours. My gut felling is ci$co / m$ and others * are going to be on buffet line also if its followed to the end...
Cisco and MS both used Solarwinds.
Yes they're on the buffet line.
This is going to have world altering ramifications.
i was caught up in the Fed OPM hack and for a long while anytime I reoriented my phone, it switched to Sinitic language for a few seconds. That's just the beginning.
Not to rub it in or anything, but y'all should have been using Splunk in the first place. Just sayin'
Whoooo - I should probably buy some Splunk stock, now that I think of it...
Given the sophistication of infiltration of the chain of development, it was more than likely an insider threat. Same shit could happen to Splunk.
That being said, two different type of products but Splunk is superior.
Splunk is superior. Why? Because you gotta set it up yourself and get it working for yourself.
Once that's done however? It's yours. Yours to command, control, and keep safe.
Key thing I said is "two different type of products". Splunk is better at what they do. I've had a heavy hand in both solutions.
My other point is that they too can be susceptible to an insider threat injecting malicious code into their updates.
We're also only getting the initial reports. This whole thing smells of China trying to warn Trump not to start anything.
Thanks for taking so much time to do that write up for us.
Just your neighborhood friendly network engineer doing his best to keep the lights on...
Is it honest, but not much?
I'm fortunate enough to have a gig that pays well enough to pay the bills and have enough left over to save.
I'm hoping that once President Trump secures his second term that he does something about all of these H1Bs. I don't want to have to learn another hobby just to pay bills.
Plus you could potentially keep logs and other signs of malicious activity from being discovered and escalated to other SIEMs and alerting systems. Basically you have the keys to the kingdom since as you’ve said SolarWinds is normally responsible for every system and device on the local network and potentially write access and ability to control those devices.
I wasn't exaggerating when I used that term. Potential full system access and control was granted when this occurred.
Again: it depends on how integrated Solarwinds was into the command/control infrastructure... but again... good IT admins who were big into automation probably had a massive amount of integration.
If SolarWinds has full admin access to network equipment, with just a few commands, they can pull packet captures on pretty much any network device. They could also likely gather rough location/presence info for wireless devices, and for example, piece together a CEO's regular schedule and which locations they normally go. And this doesn't even take into account if they can install more sophisticated code on a local device.
You're thinking like a hacker. I like you.
One word can surmise this whole situation.
Fuck.
We are so boned.
could this be a group of white hats?
Solarwinds says it was a foreign group... so my magic 8 ball says NO.
I don't know what you mean by magic 8 ball. I appreciate the quick response. Hasn't the "oh gee, we were hacked" been used as cover in the past as a springboard or catalyst for launching other things?
...like, "oh. youve been breached since April and you didnt initiate any mitigated protocols, so now we're going to take your passport and raid you Austin HQ"?
Lemme connect the dots for you.
"Hasn't the "oh gee, we were hacked" been used as cover in the past as a springboard or catalyst for launching other things?"
Does Executive Order 13848 ring a bell?
Now you know what I said about EVERY Federal agency using this?
Solarwinds is admitting on it's front page that it was compromised by FOREIGN hackers.
The FEC's own monitoring system was COMPROMISED by FOREIGN HACKERS.
This is basically the justification required to trigger Executive Order 13848.
now we arrive at the silver lining
MSM immediately jumped to 'multiple sources say Russia'. So probably CCP China. lol.
Sorry if dumb question...Not techie...Could this vulnerability be used as ransomware - or, inject ransomware? If you have the keys to the kingdom, could you lock others out?
Would you rather kidnap a country's systems for a little while or for full 4 years? Most of the time ransomware is a wasted opportunity -- it only pays out if the data is sensible to the victim but not to the invader.
Yes to all of those.
Yes to any of the above.
What you are describing is something we have to do for RCCL Cisco/Meru systems so we can fix issues remotely via satellite if the ship has any software issues at sea.. (9/10 its a blade followed by a trip to the airport.. but thats off topic.)
How do we pedes find out if our networks have been impacted? Sometime last month, after an MS update, my computers stopped communicating with one of my printers, an HP Laser.
I'm not looking for tech advise here, I would genuinely like to know how to find out if that Oct MS update was related to this Solarwinds issue, thus a malware infection.
And, do we know what the objective/target of today's hack is?
I wish I could answer your questions when it comes to this. I'm not that far advanced into my studies of network and systems security.
As for what the objective of today's hack was? Remember that this started back in APRIL. They had control from APRIL until NOW without anyone knowing.
I just got a shiver reading your last sentence.
Thanks for you insights. Terrifying, but much appreciated.
Why didn't they just have ad-aware and Norton installed bro!!!!!!!!!!!!11
in short: the US's secrets are wide-known, right?
Basically.
Yeah, well Im pretty sure the us military uses windows computers. They did when I was in at least.
Everyone does.
So your thoughts aren’t good?
We're fucked.
orion agent runs as admin. everyone using it is fucked
Just reading this techno-shit makes me think some people did some things and we don't even know what the fuck they did? Dammit!
My company uses it for handling IT trouble tickets and god knows what else. Can't imagine anyone trying to do things to my company other than stamping down competition but what do I know. We only do stuff like design bridges and bs like that. Nothing important /s
Also, we just started using it if I recall correctly. I think we implemented it back in April or May.
You tell your corporate/IT heads to pay attention and begin re-evaluating their implementation... preferably also looking for alternatives.
They are already aware. They were acting pretty weird today. I didn't discuss anything but I couldn't get anyone to help me with the brand spankin' new work laptop I got that refuses to work. They seemed...busy.
SPOF doomer here, also an engineer (systems).
Never EVER EVER EVER EVER give more perms than absolutely necessary. Maybe I'm paranoid, maybe old-school. Tyrants taking advantage of this architecture is preventable, but to be frank, I fully expected some malicious actor to do this at some point.
We as an industry are too reliant on automation processes to do things we really need to have eyes on. How long does to take to run a daily or weekly or quarterly (script) anymore? A few minutes?
TL/DR: Point is, we have traded security for convenience (we think) and in our arrogance, gotten neither. Same argument we have with security vs liberty right?
Step one of sensible security engineering: limit perms.
Step two: compartmentalize.
Step three: wherever possible eliminate SPOFs.
Boom,
You are now worth at least 100k.
:P
Crazy I never heard of this company.
They're one of those large companies you rarely hear about but really should know more about.
On a scale of 1-10 (1 being "don't be silly", 10 being "should've been done yesterday", how urgently should I withdraw my savings from my bank accounts?
If you have to ask...
Interesting read. I don't keep shit in the cloud.
Keep it that way.
Almost like allowing the communists access to all our inner workings was a bad idea.... lessons learned?
Are we even sure they were hacked? I'm getting the feeling it was an inside event that occurred and the company is saving itself legally by claiming an outside hack.
Who knows at this point...
Can anyone explain to the average Joe what this all means ?
Simple:
You hired a security guard to protect the most valuable things in your company.
He just got mugged and his keys got stolen.
You now find out that for the last 9 months someone has been using a copy of said keys to access everything in your operation without him or you knowing.
That's what this amounts to.
Now imagine that across every Civilian agency in the US government along with 400+ out of the Fortune 500 companies out there in the world.
Got a good picture of how big the problem is?
Thank you, wow,a brilliant explanation. Frightening
You're welcome.
If I’m reading this correctly; solarwinds worked directly with Microsoft for azure’s authentication services. Said service is Orion and said service was what was compromised.
If they used Orion they're compromised.
I've always known all the "Cloud" hype was a bunch of crap. NOTHING on or connected to the Internet is secure, giving all your data to strangers to "safeguard" is absolute insanity.
So, like....does anyone remember when secret documents were just transported in a briefcase chained the the deliverer's arm? Documents printed on paper!? Turns out that was better.
Transit security was also a lot more reliable when said deliverer could carry a briefcase in one hand... and a gun in the other...
Orion? Explain further please.
Orion is Solarwind's back-end.
Orion is the engine that receives logs, parses them, and runs the commands in the background when events are triggered.
They’re together, not separate? Yes
Technically they're not together. You can use Solarwinds Without Orion... you just have to have your own devices feeding into a Solarwinds cluster you're running on your own hardware.
Orion is Solarwinds service that they sell as a back end to work with Solarwinds... in case you didn't want to set it up yourself.
Hey thanks, just clarification. My wife works for Orion. It’s a 401k sort of mini ameritrade type company and she’s been hounding me about this.
I’d be very interested to know if these vulnerabilities existed before or after Solar Winds acquisition.
Inquiring minds want to know.
This shit is way to similar sounding to Cyberdine to make me fee comfortable.
Perhaps we should be learning how to build small EMPs for the coming robot wars
This was a system used by THE ENTIRE US GOVERNMENT and 400+ out of the Fortune 500 companies.
NO ONE WHO WORKS FOR ANY OF THESE COMPANIES WILL WANT YOU TO KNOW.