For a while, our issue was with scaling frontend servers. We started off running on dedicated server(s), but we're now on a hybrid of dedicated and cloud.
The issue tonight was with bandwidth between our servers, capping out around 500mbps/up. We've only located the problem tonight, having encountered this specific issue only three times: the two Presidential debates, and the Vice Presidential debate.
Obviously now that we've identified this bottleneck we'll be upgrading that server so that it's no longer something to have to even think about.
We can't give a huge amount of details because it's not just pedes reading this - but I'll answer any technical questions or concerns that I can.
I just figured a bunch of shitheads were doing DOS attacks. Kudos on how popular this site has become. I didn't know about it until a few months ago so I would count as one of the people overloading due to popularity.
IIRC a DOS attack involves basically spamming the absolute shit out of the server.
Which would mean we sorta do DOS the site, its just high energy shitposting rather than a malicious attack
2003 Civic Ex Coupe. Its nice and paid for, and it's been a dutiful daily driver for the past 17 years. Its been from Alaska, to Hawaii, to Germany, to Washington State, and finally back to Alaska.
I never got one, but I probably would have if I could have afforded it while the commercial with the guy who blew up his cheeks at a school bus was out :)
I stopped understanding after "scaling frontend servers"...
Since you seem to be a gearhead, think of it like this. How do you get more performance out of an engine? Maybe a ramscoop, maybe a turbo or supercharger, maybe some nitro if it's legal. Polished heads, intake and exhaust manifolds, clearance adjustments, pistons, and valves and so-on. Then adjusting timing, fuel mix ratios, etc. If you're track timing for a 1/4mi, you'll get into aerodynamics, pressure forces on the body and tires, grip off start, etc. You always hit a point where "what you've got will never get you further."
For websites it's not too much different, for server-side stuff like this it's especially true. When you're taxed to capacity, other components start to fail as something "requests this(e.g. post a comment)" and has to wait for the server to respond, so it keeps waiting. In the meantime, it keeps waiting and waiting, and suddenly the next component in line is tasked to capacity dealing with all old requests.
You can also run into issues where "the outbound pipe" is too restrictive and can't send enough data to complete requests. This can lead to an issue where "requests are queued up" between two servers because one is always waiting on the other but both are still stuck on other tasks. When that happens, the server will start shitting itself and dropping oldest requests(ie not posting any comments), which can roll into it's own snafu.
So, "scaling frontend servers" basically means chunking for the next big upgrade in line and making sure the rest of the stuff can run fine. Hopefully that didn't boil it down enough that it loses all meaning.
I don't want to come off as a disgruntled customer because A) I am not a customer and B) this is the greatest website ever created by mankind so, thank you for all your hard work.
But, I will say this...
This site needs to be locked and loaded come election night. Please, do what you can.
It might be pretty useful to see registered voter #s for each state/county BEFORE election night so that the Democrats know they are on notice.
If a county has 3000 registered voters and 2990 votes are tallied on election night, Trump is winning by 400 but John Roberts gives them a week and a half to find 401/10 votes... [Not even talking about NY and Cali auto-registering illegals and all the (D)ead voters.]
But I'm not sure it addresses the recent ddos issues. I know they've tried to use cloudflare, and I've never implemented it myself to fully understand the issues, but other sites seem to handle it better.
The issue today appeared to have slightly different behavior.
We've used Cloudflare from the start - we haven't been successfully taken down by a DDoS in a long time.
At the very start, we only had dedicated servers and no cloud. For some events like when Bernie dropped out, the site needed more servers and we didn't have any.
So we started using cloud when we were hitting unusual traffic. This was fine, but since we didn't want to be reliant on a big cloud company, we kept on with that manually for too long.
That's no longer the case, and we're mostly in the clear - in fact tonight was the first extremely high traffic event with the current setup. It allowed us to diagnose the current bottleneck (with a backend server, which the frontend servers connect to) which we'll work to fix today.
There was a time not too long ago that the site would struggle under one of the rally threads that we now don't have to worry about at all.
B) this is the greatest website ever created by mankind so...
check out creation.com some day. A drive crash took out their old front end before the new front end was ready. This was back when reddit.com (as opposed to old.reddit.com) sucked. I mean sucked worse than it does now. creation.com's new front end past Reddit's new front end like they were standing still. ...well, they were actually standing still, but I was still impressed with how fast creation.com got its crap back together. (Also, they are much closer to "four score and seven years ago" (quoting OP): that would be 1973 and Creation Magazine started in 1978 - I think the website in 1998. Edit: that would be 1933, I forgot middle school age math, lol.)
If this policy ever changes, I will definitely donate. Best way to minimize "money buys influence" would be to have a large number of anonymous donors and a cap on donation amount. Nobody gets influence for 10 bucks a month lol
I gotta agree with QueMala. Not accepting donations or memberships keeps this site kosher, and allows us to stay above board with the health of this community’s organic life. I don’t mean this to accuse the mods, but money muddies things up quick, its best for us to retain our sovereignty as a site if we avoid bringing money into it
My biggest concern is we need this place up and running 99% of the time on election night. Will it be?
In 16 on r/TD, we were instrumental in motivating each other, our friends and family, TO STAY IN LINE AND VOTE. PANHANDLE KEEP VOTING ITS NOT DONE! Anyone who was around will know what im talking about.
We NEED this website at basically 100% on nov 3rd. Will it be?
Obviously, you guys are doing gods work. Thank you
I remember thinking the fix was in when crooked Broward County in Florida stopped reporting for a while and then flooded in with the Clinton votes, but they were not prepared for just how many votes Trump got and they didn't have the amount of fake provisional ballots they needed to steal it.
Election officials there were panicking, desperately trying to fill out as many provisional ballots they could. Too bad, a dozen degenerates can only do so much to scratch a 200,000 vote lead.
On reddit TD, we were able to post in real time the amount of fraud going on in Pennsylvania. THATS HUGE! In fact, we were posting all the shady shit going on around the country and we were able to curb the Dems in their cheating schemes.
I'm not confident we will be able to do this on Nov 3. I love this site, but every single time something big happens, this site dies.
I just hope the mods find a way to fix this before Nov 3.
I keep saying they need to introduce a gold system like reddit.
but instead of gold its bricks.
If I like your comment, I should be able to guild it with a brick. The money in turn would be used to upgrade the shit out of this site and make sure the servers are running smoothly.
☝☝☝This can't be overstated. It's very important from now through the first few days after election that this site be up and working in real time. Especially on election day.
For a while, our issue was with scaling frontend servers. We started off running on dedicated server(s), but we're now on a hybrid of dedicated and cloud.
The issue tonight was with bandwidth between our servers, capping out around 500mbps/up. We've only located the problem tonight, having encountered this specific issue only three times: the two Presidential debates, and the Vice Presidential debate.
Obviously now that we've identified this bottleneck we'll be upgrading that server so that it's no longer something to have to even think about.
We can't give a huge amount of details because it's not just pedes reading this - but I'll answer any technical questions or concerns that I can.
Doggos, we love you guys. I'm not complaining! Keep up the good work brother. Patience is a virtue!
I complain a lot, but I apologize. Thanks to everyone who makes TDW possible!
nah, there is a mole
You guys are awesome and appreciate the honesty and update. Keep it up!!
I just figured a bunch of shitheads were doing DOS attacks. Kudos on how popular this site has become. I didn't know about it until a few months ago so I would count as one of the people overloading due to popularity.
IIRC a DOS attack involves basically spamming the absolute shit out of the server. Which would mean we sorta do DOS the site, its just high energy shitposting rather than a malicious attack
...I thought it was older than DOS ;)
I'll never get tired of winning!
So much winning we DDOS'd ourselves
OH SHIT THE HACKER KNOWN AS FOR-CHAN
I stopped understanding after "scaling frontend servers"...
You guys are doing a damn fine job with the tools you have!
If I were running this place, it would be all stick figures and smiley faces...
Shitposter in Chief vs Old Fag
FATAL1TY
Baudmouth Joe
Citric acid or white vinegar can help with the scaling.
Wipe it on with a cloth or something?
NBC: 5 PINNOCHIOS! It was bleachbit, not acid wash!
https://twitter.com/NBCNews/status/785299709342654465
Our HF got 50 mpg regularly. The SI was great, too.
2003 Civic Ex Coupe. Its nice and paid for, and it's been a dutiful daily driver for the past 17 years. Its been from Alaska, to Hawaii, to Germany, to Washington State, and finally back to Alaska.
I never got one, but I probably would have if I could have afforded it while the commercial with the guy who blew up his cheeks at a school bus was out :)
Since you seem to be a gearhead, think of it like this. How do you get more performance out of an engine? Maybe a ramscoop, maybe a turbo or supercharger, maybe some nitro if it's legal. Polished heads, intake and exhaust manifolds, clearance adjustments, pistons, and valves and so-on. Then adjusting timing, fuel mix ratios, etc. If you're track timing for a 1/4mi, you'll get into aerodynamics, pressure forces on the body and tires, grip off start, etc. You always hit a point where "what you've got will never get you further."
For websites it's not too much different, for server-side stuff like this it's especially true. When you're taxed to capacity, other components start to fail as something "requests this(e.g. post a comment)" and has to wait for the server to respond, so it keeps waiting. In the meantime, it keeps waiting and waiting, and suddenly the next component in line is tasked to capacity dealing with all old requests.
You can also run into issues where "the outbound pipe" is too restrictive and can't send enough data to complete requests. This can lead to an issue where "requests are queued up" between two servers because one is always waiting on the other but both are still stuck on other tasks. When that happens, the server will start shitting itself and dropping oldest requests(ie not posting any comments), which can roll into it's own snafu.
So, "scaling frontend servers" basically means chunking for the next big upgrade in line and making sure the rest of the stuff can run fine. Hopefully that didn't boil it down enough that it loses all meaning.
Remove the muffler and go from 4"OD to 8"OD
Nice! Thanks for the explanation!
Yeah, im going full reddit when I see this so.......
Spez is a cuck
I don't want to come off as a disgruntled customer because A) I am not a customer and B) this is the greatest website ever created by mankind so, thank you for all your hard work.
But, I will say this...
This site needs to be locked and loaded come election night. Please, do what you can.
Nothing but love...
Election night will be yuge
It might be pretty useful to see registered voter #s for each state/county BEFORE election night so that the Democrats know they are on notice.
If a county has 3000 registered voters and 2990 votes are tallied on election night, Trump is winning by 400 but John Roberts gives them a week and a half to find 401/10 votes... [Not even talking about NY and Cali auto-registering illegals and all the (D)ead voters.]
But I'm not sure it addresses the recent ddos issues. I know they've tried to use cloudflare, and I've never implemented it myself to fully understand the issues, but other sites seem to handle it better.
The issue today appeared to have slightly different behavior.
We've used Cloudflare from the start - we haven't been successfully taken down by a DDoS in a long time.
At the very start, we only had dedicated servers and no cloud. For some events like when Bernie dropped out, the site needed more servers and we didn't have any.
So we started using cloud when we were hitting unusual traffic. This was fine, but since we didn't want to be reliant on a big cloud company, we kept on with that manually for too long.
That's no longer the case, and we're mostly in the clear - in fact tonight was the first extremely high traffic event with the current setup. It allowed us to diagnose the current bottleneck (with a backend server, which the frontend servers connect to) which we'll work to fix today.
There was a time not too long ago that the site would struggle under one of the rally threads that we now don't have to worry about at all.
check out creation.com some day. A drive crash took out their old front end before the new front end was ready. This was back when reddit.com (as opposed to old.reddit.com) sucked. I mean sucked worse than it does now. creation.com's new front end past Reddit's new front end like they were standing still. ...well, they were actually standing still, but I was still impressed with how fast creation.com got its crap back together. (Also, they are much closer to "four score and seven years ago" (quoting OP): that would be 1973 and Creation Magazine started in 1978 - I think the website in 1998. Edit: that would be 1933, I forgot middle
schoolage math, lol.)Honestly brah it's like just shut up and take my money buy new servers lol
>and B) this is the greatest website ever created by mankind so, thank you for all your hard work.
hnnnnggg <3 <3 <3
Which is exactly why we are begging you to take our money! We cannot be silenced!
If this policy ever changes, I will definitely donate. Best way to minimize "money buys influence" would be to have a large number of anonymous donors and a cap on donation amount. Nobody gets influence for 10 bucks a month lol
Alex Jones has a subscriber star, if it works for him (the most banned on the internet lol) then i don't see why it wouldn't work for us.
Granted as a public figure he does have to fight with millions of dollars in lawsuits every single year.
the fact that they dont want money, its proof we are the underground
they should take Monero
I gotta agree with QueMala. Not accepting donations or memberships keeps this site kosher, and allows us to stay above board with the health of this community’s organic life. I don’t mean this to accuse the mods, but money muddies things up quick, its best for us to retain our sovereignty as a site if we avoid bringing money into it
My biggest concern is we need this place up and running 99% of the time on election night. Will it be?
In 16 on r/TD, we were instrumental in motivating each other, our friends and family, TO STAY IN LINE AND VOTE. PANHANDLE KEEP VOTING ITS NOT DONE! Anyone who was around will know what im talking about.
We NEED this website at basically 100% on nov 3rd. Will it be?
Obviously, you guys are doing gods work. Thank you
I remember looking at the numbers in FL when Coward County results came in. I was yelling at the TV: "HOLD!!!, HOLD!!, HOLD!!"
I remember thinking the fix was in when crooked Broward County in Florida stopped reporting for a while and then flooded in with the Clinton votes, but they were not prepared for just how many votes Trump got and they didn't have the amount of fake provisional ballots they needed to steal it.
I can only pray they don't steal it this time.
Election officials there were panicking, desperately trying to fill out as many provisional ballots they could. Too bad, a dozen degenerates can only do so much to scratch a 200,000 vote lead.
This.
On reddit TD, we were able to post in real time the amount of fraud going on in Pennsylvania. THATS HUGE! In fact, we were posting all the shady shit going on around the country and we were able to curb the Dems in their cheating schemes.
I'm not confident we will be able to do this on Nov 3. I love this site, but every single time something big happens, this site dies.
I just hope the mods find a way to fix this before Nov 3.
That's what I'm saying just take some money at least, I'm willing to drop a decent amount
I keep saying they need to introduce a gold system like reddit.
but instead of gold its bricks.
If I like your comment, I should be able to guild it with a brick. The money in turn would be used to upgrade the shit out of this site and make sure the servers are running smoothly.
u/WallBot get this man a brick.
☝☝☝This can't be overstated. It's very important from now through the first few days after election that this site be up and working in real time. Especially on election day.