500 Internal Server Error

[Cthulhu]

I should go find a good log analyzer that can do things like count the number of requests / second, log cpu usage, that kinda thing. Actually I found a few neat commands that can run a simple analysis on the log to display requests per day, hour, so here we go:

# per day, 30 nov is duplicate because the command spans multiple log files
root@localhost:~# awk '{print $4}' /var/log/apache2/access* | cut -d: -f1 | uniq -c
 117292 [24/Nov/2014
  97741 [25/Nov/2014
  97071 [26/Nov/2014
 131478 [27/Nov/2014
  97302 [28/Nov/2014
  86147 [29/Nov/2014
  25442 [30/Nov/2014
  72419 [30/Nov/2014 # total about 100K
 371585 [01/Dec/2014
 136038 [02/Dec/2014

# per hour, omitted the uninteresting figures
root@localhost:~# grep "01/Dec" /var/log/apache2/access.log | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":00"}' | sort -n | uniq -c
<snip>
   3583 14:00
   3156 15:00
   8037 16:00
  54661 17:00
   7980 18:00
   7233 19:00
   8219 20:00
  36071 21:00
  97089 22:00
  97202 23:00

root@localhost:~# grep "02/Dec" /var/log/apache2/access.log | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":00"}' | sort -n | uniq -c
  69212 00:00
   5402 01:00
   3885 02:00
<snip>

So basically, about 20 - 24 hours ago, we had a pretty significant peak of 70 - 100K requests an hour, daily total of more than three times as much as normal. Counting requests done by IP address, we get a top 10 of most active IP addresses; I cross-referenced them with users for the lulz to see who whores TLS the most, or whether it's a spambot or whole country that needs to be banned .

cat /var/log/apache2/access.log | awk '{print $1}' | sort | uniq -c | sort -n | tail

   2377 58.179.x.x        # Ghost X 
   2482 211.26.x.x        # Also Ghost X
   2501 68.180.x.x        # no known member 
   2834 98.206.x.x        # Skan
   2902 98.93.x.x         # Benoist
   3185 172.250.x.x       # Howl
   3783 105.237.x.x       # Airling & Darth
   7262 84.208.x.x        # Fangu wins
  11116 217.44.x.x        # Octo has a problem
 340641 75.177.x.x        # ...Ryushikaze

So Ryu or his browser apparently did over 340K requests. Going over the logs, it looks like he's got the same problem as Red had - his browser, or an addon, causes the top 10 stats thing to go batshit and spam the shit out of it.

I'm tempted to just disable the plugin completely, IDK what's wrong with it besides it being five+ years old. I've disabled the plugin for Ryu for now - let me know if I should turn it off completely for everyone because somehow I'm sure it'll happen again.

 

[Airling]

Don't you dare touch my top ten plugin! I will hissy-fit girl you to your doom!

Nah, ditch it if you must. If it only breaks shit anyway, I can live without it.

 

[Cthulhu]

Hey, I'm okay with you keeping it, but I will turn it off for you if it turns out your browser breaks it,

Out of curiosity, Ryu, what kind of browser and browser addons do you have installed, and which are the same as Red?

 

[Lex]

^That couldn't be something to do with him recording the IRC chat log could it?

 

[Flintlock]

^That couldn't be something to do with him recording the IRC chat log could it?

No, it couldn't.

I would miss the top 10 stats plugin if it disappeared. Not because I actually care about the stats - they can go, for all I care - but because it's a handy way of seeing which threads have been updated most recently. I know that New Posts does the same thing, but it's not as compact, and threads disappear from there after I view them.

Suggestion: find a way to turn off the stats part of the top 10 stats plugin

 

[Fangu]

^ That, or get a different plugin.

100% agree with Flint's post.

 

[Cthulhu]

I've gone with a mild alternative for now - I installed mod_security, which with some foreign rules allows you to set a rate limit for certain resources. I've set it rather lenient to allow up to 30 requests a minute for the misc.php service (also because I don't know what misc.php is used for exactly beyond those stats); exceed that, and the user gets a 500 error (although the user won't actually see it because it's an AJAX request).

It's not ideal though; requests still come pouring in, each one generating an entry in the access log. It does reduce server load though; before, we'd be using 80% CPU constantly (per core, so total of 160%), atm (with Ryu opening his browser and walking away apparently) it's hovering between 5 and 30% per core. Have some random blurb:

SecRuleEngine On

<LocationMatch "^/forums/misc.php">
  SecAction initcol:ip=%{REMOTE_ADDR},pass,nolog,id:1234
  SecAction "phase:5,deprecatevar:ip.somepathcounter=1/1,pass,nolog,id:5678"
  SecRule IP:SOMEPATHCOUNTER "@gt 30" "phase:2,pause:300,deny,status:509,setenv:RATELIMITED,skip:1,nolog,id:9101"
  SecAction "phase:2,pass,setvar:ip.somepathcounter=+1,nolog,id:1337"
  Header always set Retry-After "10" env=RATELIMITED
</LocationMatch>

ErrorDocument 500 "Rate Limit Exceeded"

I should probably take a closer look at mod security, it should mitigate a lot of hostile attempts to fuck with our site, and I'm sure we get a lot of 'em.

Anyway, that should keep the site reasonably responsive for a while, at least until Ryu gets back to his keyboard and gets my messages and shit. Meh.

edit: hue, just tested it myself, in the chrome console:

var rq = new XMLHttpRequest();
rq.open('POST', 'http://thelifestream.net/forums/misc.php?show=thanked&resultsnr=10', true);
rq.send();

spam that a few dozen times and one tends to get 500 errors, as planned. It doesn't seem to be very consistent (as in sometimes it works after hitting the rate limit), but, eh. Good enough.

 

[Ⓐaron]

The site just went down for a couple of hours, btw (once again, not a 500 error this time). Not sure what caused it or if you're aware of it, but there you go.

 

[The G'randiest' Daddy]

^Ninja'd, I was just coming to report this.

Don't know how long it has been down, I've been trying since around 11 my time, it's now 1, so at least 2 hours. I didn't get any error message, it just told me the page couldn't load at all.

 

[Ⓐaron]

That's what I got as well. And one time my phone didn't think it was connected to the network (though that was probably the phone's fault; it does that sometimes).

 

[Tetsujin]

I had told Yop about it on Skype, he fixed it.

 

[Octo]

It just said Rate Limit exceeded. And then it wouldn't load at all. For a bit. More than 15 mins but less than an hour. I think. I went and did something else.

 

[Shademp]

^same here.

 

[Cthulhu]

Yeah, weird shit y0. I should probably get to work on replacing / modernizing our softwarez; it seems some connections are not released (or something), and Apache runs out of workers. Or something.

 

[Ⓐaron]

Oh yeah, I got the Rate Limit Exceeded thing at first too.

 

[The G'randiest' Daddy]

I got it with this last bout, but not earlier...

 

[Ⓐaron]

Got another "rate limit exceeded" for about twenty minutes. I assume someone else contacted Yop on Skype for this one (I couldn't remember what his Skype name is and it wasn't cached on his forum profile so I couldn't do it myself).

 

[Cthulhu]

Something seems to be holding connections open or something, or maybe occupy the php engine for a long time (effectively a DDOS), so that regular users can't make use of said services. I'm keeping an eye out on performance and shit now.

Also IP banned a bunch of Russian and Ukranian search engines (yandex, ahrefs), because they've been annoying me for far too long now,

 

[Ⓐaron]

I IP banned entire ranges of countries (mostly China and Vietnam) at Fool's Gold and we never got spambots after that Of course, FG got a lot less traffic than this place, so it probably caused us a lot less harm to do that. I wouldn't necessarily recommend doing that here.

 

[Cthulhu]

Hmm, we might have some kind of attack going on; if you look at the guest list and have user agents enabled (probably staff only), there's a few entries from various IPs (coming from Kiev, Minsk, Gibraltar, etc) with a user-agent string of "Synapse", which seems to be a Delphi TCP/IP library (see https://superuser.com/questions/146133/what-is-apache-synapse). It seems to do some crawling, but also various attempts at SQL injection and other exploits - login attempts, trying to open threads and users with negative IDs, etc. And an interesting HTTP COOK method, which doesn't even exist

I'll see if I can ban it,

Also yeah, I banned all of China a while ago, works great

 

[Ⓐaron]

Yes, kill it with fire

 

[Ami]

Site went down for a bit just a little while ago, got "Rate Limit Exceeded".

 

[Ghost X]

Does that mean we have to perform a limit break? I got dibs on Omnislash. Hurrhurrhurr.

 

[Flintlock]

IIRC the "rate limit exceeded" message is shown to users who have been using too much of TLS' resources. It happens to me when I accidentally leave the "top 10 stats" page open in a background tab, since that page auto-refreshes (I don't see why it needs to, but it's probably a vB default).

 

[Cthulhu]

Yeah, the top 10 stats thing is broken and will occasionally freak out and continuously refresh, thus spamming the server. Actually that might be the whole rate limit exceeded message - I have installed a system that will temporarily block anyone that performs too many requests to that particular resource in a short period of time. We've had major downtime a few times (see earlier in this thread) because some people's browsers went haywire with that script and spammed the server until it could no longer handle anyone else's requests.

(I had forgotten all about that one actually)

If you get the rate limit exceeded error, disable the top 10 stats on the front page. If you don't have that though, I could see to changing the settings for it.

Also I was making a fresh file backup this morning / fiddling with the backup script, since a few of the weekly file backups had failed.