Heads up: Big maintenance coming up soonish

So, apparently there's a pretty major bug in one of the core components of pretty much every server, ever . I don't think TLS is directly vulnerable, at least not yet, but it might just be a matter of time before someone finds out a way to abuse the Shellshock bug.

It turns out I haven't updated our server software for a while (not gonna tell how long etc, spais), but a round of updates is long overdue. I think I'll get cracking on Sunday, updating the operating system and various software packages we use (most notably Apache, PHP and MySQL). If all goes well the downtime shouldn't be more than a couple minutes while the server and individual packages restart; if it doesn't, though, downtime might be a bit longer.

There's also another option btw; I could rent a second server (maybe a little bit more powerful), set it up right, and once it's done move the databases over and switch out the domain name to the new server. I haven't heard of any errors recently though (but remind me if that was the case), so idk if an upgrade is required.

I'm also gonna shut down fhtagn (our test server); it's been doing nothing for a long while now and it's costing me money, money which I could use to buy beer and shit. Actually I could probably upgrade the server and keep the same monthly expenses if I shut that one down.

Tally-ho, good sir.

Goodbye, TLS

Cthulhu said:

I'm also gonna shut down fhtagn (our test server); it's been doing nothing for a long while now and it's costing me money, money which I could use to buy beer and shit. Actually I could probably upgrade the server and keep the same monthly expenses if I shut that one down.

Click to expand...

As all of my work on Twenty Twelve is on there, it would be marvelous if you'd take a back up of the stuff there before you do (which I guess you'd do anyway). If you hold your horses until Sunday I can do one myself as well. (fhtagn has been 500'ing for several months now by the way.)

That being said, I've said before that even for that update being 80% done, I'm not sure I'll ever BE done. It's a combination of lack of time, but also, dear owner, a lack of trust and encouragement from your end. Also, a lack of Flintlock.

Anyway now that I've wandered in the beautiful world of source control for a while and realised its necessity - if we want to keep developing TLS, we should totally set up a proper development environment.

Yeah I was pondering if I could get an image of that one and release it as a virtualbox package or something, but I don't think it'd be as easy as that (even if it's a virtual machine already). I'll make a backup of it; I'm pondering if I should just chuck it onto a public (open source) github repository or make it a private one, I don't think it'll harm anyone if it's open source. But I'd also have to check the licences.

More like a failure of me to create a virtual image or a development environment of sorts. And of course lack of my own motivation, but then, CBA is nothing new .

I'm kinda hoping updating stuff will help with the server errors, but TBH I doubt it.

Fangu said:

Also, a lack of Flintlock.

Click to expand...

I feel like I should respond to this.

I often think about how I'd love to get back to working on the front page, but then I realise that, just like before, I'd put lots of time and energy into it for a while, and then reach a point where I am left scratching my head about what to do next.

See this post from August 2013, some four months after I finished the most recent front page design draft. I asked "how can I help move this forward?" and the answer was, in a nutshell, "you can't right now".

Nobody ever got back to me on what they wanted me to do. Then a lot of things changed in my personal life and I chose to take some time away from the site. But even if I hadn't left, I doubt we would be any closer to the new goal, and that ultimately comes down to this:

dear owner, a lack of trust and encouragement from your end.

Click to expand...

It's difficult to work hard on something when it's not clear what the owner wants, or if they really care about it at all.

I'll work for TLS again, but I would want it to be in a much more structured manner than before. I would like to see somebody with a vision for the whole site - both the front page and the forum - put in charge of running the whole thing. If that person set out their plans and said "Flint, I want you to do X", I'm sure I would. I know there are upsides to the rather more anarchic model we're using at the moment (it appears to be working for Valve), but without some sort of tangible reward or at least recognition, it doesn't motivate me to contribute.

I'm glad you're upgrading server stuff though, Yop. We still get too many 500 server errors at the moment.

TBH I don't want a visual update, and I don't mind things as they are right now. So far, all of the updates re: styling have come from you guys, as did all the initiatives and actual work and shit - I just tried to facilitate when given direct requests. Like you said, what a project like a redesign needs is a clear leader - a product owner, if you will - that 'pulls' the work and gets everyone involved and shit. I'm not that person though, despite what some people might take me as, I'm not a leader. I can't be, because I don't care enough, .

I don't think I'm that person either. I'm happy to manage content and organise the front page staff, do events etc but when it comes to site redesign/ update I don't feel like it's entirely appropriate for me to be telling other people what to do when fangu and yop know code and I don't, and Flint has a vision for layout etc that I don't.

As it stands I do like it the way it is and I think we just need some minor tweaks. I do think we need to think about updating the forum software and we need to start updating our plugins and stuff, but when I talked about updating the plugins before I think it was met with "might cause instability, best not" or something.

If it were up to me we'd be using the latest version of everything and just dealing with the problems that it may or may not cause when it happens, but that's just how I work and I don't code so... that's where I am on this.

EDIT: With regard to site direction we need to start using SEO properly, get registered with google news (which Yop doesn't want to do because they want a real life address) and start pushing our articles through N4G when they're published. That'll get us hits and stuff. Content wise we're fine, but I'm looking at everything and planning a bit of a reorganisation because there are certain things that need updating but it's nothing I can't handle myself.

Not that we necessarily need a visual update, but, as I've said before, I do think Flint's proposal is superior to what we have now. I'm not going to promote treason if it doesn't happen, but I do think "cba" and an acknowledgement of a lack of personal motivation are silly reasons not to facilitate others carrying out what literally everyone else who has commented upon thinks is awesome, and which would really require a limited level of commitment from the one needing to do said facilitating.

Of course, I guess all this is moot if the one making that call really, really doesn't want it to happen, in which case, though, one should have said so a year and a half ago -- or at least said something other than "Looks very nice" -- before many pages of discussion about it followed and people put a lot of time into it.

^ I think that's pretty much the problem. Yop is allergic to having stuff done directly on the production server (which is generally good policy) unless it's something he can take care of himself. In that case, he fixes it straight on the prod server (olol gais I might break something) while I still have to go through fhtagn whenever I work on a change. Which means that every time I work on the new design, I have to check if the test server (fhtagn) is up to date before I even get to work, because I don't know whenif Yop has changed something. (His response will be 'olol I posted in the thread, so there you go'.) Then I need to update to the same version of WordPress, check if we're using the same plugins of the same version. Basically making fhtagn as similar to the production server as possible.

He is also known to step in and change things I've done without consulting me if he's not happy with it. 'Hurr it's my site' etc. When we did that major release, he went ahead and started the roll out without me even knowing it, even though I did most of the work. I think Shademp wanted to hide behind the nearest rock when I found out they'd started already. It takes away ownership, you know? When you work unpaid, you at least have to have that. It's bad psychology.


The problem, as I see it, is two parted:

1) Our owner and main tech is the same person. Techs want, naturally, to do as little as possible except basic maintenance, because stuff might break otherwise and people start whining and they have to spend their spare time fixing broken shit. As long as the tech is making the decision, there will be no progress because every time someone comes up with a good plan and goes through with it, you almost have to break his arm to have it rolled out.

2) Both Yop and I work with project management in one way or the other on a daily basis. Technology and people can be a tiresome mix - it's tl;dr. Which is why working on TLS was a great thing to do when in school, because it gave me the opportunity to work project based on something other than school projects. Now that I'm working, I simply can't be arsed with the whole spare time project management thing. I need to function as a tech only, a tech who has trust = space enough to solve the problems we need to solve.

I have never broken anything on TLS. I've never sunk the database or messed up in any other way, but to Yop, you're guilty until proven otherwise because he's the main tech and the main tech want to eliminate any possibility of failure.

I think it's wise of Lex to say 'hey, I manage the content, and that's it'.

Pretty much I don't think any of us can convince Yop to care enough to get to the level of care he needs to, and I understand him on that. Add into the mix that Yop and I haven't had the best friendship over the last year, and - well, the TLS tech projects were semi working when that was working, but he needs to find a co-tech he can trust.

I can't be arsed spending time on projects I feel no ownership to, that will be completely rewritten (to the point where Yop actually broke the code, outputting articles in the article list that was already in the previews) because 'he knows a better way to do it', without even testing that the result is according to the person requesting the change.


Kudos for preparing for that bash bug Yop. As a main tech, you do a really good job. I just needed to type this out to explain why the new design project crashed.

For what it's worth, I think this place could use an update in appearance. I'm not really sure how to bring that about, though. I don't know that much about design, certainly not enough to update the place's appearance myself, but I know enough to know that this site looks like it's several years old now (for example, gradients are apparently considered passé now). Some of the forums' appearance is probably impossible to fix; vBulletin has a table-based, boxy appearance that looks hopelessly stuck in the early 2000s, and that's likely to remain no matter how many template edits you make. (For what it's worth, vBulletin 4 and 5 look just as dated as vB3). But there are no doubt plenty of changes that could be made to the front site to make it look more current. I just haven't a clue what they are.

The appearance of a site probably isn't as important to attracting new members as the content or whatever, but an appearance overhaul probably can't hurt anything either. I agree with a lot of what Lex suggests except that I think upgrading the vBulletin will probably cause more problems than it solves. (For starters we'll have to completely redesign the skins from scratch and we'll probably have to get rid of the thanks system, which would make this place no longer feel like TLS). We could always try switching to a different forum software, I suppose.

FWIW, all that stuff Fangu mentioned about not being allowed on the live server was well over a year ago (IIRC); in the meantime, Aaron (who asked for SQL access and got it) b0rked the server at least once by doing queries right on the live server instead of asking me about a copy of the database, for example. He's also got quite a bit more experience, so I was expecting better of him.

Fangu on the other hand pretty much said "Yeah uh, I'm new to this" - php, styling, wordpress, server management, all that stuff. I don't think it's unreasonable for me to be reluctant to give someone full access, especially since at the time we didn't have a proper backup system in place (and TBH I won't say our current one works for sure until we've had an instance where we have to use it). I probably would've been less reluctant if she was more assertive (GIEF ACCESS NAO) instead of beating around the bush, and if she'd been a bit more experienced with the systems (which is why I did give her full access to the test server - even if I didn't maintain that one properly).

If anyone wants to keep working on the redesign (or start over, or whatever), just say so, and state what you need. I'm not going to let the brunt of development be done on the live server (for obvious reasons), but I can provide a backup of the site if needed. I can also create a virtual machine with a running webserver etc etc etc, but TBPH setting up a local development environment, installing Wordpress and importing a database should be part of one's basic skillset.

If my competence / arsedness / suitability to own this place is in question, feel free to send me an offer to take the site off of me - bearing in mind that I've only set it up and paid for it, I didn't turn this place into what it is today, so compensation for the people that worked on this site should be included.

Re: maintenance, I've manually updated Bash to at least partially fix the issue (they say even a second patch doesn't fix0r it, but apparently the issue's been there since probably 1992 and I doubt it'll be the last one). I'd still like to try and update the server though, so that future updates will be easier. Expect to see some downtime; first one will be a five-minute one while the database is backed up, then it'll probably be a few restarts until we're at the last version. If all goes well; if not, I can't make any estimates, could be hours or days.

Edit: Also, while this stuff is going down, I'll be in the TLS IRC; http://thelifestream.net/chat/ should keep working as long as you don't refresh the page, if not, the IRC is hosted at irc.ipocalypse.net, port 6667, channel #tls (although they changed their name to something else, we'll probably have to update that at one point too)

Cthulhu said:

If my competence / arsedness / suitability to own this place is in question, feel free to send me an offer to take the site off of me - bearing in mind that I've only set it up and paid for it, I didn't turn this place into what it is today, so compensation for the people that worked on this site should be included.

Click to expand...

I don't think anyone is questioning your suitability to own this place. At least, they shouldn't be. There's really nothing to question; it's yours.

I understand why you want to take a hands-off approach and I appreciate the things you do do for TLS. I just wish there was someone else you trusted enough to say "I'm the owner, but you're in charge of running the damn thing".

Edit: Also, while this stuff is going down, I'll be in the TLS IRC; http://thelifestream.net/chat/ should keep working as long as you don't refresh the page, if not, the IRC is hosted at irc.ipocalypse.net, port 6667, channel #tls (although they changed their name to something else, we'll probably have to update that at one point too)

Click to expand...

It's irc.idlechat.net now, and yes, we should update the chat page information.

Lex said:

As it stands I do like it the way it is and I think we just need some minor tweaks. I do think we need to think about updating the forum software and we need to start updating our plugins and stuff, but when I talked about updating the plugins before I think it was met with "might cause instability, best not" or something.

Click to expand...

Minor tweaks seems to be the way to go from here. There are still half a dozen small things that I'd like to be fixed before I can put my hands up and say "it's not perfect, but I'm happy". We should use the Our Site thread for that though.

I would if someone stepped up and said "I want to run this thing". That was easy enough with the content side of things - which I haven't involved myself in since the very beginning, keep up the good work - but it's kinda harder with the technical side of things, also - and excuse my shameless boasting here - because I'm probably the most knowledgeable and experienced guy around here with delusions of grandeur (FHTAGN), .

Cthulhu said:

FWIW, all that stuff Fangu mentioned about not being allowed on the live server was well over a year ago (IIRC); in the meantime, Aaron (who asked for SQL access and got it) b0rked the server at least once by doing queries right on the live server instead of asking me about a copy of the database, for example. He's also got quite a bit more experience, so I was expecting better of him.

Fangu on the other hand pretty much said "Yeah uh, I'm new to this" - php, styling, wordpress, server management, all that stuff. (...) I probably would've been less reluctant if she was more assertive (GIEF ACCESS NAO) instead of beating around the bush, and if she'd been a bit more experienced with the systems (which is why I did give her full access to the test server - even if I didn't maintain that one properly).

Click to expand...

It's not access rights I have issues with, because you've granted me those over time, even when I didn't ask for it but other people did FOR me (Tres, Lex) - I already have access to pretty much everything on the server right now, as I have admin rights to both the WP and forum DB (just not through the forum GUI). I don't go in unless I have to, and when I do, I'm very careful.

But I have to say I find it odd that you think Aaron had more server/DB knowledge than me to begin with. If that's the impression I've given off, I apologize, because that's not the case. I've known since the first discussion Aaron and I had about PHP and MySQL that I know more than him. idk maybe it's a female thing.

I guess it explains what I've always found to be strange - you rewrite my code into a syntax I can't understand, even if there was no need to do that in the first place, and then you give Aaron full access so he can run straining SQL queries to the prod database to check his Thanks count every 20 minutes.

Major shrug, you know. Herpaderp. idek.

Main difference being there that I've known Aaron for probably as long as I've been on the internets (so 8-10 years), a long time of which he's been in charge of forum software, whilst you came across as being completely new to anything related to software development, etc. Besides, he's only got access to the forum software (it's got an SQL field somewhere) and I think FTP access, not full-on ssh root access like you do. I hope.

Re: changing codes, it's not just you, I change everyone's code, .

Well you know as well as I that QA is about pointing out what you want changed and then letting the developer themselves change it

And, hm

• I've done HTML and CSS since 1997/ whenever CSS was introduced, making web sites, of which two were paid work
• From 2005 - 2009 I was mod on a phpBB forum for a large US comic book
• I've had my own domain since 2002 or whatnot
• I've set up countless WordPress blogs on this domain, tweaking them - before there was even WordPress, there was other blog software that didn't even have comment sections (those were plugins you had to manually install)
• I've worked with SQL in relation with said blogs, and also in relation to an Access DB when being given millions of order lines to analyze, create a DB, chuck into tables (normalize), analyze and present in Excel (this was 2009)
• I've tweaked an Access/ SQL db at work (the IT training dpt I worked for; their course registration handling system was MS Access towards an MSSQL db) - this was 2010
• iirc I started working on TLS in 2012?

So idek... I guess I failed to mention all of that Oh well.

You did, all I knew back then is that you worked as network support and wanted to get into software development,

ANYWAY, backup is almost done, update commences. Next time (if any) I'll do it ninja-style and will just wait for the panicked OMG TLS IS DOWN YOP HALP US, or do a much shorter announcement. I expected an 'ok good luck', not a spanish inquisition, .

Yes yes you are entitled to bask in the glory; good job Yop, well done

So um...like the quote buttons and reply buttons etc look strange - is that meant to happen? Not that I care off course - as long as they function.

Yop actually managed to upgrade vB (!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!) so yeah, a few things might look strange at first. If it persists over the next few days, use "Report Bugs", I guess

It's back to narmal now

Okay so that was a bit of a stressful afternoon,

So, got cracking, was bored for a long while because I had made a backup first, took a while to upload to Dropbox where we stash our backups for great justice (they recently increased pro members' storage space tenfold, so I should tweak the backup settings a bit so we can keep backups for longer - but I digress).

Then I did the ol' do-release-upgrade command, which will trigger the Ubuntu update mechanism. When I started we were at Ubuntu 11.04 I think, some version from 2011, which wasn't maintained anymore - regular Ubuntu versions are supported for about a year, I believe. The first upgrade went to 12.04, from 2012, which is an LTS release (long-term support) which gets security updates for five years. The main reason I started this project, the Bash exploit, would have been covered under that release.

Anyway, did the thing, twiddled my thumb, and wonder of wonders after about half an hour, it was done. The server restarted and... well, that was kind of the anticlimax of my day, since everything seemed to be working fine. The upgrade process did indicate there was a difference between config files between the ones we had and the ones that it wanted to install, but most of that configuration were attempted performance tweaks I did myself, things like allowing mysql to use more memory - I figured those were easy enough to redo.

Of course, I wasn't content - I figured since I'm at it, why not keep upgrading to (eventually) the 14.04 version, this year's long-term support version. So I did another do-release-upgrade because the other one seemed to be easy enough,

Might've had a more relaxed afternoon if I hadn't done that, lol. The site didn't work out of the box anymore after the update, and I had to spend some time trying to figure out what was wrong. The first issue was that there was some changes in how the webserver (Apache) communicates with PHP, something having to do with a socket file (everything's a file in linux) having changed permission. After some and editing configuration files, I managed to fix it - had to redo the basic webserver -> php configuration, and change the configuration given by every tutorial ever to look for a socket instead of a HTTP server somewhere. TL;DR stuff.

I think that's when the front page was back online, fhtagn! However, the forums weren't; it gave me the 'database error' screen, and since vB sucks, it didn't give me any clue what was actually wrong. However, after some googling, I quickly discovered that the version of vBulletin we're on - 3.8.4 I believe - isn't compatible with the newer versions of PHP or MySQL. The last upgrade apparently upgraded php to 5.5 and MySQL to 5.5, and it is said our version of vB doesn't support those. TBF, PHP made some pretty big changes in those versions, apparently.

Anyway, so first I went like "Oh that's okay we'll just downgrade php, easy right?". Well not rly, apparently; I lost my cool and started to blindly follow instructions on the internets, removing php and apache, etc. Didn't make it easier. I eventually just reinstalled apache and mysql, but that caused other problems - the front page gave a big-ass FORBIDDEN error, mainly.

I did some googling and tried a lot of shit, but eventually it turned out that the problem was in Apache, which was upgraded from 2.2 to 2.4. It seems that in that update, the configuration for granting or denying access to folders had changed. In the webserver configuration, there's a few blocks that go like this:


But that Order / allow thing had apparently changed. It took me far too long to find out that every instance of 'Order allow, deny allow from all' had to be changed to 'Require all granted' and similar statements.

Anyway, sorted that out, and I got the front page back online at least. The forums were still down though, so I should get to downgrading php / mysql again.

But then I went "fuck it". My nerves were already frayed by then, I knew our version of vB was outdated, and it was advertised that vB 3.8.8 would work with the newest versions of PHP and MySQL. So I took a gamble and decided to just renew our license and upgrade. I spent $200 - which was in my PayPal account, so we had that in accumulated donations at least - and renewed the license.

Downloaded vB 3.8.8, the latest (and probably last) of the 3.x versions, uploaded it, ran the install scripts, and... whoa, it just works!1one. TIL shelling out $200 to renew our vB license and upgrading vB is probably easier than downgrading php / mysql, .

Fangu reported that some of the images were borked though, but that's because we customized the images (icons and the like) by simply overwriting them; I don't know if that can be done on a per-theme basis actually. But I copied those over and they look fine again.

Let me know if something's still broken; I've only done a quick scan of things. I think FTP access isn't working right, I'll need to check that.

Also, since I bought the license, we should probably discuss upgrading to vBulletin 5; I don't know anything about it, but if I were to guess I'd say we'd have to redo the forum styling. But I'll have to dig into that I guess.

edit: sorry about that other downtime, I was making a post-update backup.