7123 – After the great LJ Crash of ’05

And, we’re back.

Thank you, Jambi.


Obligatory Star Wars ref:

TOSHI STATION, Tatooine (Imperial Press) — Local wizard Ben Kenobi yesterday announced he had detected a “great disturbance” that could possibly affect the entire universe.

It was as if millions of voices suddenly cried out in terror and were suddenly silenced,” Kenobi said. “I fear something terrible has happened.”

I’ve had three people (jokingly) accuse me of knocking down LJ. It wasn’t me!!

Our data center (Internap) lost all its power, including redundant backup power, for some unknown reason. (unknown to me, at least) We’re currently dealing with bringing our 100+ servers back online. Not fun. We’re not happy about this. Sorry… :-/ More details later.

Update #1, 7:35 pm PST: we’re up on ‘dirty’ power for now (it works, but it’s unreliable), and we’re working to assess the state of the databases. The worst thing we could do right now is rush the site up in an unreliable state. We’re checking all the hardware and data, making sure everything’s consistent. Where it’s not, we’ll be restoring from recent backups and replaying all the changes since that time, to get to the current point in time, but in good shape. We’ll be providing more technical details later, for those curious, on the power failure (when we learn more), the database details, and the recovery process. For now, please be patient. We’ll be working all weekend on this if we have to.

Update #2, 10:11 pm: So far so good. Things are checking out, but we’re being paranoid. A few annoying issues, but nothing that’s not fixable. We’re going to be buying a bunch of rack-mount UPS units on Monday so this doesn’t happen again. In the past we’ve always trusted Internap’s insanely redundant power and UPS systems, but now that this has happened to us twice, we realize the first time wasn’t a total freak coincidence. C’est la vie.

Update #3: 2:42 am: We’re starting to get tired, but all the hard stuff is done at least. Unfortunately a couple machines had lying hardware that didn’t commit to disk when asked, so InnoDB’s durability wasn’t so durable (though no fault of InnoDB). We restored those machines from a recent backup and are replaying the binlogs (database changes) from the point of backup to present. That will take a couple hours to run. We’ll also be replacing that hardware very shortly, or at least seeing if we can find/fix the reason it misbehaved. The four of us have been at this almost 12 hours, so we’re going to take a bit of a break while the binlogs replay… Again, our apologies for the downtime. This has definitely been an experience.

Update #4: 9:12 am: We’re back at it. We’ll have the site up soon in some sort of crippled state while the clusters with the oldest backups continue to catch up.

Update #5: 1:58 pm: approaching 24 hours of downtime… *sigh* We’re still at it. We’ll be doing a full write-up when we’re done, including what we’ll be changing to make sure verify/restore operations don’t take so long if this is ever necessary again. The good news is the databases already migrated to InnoDB did fine. The bad news (obviously) is that our verify/restore plan isn’t fast enough. And also that some of our machine’s storage subsystems lie. Anyway, we’re still at it… it’s long because we’re making sure to back up even the partially out of sync databases that we’re restoring, just in case we encounter any problems down the road with the restored copy, we’ll be able to merge them. And unfortunately backups and networks are too slow.

slashdot reported on it, and brad was kind enough to supply some tech details

The Art of William Schmidtmorbid and fun. [via]

Oh I agree…

“This is film feminism in the 21st century: Take a cruel, bloodthirsty, homicidal character, but instead of making the character a man, make it a woman. Then pretend you’ve created an exemplar of female strength. I liked this fake feminism better when it was called misogyny.”

-Mick LaSalle San Francisco Chroinicle (referring to Electra.. but it could be any of a big bundle of flicks lately.)

Pentagon reveals rejected chemical weapons

THE Pentagon considered developing a host of non-lethal chemical weapons that would disrupt discipline and morale among enemy troops, newly declassified documents reveal.

Most bizarre among the plans was one for the development of an “aphrodisiac” chemical weapon that would make enemy soldiers sexually irresistible to each other. Provoking widespread homosexual behavior among troops would cause a “distasteful but completely non-lethal” blow to morale, the proposal says.

Other ideas included chemical weapons that attract swarms of enraged wasps or angry rats to troop positions, making them uninhabitable. Another was to develop a chemical that caused “severe and lasting halitosis”, making it easy to identify guerrillas trying to blend in with civilians. There was also the idea of making troops’ skin unbearably sensitive to sunlight.

The proposals, from the US Air Force Wright Laboratory in Dayton, Ohio, date from 1994. The lab sought Pentagon funding for research into what it called “harassing, annoying and ‘bad guy’-identifying chemicals”. The plans have been posted online by the Sunshine Project, an organization that exposes research into chemical and biological weapons.

Spokesman Edward Hammond says it was not known if the proposed $7.5 million, six-year research plan was ever pursued.

Man… I wish.

How’d you win the war, grampa? Oh, it was easy, once we had the enemy buggering themselves while angry wasps and rats closed in on them, causing them to die severely sunburned and uttering their last words on really stinky breath.

Brief sojourn out with Danny today… sushi, a little mage knight (at the library, because twistyland had a yu-gi-oh tourney going on), bad traffic and pleasant conversation. we listened to Doug’s old cassette.. Danny’s been listening for about 2 weeks now. I have to find Doug’s number, so Danny can call him up to say howdy.

I loaned him the pirate game and a few figures for him to teach his sci-fi club how to play next Friday. We had a nice time.

Current music’s Lyrics – Now You’re A Man – DVDA


What makes a man?
Is it the power in his hands?
Is it his quest for glory?
Give it all you’ve got,
to fight to the top
So we can know your story

Now you’re a man

A man, man man
Now you’re a man

A manny manny man
A man man man

You are now a man
You’re a man
Now you’re a man

Live it, Live it

What make a man?
Is it the woman in his arms?
Just cause she has big titties?
Or is it the way he fights every day?
….No, it’s probably the titties

Now you’re a man

A man, man man
Now you’re a man-man

Man-man man-man man
Now you’re a man

M-A-N, man
Man, man-man man
Now you’re a man

1 year ago – DVD cereal, Mars, cluck cluck cluck

2 years ago – smokies, palm synch, empenadas, Malawi Vampires

3 years ago – Not currently available – 6 entries [update – nice walk, moody, gave sappho a lj code, midnight pooper, newt patrols]

4 years ago – pictures of me with tarpo at No-nose maldanado’s, company on the rocks, ramble on humanity, temptation island, super-soldiers, link to 100 scotto-things

Leave a Reply