Talk:Wikimedia servers unreachable for 3 hours due to network outage

Server

I experienced the downtime today. I couldn't even get on any of the Wiki's. FellowWikiNews (W) 20:18, 18 August 2006 (UTC)

Newsworthiness

Do we really have to report every incident where wikiaddicts like myself are on cold turkey for a few hours? --+Deprifry+ 20:29, 18 August 2006 (UTC)

Not really. FellowWikiNews (W) 20:31, 18 August 2006 (UTC)

I think it is news-worthy. Lots of people at my place of work (yes, I have a job) were bummed that they couldn't reach wikipedia. (sorry to report I am the only wikinews addict.) --SVTCobra 03:45, 19 August 2006 (UTC)

This is especially newsworthy as it appears as though some idiot of a network engineer fell asleep and decided to change the routing tables for all Wikimedia projects on the main network backbone. Some of the network routing information didn't get updated right away, notably AOL of all things. It doesn't inspire confidence in the network engineers at AOL as this update should have occured and instead their routers had old data that in this once instance worked to their favor. Usually that is a major bug that should be causing problems and may cause problems in the future for AOL.

In fact, on a minor note is that there was some rampant vandalism going on with Wikimedia projects (especially Wikipedia) as most of the admins were blocked but the vandals had free reign on killing Wikipedia (AOL usually has many vandals than typical). All of the servers for all Wikimedia projects were subsequently locked down and editing stopped for those few individuals who could actually get on to them.

When I did an IP traceroute for this, instead of going to the server farm in Florida, the ip route took me to Houston of all places. I guess this is the corporate hq for Cogent? The route terminated on cogent routers for me.

BTW, I consider information about the DCMA to be just rumors, but it will be interesting to see what was going on, as I think there was more to this as well. Regardless, Brion certainly had a busy day today. If I had to guess, there was a contract dispute between Cogent and the Wikimedia upstream ISP, or perhaps mis-reading the wrong contract that was terminated.. Likely somebody was told to change the routing information for a portion of close address space and they changed a larger block of addresses than was necessary. --Robert Horning 06:35, 19 August 2006 (UTC)

Bad link

"block" (after IP address) currently leads to WP's blocking policy... 71.224.81.82 20:39, 18 August 2006 (UTC)

I took away the link. FellowWikiNews (W) 20:43, 18 August 2006 (UTC)

It's a block of IP addresses. Nothing to do with blocking. :-) --Eloquence 20:44, 18 August 2006 (UTC)

OR notes

"Anonymous source" comes from Amgine on IRC, who is a member of Wikimedia's Communications Committee and very much in the loop.--Eloquence 21:00, 18 August 2006 (UTC)

Bush was in the loop but the WMD info was wrong as mud. Neutralizer 00:20, 19 August 2006 (UTC)

I'm not sure that's a fair comparison. 68.39.174.238 03:37, 19 August 2006 (UTC)

Funny though. --SVTCobra 03:47, 19 August 2006 (UTC)

When she is a member of Wikimedia's Communications Committee, why is there no statement on wikimedia.org? --81.173.164.199 06:13, 19 August 2006 (UTC)

I would point out that this story hasn't been fully told yet. There will likely be an "official" report about what happened at a later date, when everybody can finally understand what happened and a proper investigation occurs, as Brion pointed out. --Robert Horning 06:44, 19 August 2006 (UTC)

Also, Amgine is a he... :D Or a xe... :D --Chiacomo ^(talk) 06:49, 19 August 2006 (UTC)

and a self-described enigma :) Neutralizer 12:13, 19 August 2006 (UTC)

81.173.164.199's question is important,I think. I am annoyed that there is administrative secrecy creep underway throughout wikimedia which results in backroom IRC discussions and "in the loop" information being closely held for selective distribution to the rest of us (the masses) at the whim of the bosses. I find this communications methodology to be quite classist and "USAGOV like" in feel as well as reality. The janitors have taken over the building. Neutralizer 13:29, 19 August 2006 (UTC)

It should be understood that, much as we like want to, when dealing with other businesses not everything can be made public. This must wait until the entire world is open and wikified :-) 82.29.217.87 14:48, 20 August 2006 (UTC)

I want to add that I do not criticize the folks running the show at all; I think the rest of us are slacking off in our duties as revolving foundation stones and grass roots activists; we need to demand more immediate transparency when shit (good and bad) happens and then (maybe only then) we'll get it. Neutralizer 23:07, 19 August 2006 (UTC)