May 1, 2008

Massive technical troubles hit LANG (updated*, **)

Apparently the switchover to a faster Internet server has wreaked havoc across Singleton's SoCal empire. All* of the papers carry a version of the following note to readers on the front pages of their websites (this one is from the Daily News):

To our readers - Due to technical problems, some sections and regular features of the of the Daily News do not appear in today's paper and your edition may arrive late. We apologize for any inconvenience.

Indeed, it looks like the server was knocked completely off-line last night and continues to be down this morning. I'm told the morning papers look as though they were written on typewriters and that Unisys, the database reporters and editors use to actually create the paper, continues to be off-line as well.

Tim Berger, New Media Editor at the San Gabriel Valley Newspaper Group, sent out this memo:

There was a major failure with the main Unisys system last night and there is no indication yet when it will be restored. IT is working to bring up a backup system which, I'm told, has a lot less capacity and power than the main system and a limited number of users is allowed. Newsrooms please prepare ahead of time who is most necessary to be logged in. For now, reporters can likely write and edit stories using desktop software and e-mail them to editors. The photo department will be logged into SII for photo assignments. Send an SII message to SPEDS when you need something.

Meantime, blogs continue to be plagued by glitches - several blog links were broken when I checked them this morning, including ones trying to explain to readers what was happening. Posting new stories and comments can an hour or longer (assuming the system is even up and running).

Here's what sports reporter Fred Robledo had on his sports blog this morning:

I got a call last night that our system locked, froze or whatever, and that they couldn't get local stories in today's paper, you know, big ones like Gano "stepping away." And I'm sure you can tell we're having blog issues again, I don't even know how long it's going to take for this to post, but I'll try. For all of you that have left comments, I can see them in our system, they're just not moving over to the actual blog. When they get this fixed, I'll let you know immediately. Thanks for you patience.

Ryan Garfat, online editor at the Daily News, sent out this memo yesterday:


I have an inbox full of issues, so thank you for that. Apologies if I don't get back to you directly. It's impossible to do so with the amount of responses I am getting.

I am waiting on a request from MNGi for a status update since some of the issues we identified have not been fixed, including the preview function, some posts going up quickly- others not, the unreliability of scheduling posts and some funky commenting issues.

Once I hear from MNGi, I'll send another update. Sorry if you're sick of them, but it's the best way to keep you all informed.


*The Daily Breeze and Long Beach Press-Telegram did not have notes on their websites when I checked early this morning (maybe I didn't look in the right places), but they have added them now.

**The San Bernardino Sun has posted a more extensive editor's note on its website concerning the scope of the computer problems:

To our readers,

Because of major computer and system failures, Thursday's editions of The Sun, the Inland Valley Daily Bulletin and other newspapers in the Los Angeles Newspaper Group were abbreviated and missing features and certain sections. The problems have also affected our ability to post some of our news online.

We hope to have the issues resolved today.

The problem involved our main computer network for writing stories and producing pages, and was so extensive that even backup systems were largely immobilized. This forced us to use available computers and programs not connected to the network, and to reformat - and "tighten" - the newspapers in order to print them in time for distribution.

We apologize for the inconvenience, and, again, hope to have the issues resolved today.


Len Cutler said...

Trust me, this is nothing new for LANG.

It's a bit worse than usual, but their lowest-bidder approach to tech has caused many a long night 'round the LANG offices.

Jennifer said...

The Daily Breeze does have a note to readers, its at the top of the paper... in red. I can't remember what it says but its there. The Press-Telegram carries a similar one in the bottom right corner.

Bill said...

I have zero sympathy for this problem with LANG. That's what they get for going on the cheap. Hope it continues to bite them over and over and over.

Gary Scott said...

Jennifer, thanks for pointing that out. I see now that the "note to readers" memo is on the front pages of both websites.

Anonymous said...

If Singleton sees they can get the paper out on typewriters without the server using bad college fonts, this could be permanent.

Frank Girardot said...

Crime Scene blog remains down, but I'm running a mirror site at:

Anonymous said...

I guess Singleton is also cutting back on hamster chow. Feed them!

Anonymous said...

I'm confused. What is the connection to MNGi if LANG front end is down?

Sys guy in SLC

Anonymous said...

While the Unisys Hermes system may be many things, the cheapest isn't one of them. It's used by the Wall Street Journal and the Sacramento/Fresno/Modesto Bees and, last time I checked, came in around midway in the price spectrum.

This smells more like an administration problem than an architecture problem.

Len Cutler said...

The base system might be the same as used by the WSJ, but I'll bet there's something different in the licensing or cost structure, because there's no way the WSJ would tolerate the buggy performance and chronic downtime that's plagued LANG since they adopted the system.

I'm speaking from firsthand experience here. The first year of rollout for LANG had at least an outage every single month, lasting several hours or more every time, and the first three months had virtually zero functionality at all. Even now, they're still ironing out bugs in the system.

Anonymous said...

The hamsters can only run so fast. What's the big deal? There will another irrelevant paper on the driveway sometime tomorrow.

Anonymous said...

One good thing about this mess is that a couple of designers are back at the P-T to lay out the paper on Quark. So, at least for today, the newsroom doesn't feel so empty.

Emily said...
This comment has been removed by the author.
rknil said...

More of newspapers' misuse of technology. The root of these types of problems stems all the way back to using pagination to justify making copy editors into de facto designers.

"One good thing about this mess is that a couple of designers are back at the P-T to lay out the paper on Quark. So, at least for today, the newsroom doesn't feel so empty."

Yes, hallelujah! A couple of design dolts returned. Journalism as we know it is saved.

Anonymous said...

Tomorrow's paper should have this headline in 72pt type.


followed by a commentary

"Why are you still subscribing to this?"