The Bad Script Trip

I got some bad script the other day and it took me down for almost 36 hours.

But it was an interesting experience that opened my eyes to the the risks of taking code from various places and running it in the same place (my blog).

There is a new form of online behavior emerging and that is a massive sharing of hmtl, flash, and javascript code on blogs and social network pages. It's a huge part of the myspace phenomenom and I have embraced it with all the blog bling I've got in my sidebars.

But there is a downside to this massive experimentation with script of various kinds.

As Scott said in his comment about the page load problems:

It's certainly to do with all of the "bling".

First, I did an HTTP trace on your main page. It requires well over 300 HTTP requests to load the whole thing. That, in itself, does not "cause" slowness, but it's just the first point I wanted to make. But the fact is that those 300 HTTP request took over 1 minute to complete on a 45 Mbps connection. That's pretty slow for a single web page to load.

The second point is that most of those HTTP calls are distributed over numerous service providers and, by extension, "hosts". This means that each hostname must be looked up by the DNS client on the requesting computer (the user's PC). DNS can be flaky and has been for a few sites recently. But even when operating optimally, each request takes a little bit of time. So each little bling item you add requires numerous extra queries by the client computer.

When even one of these name lookups fails or has a delay, it generally causes the whole page to load only partially or, in some extreme cases, not at all.

Each additional widget or bling introduces more chance of this happening -- just like adding more disks to a system means one is more likely to have a failed disk.

By my count, my computer had to look up 37 different hosts just to load the first page. Once they have all been successfully looked up, the DNS client (and local DNS server -- long story) on the user's computer (or network) generally caches the results and subsequent requests can go to the web servers without having to do the lookup first. That explains why you don't see any noticable issues -- you've already looked up all the hosts.

I've noticed some inefficiencies in the way publishers, ad services, partners and the like have set up their services, but there very well may be good infrastructure or service reasons they have done it, so I'll reserve comment on that and just assume they know what they are doing.

So while the web has made it easy for consumers to mashup web services directly on their own pages, there is a looming problem with the architecture of all of this. One bad piece of code can take down the whole shebang, like what happened to me last week.

And what if some bad code on the page impacts someone else's code? My page stopped loading at the part of the left sidebar where the coComment widget started. And so many people guessed that it was coComment that was causing the problem. And taking down the coComment widget fixed the problem, but they could have just gotten hit with a conflict from some other script on the page.

The coComment guys said in an email to me:

For sure, the addition of multiple scripts coming from different places makes the all page a bit unstable. Each service is adding its portion of javascript, each one interacting indirectly with others.

At one point last week, the finger pointing got pretty funny. It was like watching the phone company and the phone equipment company cast blame at each other. Here are some of the comments I got from various places on thursday and friday:

Firefox shows javascript errors in oddcast, indeed, overture and google syndication.

one of our developers traced the error back to a call to
vhost.oddcast.com/vhost_embed_function.php?acc=23325&js=1&followCursor=1

To me, the slow part of the page seems to be:
Email this • Add to del.icio.us • View CC license • Subscribe to this
feed • Digg this • Take my survey • Advertise In This Feed • Reddit It

As my friend Brad Feld pointed out in an email, nearly all of my portfolio companies got blamed at some point in the debugging process!

The bottom line is I have a ton of other people's script running on my page generating something like 153 errors on the page. That's a bad script trip for sure. But I am not going to stop experimenting. It's too much fun.

Comments

[Marcelo Calbucci]

The most important thing I think you forgot to mention about all this bling is the security risks posed by running many scripts on your page. For example, right now, as I type this comment, one of your widgets could capture that information and send to their server. Or, it could steal my cookie on your site to extract my email address, and so on.

There is no such concept as "trusted javascript" on the Web, so people should really trust the manufacturer of a piece of javascript before they add it to their blog or site, and keep an eye for any change in the future on that script.

[Charlie Crystle]

I wonder how many of those scripts really need to make real-time http requests. The fastest sites are usually ones with low graphics and pre-rendered pages; yours requires new rendering every time, but the content doesnt' really change enough to warrant the extra overhead. That said, it seems there's an opportunity to create an optimizing infrastructure as a service to the blog bling services; Six Apart should offer it...some variation on caching and pre-rendering at the Typepad server (centralized) rather than out at our browsers (decentralized), which requires the computing power(CP) equal to CP * PageViews * Http Request * Unknown, where unknown is the load your site puts on the overall web infrastructure. Your site is, unwittingly, like the Humvee of web efficiency and consumption.

Bling on, bling on.

[PRoales]

Good post, hinting at some of the challenges that may loom ahead for the mashup world.

When does this blog become an important enough for what you’re doing at Union Square that it should no longer be a test bed?

If this blog is indeed a major publicity vehicle, maybe features should only be rolled out only when deemed stable. With a system for testing them first on a "CrunchNotes" style blog, then only later promoting them up to the big time.

Just some thoughts, I would like to interject into what could be a large discussion on the new challenges here in mashups and real time public facing work.

[rob]

Maybe, as a variation on Charlie's comment, there could be a service out there that is blog platform agnostice that hosts these services - maybe something like an Amazon S3 type service or something like that. I wonder if it would be possible to set up a P2P service like Seti@Home that could massively distribute this caching across the web - maybe taking advantage of the 10's of millions of blogs out there to host something on their machines and have those become the infrastructure that supports these applications.....

[Howard]

I read AVC through my Newsgator reader in outlook. That keeps me immune to most of the problems. I miss the bling, but get the meat of your work. About once a month I get to the site. What fraction of your readers actually download the pages vs using a reader?

[Craig Fitzpatrick]

Great post. With all the Web 2.0/mash-up/its-so-easy hype these days, it's like the whole world has forgotten that when you're building anything, architecture and design are still pretty important. Yes you CAN build something in 2 weeks that looks pretty cool, but as a friend of mine said, "how WELL was it built?"

Lest we forget.

[fred]

howard,

about 20% (11k) readers access the blog through the feed. but i am sure they are the most active/loyal readers.

fred

[Rick]

It's like you're collecting web applications. I like it.

[Jonathan Peterson]

I hadn't loaded your blog in an actual browser in quite a while as I read it through bloglines. I've got to tell you, it's magnificently horrible.

While you're blog LOOKS nice, functionally it's like a circa 1984 Mac Write document that uses 27 different fonts just because it's possible.

This stuff will likely all shake out in fairly short order as various blog platform "plug-in" api standards evolve. Done correctly the plug-in should be tested for compatibility and optimized for performance.

[mark slater]

i read your post through rss reader so dont get any issues. You are luckly - OM gets huge script errors even in a reader.

[pkenjora]

Don't most people run analytics? For example the arkayne widget drives traffic directly from other pages on the network so analytics should pick that up.

Alot of these widget are built to drive traffic away from your page and to their central hub. This in turn raises your ranking if you make the front page. Unfortunately youre competing with severl million other pages so thats not likely to happen.

In any event I think ananlytics would do the trick unless the widget provides something ananlytics doesn't.