The incident was caused when a bug in CrowdStrike’s code made it out into production. The results were catastrophic: It caused roughly 8.5 million servers to crash. Hospitals weren’t able to serve the people that needed them. Airline passengers were stranded. People couldn’t access their money in banks. Folks in distress could not get the emergency services they needed. On top of that, the financial fallout is estimated to be somewhere around US$10 billion.

Bugs happen. I’d hate to be the person who was responsible for that particular bug (or the people in the quality assurance team that should have caught it), but the reality is that none of us who write code write perfect code. We all make mistakes and sometimes those mistakes make it into production. Other times, the code we write works perfectly during development and testing, but causes an unexpected issue in production. Sometimes only in very specific “edge case” circumstances that we didn’t have the foresight to consider.

Which brings me to the lesson I took away from the CrowdStrike incident: minimize the impact dependencies can have on your customers’ ability to complete critical tasks. In other words, develop dependency awareness.

The web is a hostile operating environment. Sometimes network connections are slow to resolve or time out completely, which may cause issues with your JavaScript, may result in broken images or videos, or could make it so your user never receives your CSS. Sometimes 3rd party scripts ship bugs that can hose your site completely. Sometimes a customer’s browser plugin can wreak havoc on your site by adjusting markup or injecting code. If any (or all) of those things were to happen, could your customers still accomplish their key tasks? Could they even understand your site at all?

This is why it’s so critical to start with a fully-functional website that relies only on semantic, accessible HTML and regular ol’ links and form submissions. They aren’t sexy, but they’re solid. Then progressively enhance that experience to improve things when the CSS is downloaded. And improve some more when your JavaScript executes properly. Build an awareness for the kinds of dependencies you have in your code so you can ensure there is always a fallback.

When I think about building robust websites like this, I often think of the Chrysler Imperial. The 1964-1966 model is one of the few cars that has been outright banned from entering demolition derby events. It is just too well built. It just takes the hits and keeps on driving. We should aspire to that kind of resilience in the websites we build.

Bugs happen. Can your site withstand them? Or will you let the failure of a single dependency (a.k.a., site fragility) ruin your customers’ day?

https://www.youtube.com/watch?v=-9GGDOUDLhc&start=7&end=17

When you set up a custom domain with GitHub pages, you have to point your domain at GitHub’s servers. There are a bunch of ways to do this, but if you use an A record, you need to be careful with your DNS settings. The site in question had a wildcard hostname (*) A record pointed at GitHub’s servers. At the time I’d set it up, that was the recommendation if you wanted all traffic to go to the same place.

Fast forward a few years and it’s become a known exploit of GitHub Pages: when wildcard hostnames are in play, any repo can add a CNAME file to their repository and claim ownership of a hostname belonging to that domain. GitHub even warns you not to do this anymore, but I hadn’t checked the docs in years. In my particular case, it was an archived domain that I don’t really use anymore, but I wouldn’t have been aware of the DNS hijack if the attacker hadn’t taken the step of claiming the domain on Google’s Webmaster Central.

Thankfully the fix was simple: Remove the wildcard A record and point the Apex domain at GitHub’s IP addresses.

If you use GitHub pages to host any of your own domains, I highly recommend auditing their DNS records to ensure this doesn’t happen to you. You can also use domain verification for GitHub Pages and organizations to further protect yourself.

When I was mid-way through writing the Second Edition of Adaptive Web Design, I realized that it was pretty likely some of the links I was referencing might disappear over the years. Little did I know, some of them would disappear while I was writing the book!

The Internet Archive’s Wayback Machine is pretty good, but it doesn’t archive everything, and I often find captured pages end up broken—especially if they rely heavily JavaScript, but often images go missing as well. I wanted to make sure that when you pick up the book a year from now or even 10 years from now, the links will still work.

I evaluated a few options for creating a permanent archive of each and every link in the book (there are over 200), but then it dawned on me that Perma.cc might be the perfect answer.

Perma.cc was created by the Harvard Library Innovation Lab in reaction to the paper I mentioned earlier. It is a distributed archive of URLs for scholarly and legal documents, supported not only by Harvard, but over 90 (and counting!) libraries, distributed all over the world. It’s also open source. Each URL is preserved as a live view, an archived view, and a screen capture taken when the link is added. Archived URLs are kept for a minumum of 2 years, but may be “vested” into the permanent archive by a member organization.

I had contributed some CSS to the project a while back, so I reached out to my contacts to see if they might be interested in vesting all of the links for the book. Turns out they were big fans of the First Edition and enthusiastically offered their support.

Converting all of the links took time (and a lot of double- and triple-checking), but the result is that every article, blog post, and web page that I referenced in the book will remain accessible to you in perpetuity. I think that’s pretty awesome. And, as an added bonus, since Perma.cc creates unique URLs that are relatively short, those of you who read it in print won’t have to re-type the often incredibly-lengthy original URLs.

I can’t thank Matt Phillips, Adam Ziegler, Jack Cushman, and everyone else at the Harvard Library Innovation Lab enough for creating Perma.ccand for offering their service to my readers. You all are amazing!

Here’s a roundup of some of my “favorite” network issue related headlines from the last few years:

• Sky Broadband misclassified the jQuery CDN as a malware site and broke much of the web for their users.
• Comcast admitted to injecting self-promotional advertising into web pages served by their Xfinity routers. (They have also been called out for artificially inflating subscriber bandwidth usage with their own crap.)
• United was recently called out for blocking access to the New York Times on their in-flight Wi-Fi.
• Someone discovered AT&T was injecting CSS, images, and JavaScript into pages served via their airport hotspots.
• Samsung smart TVs were found to be injecting video ads into video streaming apps.
• Sprint injects JavaScript into pages served via its data connections.
• Browser add-ins can inject their own advertisements. They can also alter the DOM, load conflicting versions of JavaScript libraries, and more. Awesome, I know. (This is being addressed, but is a persistent issue when add-ins have the ability to manipulate the DOM.)

Some of these issues can be avoided by serving content over HTTPS, but that still won’t enable you to bypass things like firewall blacklists (which led to the jQuery outage on Sky). Your best bet is to design defensively and make sure your users can still accomplish their goals on your site when some resources are missing or markup is altered.

We can’t control what happens to us in this world, we can only control our reaction to it.

Getting a website successfully delivered to a visitor depends on a series of actions. My server must spit something out. That something must be passed over some network. That something must then be consumed by another something: some client (often a browser) on some device. Finally, the visitor views that something in whatever context they happen to be in.

There are a lot of unpredictable layers here.

See also (from this site):

• A Fundamental Disconnect
• The Network Effect

Back-enders work with only one environment, their server. They have the option of changing their environment; for instance by installing new software or upgrading the hardware.

Front-enders work with an unknown number of radically different environments, ranging from state-of-the-art desktop computer to three-year-old, crappy phones with limited memory and processor time. They cannot change these environments because they have no power over their users’ browsers. Still, their code should work in all of them.

I shared similar thoughts in my post A Fundamental Disconnect.

This is a topic that has been weighing on my mind for quite some time now. I’ll start by saying that I’m a big fan of JavaScript. I write a lot of it and I find it incredibly useful, both as a programming language and as a way to improve the usability and accessibility of content on the Web. That said, I know its limitations. But I’ll get to that in a minute.

In the early days of the Web, “proper” software developers shied away from JavaScript. Many viewed it as a “toy” language (and felt similarly about HTML and CSS). It wasn’t as powerful as Java or Perl or C in their minds, so it wasn’t really worth learning. In the intervening years, however, JavaScript has changed a lot.

Most of these developers first began taking JavaScript seriously in the mid ’00s when Ajax became popular. And with the rise of JavaScript MVC frameworks and their ilk—Angular, Ember, etc.—many of these developers made their way onto the Web. I would argue that this, overall, is a good thing: We need more people working on the Web to make it better.

The one problem I’ve seen, however, is the fundamental disconnect many of these developers seem to have with the way deploying code on the Web works. In traditional software development, we have some say in the execution environment. On the Web, we don’t.

I’ll explain.

If we’re writing server-side software in Python or Rails or even PHP, one of two things is true:

1. We control the server environment: operating system, language versions, packages, etc.; or
2. We don’t control the server environment, but we have knowledge of it and can author your program accordingly so it will execute as anticipated.

In the more traditional installed software world, we can similarly control the environment by placing certain restrictions on what operating systems our code can run on and what the dependencies for its use may be in terms of hard drive space and RAM required. We provide that information up front and users can choose to use our software or use a competing product based on what will work for them.

On the Web, however, all bets are off. The Web is ubiquitous. The Web is messy. And, as much as we might like to control a user’s experience down to the very pixel, those of us who have been working on the Web for a while understand that it’s a fool’s errand and have adjusted our expectations accordingly. Unfortunately, this new crop of Web developers doesn’t seem to have gotten that memo.

We do not control the environment executing our JavaScript code, interpreting our HTML, or applying our CSS. Our users control the device (and, thereby, its processor speed, RAM, etc.). Our users choose the operating system. Our users pick the browser and which version they use. Our users can decide which add-ons they put in the browser. Our users can shrink or enlarge the fonts used to display our Web pages and apps. And the Internet providers that sit between us and our users, dictating the network speed, latency, and ultimately controlling how—and what part of—our content makes it to our users.

All we can do is author a compelling, adaptive experience, cross our fingers, and hope for the best.

The fundamental problem with viewing JavaScript as the new VM is that it creates the illusion of control. Sure, if we are building an internal Web app, we might be able to dictate the OS/browser combination for all of our users and lock down their machines to prevent them from modifying those settings, but that’s not the reality on the open Web.

The fact is that we can’t absolutely rely on the availability of any specific technology when it comes to delivering a Web experience. Instead, we must look at how we construct that experience and make smarter decisions about how we use specific technologies in order to take advantage of their benefits while simultaneously understanding that their availability is not guaranteed. This is why progressive enhancement is such a useful philosophy.

The history of the Web is littered with JavaScript disaster stories. That doesn’t mean we shouldn’t use JavaScript or that it’s inherently bad. It simply means we need to be smarter about our approach to JavaScript and build robust experiences that allow users to do what they need to do quickly and easily even if our carefully-crafted, incredibly well-designed JavaScript-driven interface won’t run.

A Comcast spokesman told Ars the program began months ago. One facet of it is designed to alert consumers that they are connected to Comcast’s Xfinity service. Other ads remind Web surfers to download Xfinity apps, Comcast spokesman Charlie Douglas told Ars in telephone interviews.

I wish I could say this is surprising, but it’s not: Any service that routes your content has the opportunity to modify the response being returned. Comcast is exploiting that opportunity and injecting JavaScript that, in turn, injects the ads.

The fact that middlemen can manipulate server responses is one reason Google is pushing for all sites to be served under HTTPS. With traffic running to and from your server in an encrypted fashion, man-in-the-middle attacks—which, if we’re honest, is what this amounts to—become much more difficult.

Assuming you can’t run under HTTPS for one reason or another, how do you harden your web thang against 3rd party manipulation you can’t control? What if Comcast’s JavaScript interferes with your own? Remember when Sky blocked jQuery for all of their customers? That was a bad couple of hours for most UK-based internet users.

Comcast’s move only serves to remind us—yet again—that we don’t control how our sites are delivered or what our users see. Or rather we do, but only up to a point. So rather than focus on some ideal experience we expect everyone to have, we must focus on building great experiences that work in a variety of contexts and situations.

We need to develop the 1964 Chrysler Imperial of websites: Sites that soldier on even when they are getting pummeled from all sides. After all, browsers, plug-ins, users, networks, and, yes, even the very routers that deliver our connections all have a say in how (and what) content gets to our users.

I’ll leave you with this scary quote from the Ars piece:

Security expert Dan Kaminsky said in an e-mail that JavaScript injection has the potential to break “all sorts of stuff, in that you no longer know as a website developer precisely what code is running in browsers out there. You didn’t send it, but your customers received it.”

Hooray!