I had the original idea for “projections” way back in 2019. Inspired by OS X’s Dashboard Widgets and Adobe AIR, I’d begun to wonder if it might be possible to project a component from a website into those kinds of surfaces. Rather than building a bespoke widget that connected to an API, I thought it made sense to leverage an installed PWA to manage those “projections.” I shared the idea at TPAC that year and got some interest from a broad range of folks, but didn’t have much time to work on the details until a few years later.
In the intervening time, I kept working through the concept in my head. I mean in an ideal world, the widget would just be a responsive web page, right? But if that were the case, what happens when every widget loads the entirety of React to render their stock ticker? That seemed like a performance nightmare.
In my gut, I felt like the right way to build things would be to have a standard library of widget templates and to enable devs to flow data into them via a Service Worker. Alex Russell suggested I model the APIs on how Notifications are handled (since they serve a similar function) and I was off to the races.
I drafted a substantial proposal for my vision of how PWA widgets should work. Key aspects included:
iframe
);After continuing to gently push on this idea with colleagues across Microsoft (and beyond), I discovered that the Windows 11 team was looking to open up the new Widget Dashboard to third-party applications. I saw this as an opportunity to turn my idea into a reality. After working my way into the conversation, I made a solid case for why PWAs needed to be a part of that story and… it worked! (It no doubt helped that companies including Meta, Twitter, and Hulu were all invested in PWA as a means of delivering apps for Windows.)
While the timeline for implementation didn’t allow us to tackle the entirety of my proposal, we did carve out the pieces that made for a compelling MVP. This allowed us to show what’s possible, see how folks use it, and plan for future investment in the space.
Sadly, it meant tabling two features I really loved:
I’m sincerely hopeful these two features eventually make their way to us as I think they truly unlock the power of the widget platform. Perhaps, with enough uptake on the current implementation, we can revisit these in the not-too-distant future.
To test things out, I decided to build two widgets for this site:
Both are largely the same in terms of their setup: They display a list of linked titles from this site.
Given that they were going to be largely identical, I made a single “feed” template for use in both widgets. The templating tech I used is called Adaptive Cards, which is what Windows 11 uses for rendering.
Adaptive Card templates are relatively straightforward JSON:
{
“type”:“AdaptiveCard”,
“$schema”:“http://adaptivecards.io/schemas/adaptive-card.json”,
“version”:“1.6”,
“body”:[
{
“$data”:“${take(items,5)}”,
“type”:“Container”,
“items”:[
{
“type”:“TextBlock”,
“text”:“${title}”,
“wrap”:true,
“weight”:“Bolder”,
“spacing”:“Padding”,
“height”:“stretch”
}
],
“height”:“stretch”
}
],
“backgroundImage”:{
“url”:“https://www.aaron-gustafson.com/i/background-logo.png”,
“verticalAlignment”:“Bottom”,
“horizontalAlignment”:“Center”
}
}
What this structure does is:
items
from the data being fed into the template (more on that in a moment);item
andtitle
and url
keys from the item
object)The way Adaptive Cards work is that they flow JSON data into a template and render that. The variable names in the template map directly to the incoming data structure, so are totally up to you to define. As these particular widgets are feed-driven and this site already supports JSONFeed, I set up the widgets to flow the appropriate feed into each and used the keys that were already there. For reference, here’s a sample JSONFeed item
:
{
“id”:“…”,
“title”:“…”,
“summary”:“…”,
“content_html”:“…”,
“url”:“…”,
“tags”:[],
“date_published”:“…”
}
If you want to tinker with Adaptive Cards and make your own, you can do so with their Designer tool.
With a basic template created, the next step was to set up the two widgets in my Manifest. As they both function largely the same, I’ll just focus on the definition for one of them.
First off, defining widgets in the Manifest is done via the widgets
member, which is an array (much like icons
and shortcuts
). Each widget is represented as an object in that array. Here is the definition for the “latest posts” widget:
{
“name”:“Latest Posts”,
“short_name”:“Posts”,
“tag”:“feed-posts”,
“description”:“The latest posts from Aaron Gustafson’s blog”,
“template”:“feed”,
“ms_ac_template”:“/w/feed.ac.json”,
“data”:“/feeds/latest-posts.json”,
“type”:“application/json”,
“auth”:false,
“update”:21600,
“icons”:[
{
“src”:“/i/icons/webicon-rss.png”,
“type”:“image/png”,
“sizes”:“120x120”
}
],
“screenshots”:[
{
“src”:“/i/screenshots/widget-posts.png”,
“sizes”:“387x387”,
“label”:“The latest posts widget”
}
]
}
Breaking this down:
name
and short_name
act much like these keys in the root of the Manifest as well as in shortcuts
: The name
value is used as the name for the widget unless there’s not enough room, in which case short_name
is used.tag
as analogous to class
in HTML sense. It’s a way of labeling a widget so you can easily reference it later. Each widget instance will have a unique id created by the widget service, but that instance (or all instances, if the widget supports multiple instances) can be accessed via the tag
. But more on that later.description
key is used for marketing the widget within a host OS or digital storefront. It should accurately (and briefly) describe what the widget does.template
key is not currently used in the Windows 11 implementation but refers to the expected standard library widget template provided by the system. As a template library is not currently available, the ms_ac_template
value is used to provide a URL to get the custom Adaptive Card (hence “ac”) template. The “ms_” prefix is there because it’s expected that this would be a Microsoft-proprietary property. It follows the guidance for extending the Manifest.data
and type
keys define the path to the data that should be fed into the template for rendering by the widget host and the MIME of the data format it’s in. The Windows 11 implementation currently only accepts JSON data, but the design of widgets is set up to allow for this to eventually extend to other standardized formats like RSS, iCal, vCard, and such.update
is an optional configuration member allowing you to set how often you’d like the widget to update, in seconds. Developers currently need to add the logic for implementing this into their Service Worker, but this setup allows the configuration to remain independent of the JavaScript code, making it easier to maintain.icons
and screenshots
allow us to define how the widget shows up in the widget host and how it is promoted for install.When someone installs my site as a PWA, the information about the available widgets gets ingested by the browser. The browser then determines, based on the provided values and its knowledge of the available widget service(s) on the device, which widgets should be offered. On Windows 11, this information is routed into the AppXManifest that governs how apps are represented in Windows. The Windows 11 widget service can then read in the details about the available widgets and offer them for users to install.
As I mentioned earlier, all of the plumbing for widgets is done within a Service Worker and is modeled on the Notifications API. I’m not going to exhaustively detail how it all works, but I’ll give you enough detail to get you started.
First off, widgets are exposed via the self.widgets
interface. Most importantly, this interface lets you access and update any instances of a widget connected to your PWA.
When a user chooses to install a widget, that emits a “widgetinstall” event in your Service Worker. You use that to kickoff the widget lifecycle by gathering the template and data needed to instantiate the widget:
self.addEventListener(“widgetinstall”,event=>{
console.log(</span><span class="token string">Installing </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>event<span class="token punctuation">.</span>widget<span class="token punctuation">.</span>tag<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">
);
event.waitUntil(
initializeWidget( event.widget )
);
});
The event argument comes in with details of the specific widget being instantiated (as event.widget
). In the code above, you can see I’ve logged the widget’s tag
value to the console. I pass the widget information over to my initializeWidget()
function and it updates the widget with the latest data and, if necessary, sets up a Periodic Background Sync:
asyncfunctioninitializeWidget(widget){
awaitupdateWidget( widget );
awaitregisterPeriodicSync( widget );
return;
}
The code for my updateWidget()
function is as follows:
asyncfunctionupdateWidget(widget){
const template =await(
awaitfetch(
widget.definition.msAcTemplate
)
).text();
const data =await(
awaitfetch(
widget.definition.data
)
).text();
try{
await self.widgets.updateByTag(
widget.definition.tag,
{ template, data }
);
}
catch(e){
console.log(
</span><span class="token string">Couldn’t update the widget </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>tag<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">
,
e
);
}
return;
}
This function does the following:
self.widgets.updateByTag()
method to push the template and data to the widget service to update any widget instances connected to the widget’s tag
.As I mentioned, I also have code in place to take advantage of Periodic Background Sync if/when it’s available and the browser allows my site to do it:
asyncfunctionregisterPeriodicSync(widget)
{
let tag = widget.definition.tag;
if(“update”in widget.definition ){
registration.periodicSync.getTags()
.then(tags=>{
// only one registration per tag
if(! tags.includes( tag )){
periodicSync.register( tag,{
minInterval: widget.definition.update
});
}
});
}
return;
}
This function also receives the widget details and:
definition
(from the Manifest) includes an update
member. If it has one, it…tag
value and a minimum interval equal to the update
requested.The update
member, as you may recall, is the frequency (in seconds) you’d ideally like the widget to be updated. In reality, you’re at the mercy of the browser as to when (or even if) your sync will run, but that’s totally cool as there are other ways to update widgets as well.1
When a user uninstalls a widget, your Service Worker will receive a “widgetuninstall” event. Much like the “widgetinstall” event, the argument contains details about that widget which you can use to clean up after yourself:
self.addEventListener(“widgetuninstall”,event=>{
console.log(</span><span class="token string">Uninstalling </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>event<span class="token punctuation">.</span>widget<span class="token punctuation">.</span>tag<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">
);
event.waitUntil(
uninstallWidget( event.widget )
);
});
Your application may have different cleanup needs, but this is a great time to clean up any unneeded Periodic Sync registrations. Just be sure to check the length of the widget’s instances
array (widget.instances
) to make sure you’re dealing with the last instance of a given widget before you unregister the sync:
asyncfunctionuninstallWidget(widget){
if( widget.instances.length ===1
&&“update”in widget.definition ){
await self.registration.periodicSync
.unregister( widget.definition.tag );
}
return;
}
Widget platforms may periodically freeze your widget(s) to save resources. For example, they may do this when widgets are not visible. To keep your widgets up to date, they will periodically issue a “widgetresume” event. If you’ve modeled your approach on the one I’ve outlined above, you can route this event right through to your updateWidget()
function:
self.addEventListener(“widgetresume”,event=>{
console.log(</span><span class="token string">Resuming </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>event<span class="token punctuation">.</span>widget<span class="token punctuation">.</span>tag<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">
);
event.waitUntil(
updateWidget( event.widget )
);
});
While I don’t want to get too into the weeds here, I do want to mention that widgets can have predefined user actions as well. These actions result in “widget click” events being sent back to the Service Worker so you can respond to them:
self.addEventListener(“widgetclick”,event=>{
const widget = event.widget;
const action = event.action;
switch( action ){
// Custom Actions
case“refresh”:
event.waitUntil(
updateWidget( widget )
);
break;
}
});
For a great example of how a widget can integrate actions, you should check out the demo PWAmp project. Their Service Worker widget code is worth a read.
With all of these pieces in place, I was excited to see my site showing up in the Widget Dashboard in Windows 11.
You can view the full source code on GitHub:
I’m quite hopeful this will be the first of many places PWA-driven widgets will appear. If you’s like to see them supported elsewhere, be sure to tell your browser and OS vendor(s) of choice. The more they hear from their user base that this feature is needed, the more likely we are to see it get implemented in more places.
In wiring this all up, I ran into a few current bugs I wanted to flag so you can avoid them:
icons
member won’t accept SVG images. This should eventually be fixed, but it was keeping my widgets from appearing as installable.screenshots
members can’t be incredibly large. I’m told you should provide square screenshots no larger than 500px ×500px.Have you checked out Server Events? ↩︎
I briefly presented on a bunch of the different efforts I’m involved with at the W3C’s TPAC yesterday. Here’s a rundown of what I discussed as well as links you can follow if you want to get involved.
Many app catalogs enable developers to provide links to things like their app’s Privacy Policy and Terms of Use. There is no way, however, to semantically indicate this information in your markup; the closest you can get is link[rel="license"]
, but that’s limited in scope. Rather than propose including more link
or meta
elements (which would need to be included on every page of your site), we think it makes more sense to roll this information into the Manifest as part of a policies
object.
My proposal enables developers to provide URLs for an app’s accessibility statement, content license, and policies governing privacy, security, support, and usage. Developers would be free to include whichever policies their app has and they can be URLs within the app itself or link off to other websites (such as the publisher’s). Here’s what that could look like:
“policies”:{
“accessibility”:“/accessibility”,
“usage”:“/terms”,
“privacy”:“http://publisher.tld/privacy”
}
So far, this seems to meet the needs of several of the app catalogs I’ve spoken with. I don’t know that browsers will make use of this information, but I could see it being useful in the context of an installed PWA as well as in app listings within the browser UI too.
In a related effort, I’m pushing for the Microsoft Store to support accessibility statements, which are helpful for explaining application functionality and providing details of any known accessibility limitations within an app.
When you’re considering installing any app, you want to know you can trust who’s behind it. Phishing sites are everywhere though, so we have to believe that phishing apps are here too (or not too far off). You want to be able to trust that an app purporting to be from your bank actually is from your bank. With such high stakes, opening up the possibility for an app to identify its publisher is quite dangerous, which is why this proposal has taken a while to germinate.
From the developer side of things, my proposed implementation is relatively straightforward. First, the app identifies its publisher
in the Manifest:
“publisher”:{
“name”:“Organization Name”,
“url”:“https://organization.tld/”
}
The problem is that I could put anything in there, declaring my publisher to be any company or individual. In order to substantiate my app’s claim, the listed publisher needs to claim the app too. The publisher could do this by enumerating the apps it owns in a text file located at https://organization.tld/.well-known/published-web-apps
:
https://origin1.tld/manifest.json
https://origin2.tld/static/manifest.json
Instead of using a text file, it may be possible to piggyback on the proposed web-app-origin-association
file, but I’m concerned with overloading web-app-origin-association
and confusing developers who are using it for URL handling.
Regardless of the mechanism, even with this bi-directional attestation, however, the system still has a gaping security hole: the publisher["name"]
string can’t be validated this way. It would be relatively easy for an attacker to host 2 websites, one of which looks like your bank’s app and the other which merely hosts the published-web-apps
file in order to support the bogus app’s claim of being published by your bank.
This is where the proposed processing algorithm adds an opaque validation step. Implementors are encouraged to verify that the publisher’s domain actually matches the claimed name. There are a number of services that offer name-to-url (or vice versa) lookup, but an implementor could even have a manual process in place to verify the provided information (which app catalogs do as a regular part of their business).
In the end, the publisher value will either be a validated name and URL pair or, if validation fails, the web app’s URL host string (e.g., “twitter.com”), which many implementors already use as the publisher name.
Though many folks these days focus their efforts on running as much code as possible on the client, there is a lot of value in making significant changes to your app on the server, before responding to a request. We’ve heard a number of use cases from folks building PWAs and have put together three proposals to help address them:
Sec-CH-App-Installed
header would give a server information about whether or not a user agent is representing the app in an “installed” state. The value would be a boolean true
or false
. We are discussing exposing this information via a DOM API as well.display
mode. We already have a way of detecting the display
mode of a web app using matchMedia()
and CSS’s display-mode
media feature, but that is all client side code. The Sec-CH-Display-Mode
header would provide the display
value (taking into account display_override
) on the server side as well.Referer
header to indicate installation source. I was working with a partner that wanted to beta their PWA to only people who’d installed their PWA from a specific app catalog. Even if they had supplied a separate Manifest with a unique start_url
to that catalog, that would not have allowed them to limit their app’s distribution if the URL was shared. And so we proposed that the browser provide a catalog-associated Referer
header when the PWA is first launched (and in the absence of a true referral). This feature has shipped in Edge for PWAs installed via the Microsoft Store and we are encouraging other app catalogs and browsers to consider the same approach.It’s worth noting that all of these features can be used for analytics and (potentially) to violate someone’s privacy. That is why we’ve suggested they sit under the umbrella of the Privacy Sandbox.
Shortly after OSes began rolling out support for color schemes (e.g., “light” and “dark” modes), I saw a need for web apps to be able to adapt to these user preferences. In CSS we have the prefers-color-scheme
Media Query, but the Manifest had no way of accounting for these sorts of changes. My initial proposal centered around adding support for Media Queries in the ImageResource
spec. Short of having CSS-driven, adaptive SVG web app icons (which I’m also working on), being able to associate an ImageResource
bitmap with a Media Query seems like the way to go.
Here’s what that might look like in the context of the icons
member:
“icons”:[
{
“src”:“/icons/play-later.png”,
“type”:“image/png”,
“size”:“512x512”,
“media”:“(prefers-color-scheme: light)”
},
{
“src”:“/icons/play-later-reverse.png”,
“type”:“image/png”,
“size”:“512x512”,
“media”:“(prefers-color-scheme: dark)”
}
]
This approach would be applicable wherever ImageResource
gets used, which includes app icons, shortcut icons, and screenshots within the context of the Manifest as well as the Notifications and Payment Request APIs.
Beyond images, however, as Jen Simmons pointed out, we need the ability to adapt colors within the Manifest as well. It’s early days in discussing this right now, but we’re thinking about adding a generic means of redefining Manifest keys, based on a particular context. It is going to be a challenge to figure this out because we don’t want to force Manifest authors to redefine an entire nested structure (such as a shortcut) just to swap out a single part of that object. But more on that in a moment.
Here is a simple example of how this might look:
“user_preferences”:[{
“context”:“(prefers-color-scheme: dark)”,
“redefine”:{
“theme_color”:“#bdbdbd”,
“background_color”:“#000000”
}
}],
Ideally developers would not use this mechanism for redefining their icons, using the ImageResource
option instead, but it does open the door to complexity.
Speaking of complexity, have you ever tried to build a web app that is translated into multiple languages? It’s an incredibly challenging task and the Manifest does not make it any easier. At present, the guidance for creating multi-lingual Manifests is to use a separate file for each language or make it a dynamic file that can be adjusted on the server side based on path or query string adjustments.
I’m currently working on a proposal to address this within a single document by introducing a translations
member. My proposal offers two approaches, one favoring simpler Manifests, the other more useful for complex Manifests.
For simple Manifests, translations could be embedded directly within the Manifest file as members of the translations object, keyed by language code:
{
“name”:“Good dog”,
“description”:“An app for dogs”,
“lang”:“en”,
“translations”:{
“fr”:{
“name”:“Bon chien”,
“description”:“Une application pour chiens”
}
},
}
For complex Manifests that may need to redefine text in shortcuts, screenshot labels, and the like, I proposed enabling translations to be managed in a separate file, offering alternative strings only for the items that need it. Using this approach, the above example could become:
// manifest.json
{
“name”:“Good dog”,
“description”:“An app for dogs”,
“lang”:“en”,
“translations”:{
“fr”:“manifest.fr.json”
}
}
// manifest.fr.json:
{
“name”:“Bon chien”,
“description”:“Une application pour chiens”
}
One serious challenge (as with the user_preferences
block, discussed above) is how to manage changing individual parts of a complex Manifest component. For example, you would likely need to change the text label of a shortcut, but may not need to change its url
or icon. Thomas Steiner suggested that JSON Path may be a good solution here as it could enable you to do something like
{
“name”:“Good dog”,
“description”:“An app for dogs”,
“lang”:“en”,
“shortcuts”:[
{
“name”:“Pet Me”,
“url”:“/pet-me/”
},
{
“name”:“Feed Me”,
“url”:“/feed-me/”
}
],
“translations”:{
“fr”:{
“name”:“Bon chien”,
“description”:“Une application pour chiens”,
“$[‘shortcuts’][0][‘name’]”:“Nourrissez-moi”,
“$[‘shortcuts’][1][‘name’]”:“Caressez-moi”
}
},
}
This is still something we are actively working through, but it’s promising for sure.
There is a lot of really interesting work happening in Manifest-land these days. If you’d like to get involved, please follow the links and chime in on the discussions. We’d love to have you!
]]>You can read my nomination statement on the W3C site, so I won’t spend a lot of time rehashing that. What I will do is make a brief case for why I think I would be a valuable member of this particular board.
I’m a web developer whose heart belongs to standards. I may work for Microsoft in Developer Relations, but I started building stuff for the web in 1996 and never stopped. I’ve worked on sites for every kind of business you can imagine—from small mom and pop shops to huge international conglomerates and everything in between. I’ve also held just about every role you can in web projects, from strategist right through to front and back end dev, where the rubber meets the road.
I think this experience, especially when coupled with my current position at Microsoft—which affords me a lot of time to listen to the challenges faced by the web design and development community—will enable me to bring an “in the trenches” perspective to the W3C. Rachel Andrew provides similar guidance as Fronteers’ representative to the W3C and I relish the opportunity to work with her again2 in this capacity. I honestly wish there were more web designers and developers working within the W3C and my goal is to give voice to their concerns and champion their ideas.
I’m a diplomat and a pragmatist. Over the years, I’ve participated in varying capacities for a handful of boards and committees. I’ve chaired small town committees (e.g., the Energy Use Task Force in Hamden, Connecticut), been the co-president of a state political party (Green Party of Connecticut), run homeowners associations, and, of course, led the Web Standards Project, to name but a few. In all of these roles—and in my consulting work—I’ve learned how to manage personalities (and politics), set expectations, and get folks to rally together to achieve common goals.
Anyone who knows me will tell you I am incredibly diplomatic. Perhaps more soo than is warranted sometimes. I believe everyone should be heard, but I’m also unwilling to allow individuals to dominate conversations and drown out other viewpoints. I value diverse opinions and appreciate people who challenge convention. In all interactions, I look for common ground and shared goals. I don’t shy away from uncomfortable conversations and have no problem disagreeing with someone, but I will always do it in a civil and respectful way.
While idealistic—especially when it comes to the web and standards—I’m also a pragmatist. I want to understand problems from multiple angles and use that knowledge to know which battles are worth fighting and when compromise is necessary. And I always seek to build consensus, which is the W3C way.
I’ve got experience in non-profit work. You may not realize it, but the W3C does not actually exist as a legal entity. It’s currently in the process of changing that and becoming a non-profit corporation. The Advisory Board is overseeing that process. When I lived in Connecticut, I helped form a non-profit corporation. I’ve also got experience in grant writing and other non-profit related work. I think I could be a real asset in that regard.
If you can vote in this election and think I’d make a good member of the Advisory Board, please vote for me. If you can’t vote, but know someone who can, please encourage them to read this and consider voting for me. I’d be ever so grateful for your help.
Thank you!
Over the years, I’d hoped to see an organization like the World Organization of Webmasters or the Web Standards Project—both of which I helped steer in some capacity at varying times—would step up an fill this suprising gap, but alas that never happened. And so I am so thankful to see the Fronteers folks (a web design community in the Netherlands) considering formally joining the W3C to fill this role. And I’m even more excited that Rachel Andrew is their first choice to act on our behalf.
I’m hopeful the Fronteers community will vote in favor of this so we can get a few of our own advocates on key committees.
]]>picture
element:<picture>
<sourcetype=“image/webp”srcset=“/i/j/r.webp”>
<imgsrc=“/i/j/r.jpg”alt=“”>
</picture>
This picture
element has two children: one source
referencing a WebP image and an img
element referencing a JPG. This pattern demonstrates how picture
elements must be built in order to validate, but it also reinforces a best practice that uses the fault tolerant nature of HTML parsers to guarantee every user gets something.
In very simplistic terms, here’s what happens when a browser encounters this markup:
picture
element and begins parsing its content to determine how to render it, orpicture
and ignores it, moving inside to look for any elements it might recognize.In practical terms, this markup delivers two potential experiences. Older browsers that haven’t implemented picture
get the JPG image. Newer browsers that have implemented picture
get either the WebP (if they support that format) or the JPG (if they don’t). In this scenario, the img
element can be (and often is) referred to as “fallback content”.
So why have I been thinking about fallback content? Well, I’ve been thinking a lot about media formats and what happens when a user’s browser encounters a video
, audio
, or picture
element that only includes formats it doesn’t support. For instance: A video
element that only offers an AVI source
is highly unlikely to be playable in any browser.1
<video>
<sourcesrc=“my.avi”type=“video/avi”>
</video>
So what happens when most browsers encounter this markup? Nothing. Blank space. If you added a poster, that would be shown, but the user gets no video controls and receives no indication that it should even be video content.
This is the correct behavior, according to the spec:
When no video data is available (the element’s
readyState
attribute is eitherHAVE_NOTHING
, orHAVE_METADATA
but no video data has yet been obtained at all, or the element’sreadyState
attribute is any subsequent value but the media resource does not have a video channel) … Thevideo
element represents its poster frame, if any, or else transparent black with no intrinsic dimensions.
It’s also a pretty crappy user experience if you ask me. Now it’s worth noting that the spec does allow for a better experience, but it doesn’t require it:
User agents that cannot render the video may instead make the element represent a link to an external video playback utility or to the video data itself.
That would offer a much better experience. Of course, since it’s not a requirement for standards-compliance, it’s not guaranteed. In fact, I have yet to encounter a browser that provides that kind of affordance. So if we want our users to receive that kind of a fallback, we need to it using a tool we do control: HTML. Here’s a stab at what that might look like:
<video>
<sourcesrc=“my.avi”type=“video/avi”>
<p>Your browser doesn’t support AVI, but you can
<ahref=“my.avi”download>download this movie</a></p>
</video>
Given how fallbacks work, you might expect a browser to offer up the inner paragraph and download link when it realizes there is no video data it can play. Sadly that’s not the case. They all still display either an empty space or the poster
image.2
Obviously the paragraph fallback would be best, but I have no concept of how difficult that would be to do. So what if browsers did expose our fallback content when the only media types on offer are ones they don’t support?
It seems like this would offer a number of benefits:
That last one is the biggie for me. We want to support as many users as possible, now and well into the future. This might be a way to reliably do that.
I took a Twitter Poll to gauge your thoughts, but that was just a yes/no. I’d love to know your detailed thoughts on this. Is there be any downside to this approach?
Safari is the only possible exception here, and that’s only if QuickTime supports the particular CODEC that was used to create the AVI in the first place. Pretty slim odds. ↩︎
In case you’re interested, I put together a CodePen walking through some of these scenarios. ↩︎
audio
, or picture
element that only includes formats it doesn’t support.]]>video
, audio
, or picture
element that only includes formats it doesn’t support.]]>On the web, it’s tempting to focus our effort around what we know (or think we know) about our customers based on analytics data we’re collecting and our own experience of the web. Similarly, we often get hung up on trying to give every customer the exact same experience of our product. What we need to realize, however, is that analytics and anecdotal knowledge only get you so far. Our customers’ access and experience of the web is highly variable, deeply personal, and, more often than not, completely out of our control.
But take heart, all is not lost. By being flexible in our approach and embracing the unknown, we can create user experiences that are intended to vary from device to device, browser to browser, and network to network.
In this workshop, I explain the ins and outs of crafting rich web experiences that adapt to the capabilities and peculiarities of our customers and their devices, while maintaining your sanity in the process. You’ll leave with:
In Lisa’s analogy, the protons of a digital project are Web standards and Key Performance Indicators (KPIs). She believes that without these key ingredients, projects will often career out of control. Her reasoning? We all need to know how we should be doing things in order to work together well (Web standards) and we need to know what our expectations are for the project to be successful (KPI).
This really resonated with me. Mainly, it resonated because I am a Web standards guy, but I also believe in the importance of standards across the board. I think projects need copywriting standards, design standards, performance standards, coding standards, and many other kinds of standards as well. Standards hold a project together. For real.
That’s why Web standards are so important.
Without standards, the Web was an unruly mass of spaghetti code. I started working on the Web in 1996. I know, I lived it.
We used to deliver separate browser-specific JavaScript and CSS files to different User Agents. Our HTML code had to be 3-4 times as hefty to support all of the various ways browsers had decided to implement the same features. It was horrible and made building anything remotely interesting a truly painful endeavor.
Then browser makers got together to codify HTML into a generally agreed-upon set of elements and attributes that would allow authors like me to write pages that would Just Work™. That work extended into CSS, and so on, and so forth. I can’t thank them enough for doing that.
Tuesday also saw the W3C officially make Pointer Events a recommendation. I’d always liked the idea of Pointer Events because it abstracts the traditional concept of a click into a generic interaction that could be triggered by a mouse, a finger, a pen, an eye movement, or any other interaction method we come up with in the future. Sure, I work for Microsoft now—they proposed this idea—but that isn’t the reason I like the concept. I like it because it doesn’t tie us down to a single way of interacting with Web content that necessitates the creation of new specs when new interaction methods are invented. It’s future friendly and embraces the “continuum of experience” I evangelize incessantly.
When Pointer Events were first proposed, there was a lot of support behind them. Obviously Microsoft was on board, but Mozilla was too. And Google was all about Pointer Events for a while and was already using them when they did an abrupt about-face and decided they were ripping them out of Blink in favor of overhauling Touch Events (which Apple supports and which Pointer Events were intended to supersede).
And so now we have a recommendation from the W3C that browsers implement Pointer Events. Developers want them and it seems Apple doesn’t. And because Apple doesn’t want them, Google doesn’t want them now either. To quote Rick Byers (of Google) in a Pointer Events meeting in late 2014:
No argument that PE is more elegant. If we had a path to universal input that all supported, we would be great with that, but not all browsers will support PE. If we had Apple on board with PE, we’d still be on board too. The equation has shifted for us.
So, effectively, Apple is holding the Web back. Tim Kadlec wrote a great piece discussing the core issue at play here:
Let’s set any opinions about Pointer Events aside. Frankly, I need to do a lot more digging here before I have any sort of strong opinion in one direction or another. There is a bigger issue here. We have a recurring situation where all vendors (save for Apple) show interest in standard, but because Apple does not express that same interest, the standard gets waylaid.
This whole thing has caused quite a kerfuffle in the Web community. Obviously, some people are demonizing Apple (and, by proxy, Google) for holding us back. Others are quick to excuse Apple because of their history of pushing the Web forward (see CSS transitions, animations, etc.).1 Personally, I don’t think anything is ever truly black and white. Every company does some good things and some bad things. To channel Lisa Welshman again, it’s like yin and yang: The light has a little bit of the dark in it, and the dark has a little bit of the light in it.
Generally, I’ve found that Apple tends to do what is best for Apple, without considering how it affects designers, developers, or the Open Web. On this issue however, I just haven’t figured out their angle yet.
Some of their past decisions have offered a clear view into their motivations. Take offline for instance. Apple supports the Application Cache API (as most modern browsers do), but there’s a catch: You can’t store audio files in the cache. That makes it nearly impossible to build a decent game in HTML because you won’t get sound effects if you aren’t connected to the Web. But for Apple it makes perfect sense: They sell games in the App Store.
Their motivations behind other decisions are more murky, however. For example, Safari implements the HTML5 Form Validation API, which means it knows if a field is valid or invalid. The catch? It won’t halt the submission of an invalid form. Every other modern browser acts as you’d expect.2 I don’t get it. I used to think/hope they just had not figured out how they wanted to handle notifying the user of the error, but it’s been like this for about 5 years.
I’ll let Tim jump in again here:
Apple has a very, very strong influence over what standards get adopted and what standards do not. Partly it’s market share, partly it’s developer bias (see, for example, how other vendors eventually felt forced to start supporting the
webkit
prefix due to vendor prefix abuse).
Apple simply does not play well with other vendors when it comes to standardization. The same sort of things we once criticized Microsoft for doing long ago, we give Apple a pass on today. They’re very content to play in their own little sandbox all too often.
He’s channeling a bit of Remy Sharp there (circa 2012):
When are we, as a web development community, going to stop giving Apple a free fucking pass?
They’re consistently lacking in the open discussion in to improving the gateway to the web: the browser. Sure, they landed an impressive mobile browsing experience back when the iPhone launched and it’s a great device, but there’s some serious questioning about whether they’re purposely cock-blocking web development and purposely hindering our advancement as a web industry.
WebGL is in mobile Safari, yet only available if accessed via a WebView object, not the real Safari (which is a WebView anyway…). It was recently discovered that they moved all web data storage (Web Storage, Appcache, etc) in a temporary data store meaning that it can be wiped at any time without warning.
Even the mighty PPK who tells entire browser vendors “fuck you”, doesn’t call Apple out, allowing them to slither on.
Why is it we continue to allow Apple to get away with it? And can this ever change?
As Tim points out, Apple certainly isn’t the only company that plays games when it comes to standards:
The other vendors aren’t exactly perfect either. The Microsoft folks, no doubt reeling from all the negativity aimed at them over the years, have more than once been content to let everyone else duke it out over a standard, only getting involved late when a consensus has been reached. The Blink folks, despite being the best positioned to take a stand, have been happy to play the “Apple won’t do it so I guess we won’t either” card on multiple occasions.
But he’s also quick to highlight the disappointing reality about Apple with respect to the other browser vendors:
It’s easy to reach the Mozilla, Google and Microsoft folks to discuss their thoughts on these emerging standards. That’s a much harder thing to do with the Apple crew.
Apple is very much a black box and their processes are incredibly opaque. Now I’m no hater, I use their products daily.3, but I am also not an apologist. I think relationships are improved with honesty and openness. I honestly wish Apple’s processes—at least when it comes to the Web—were more open.4 Heck, they often don’t even show up to meetings at the W3C. If we knew what they were thinking or why they were doing things, we could at least understand where they were coming from rather than having to speculate about their motivations. Whether we agree or not is irrelevant.
Regardless, we are where we are and I can’t help but wonder one thing: If we stop giving Apple a free pass and continue marching forward without them, will they eventually be forced to scramble to catch up like Microsoft did when IE6 sat on the shelf for so long? I don’t know what the answer is, but I sincerely hope they come around and begin to treat the Web with the respect it deserves before that happens.
When browsers refuse to implement Web standards, we all lose. And we take one step closer to the swirling pit of chaos and spaghetti code we thought we’d put behind us.
All of which were (in true Apple style) developed in secret and then bestowed upon the world in a grand spectacle. ↩︎
Ok, Opera Mini doesn’t, but Opera Mini is also a different sort of browser. It is a proxy browser and the proxy handles all of the back and forth between the browser and the website’s origin server(s). ↩︎
I am composing this post on a Mac… provided by Microsoft. Yes, you read that correctly. ↩︎
I have felt the same way about Microsoft for years because I knew the great work they were doing behind the curtains. Part of the reason I joined them earlier this month was because they have been opening up and I want to encourage and nurture that. ↩︎
Let me back up a bit here. As many of you know, I’ve been working on the Web for a long time and, like many old codgers, lived through the first browser wars and remember not only the unveiling of Internet Explorer 6—which was pretty amazing for its time—but I also worked on the Web for the entire 5 years that browser sat on the shelf.1
By the time Internet Explorer 7 came out in late 2006, there had been a number of advancements on the Web. And there was more competition for user and developer mindshare. Safari popped up shortly after IE6’s launch and was gaining traction on the Mac with its port of Konqueror’s layout engine, KHTML, which they renamed WebKit. Netscape was in it’s death throes, but Firefox arose from the ashes2 and was capturing an ever-growing share of the market with its improved security, extensibility through browser plug-ins, and tabbed browsing. Not only that, but the Mozilla core of Firefox had also been spun into several other browsers that were similarly taking off: Camino, Flock, SeaMonkey, Galeon, and Epiphany. And then, of course, the Opera browser was still going strong on the desktop and growing rapidly in the mobile space.3
When IE7 finally made it out into the world, developers were at peak frustration when it came to dealing with standards-compatibility issues in IE. So it’s no surprise that the messaging focus for IE7 was, at least in terms of the Web designer/developer audience, focused on apologizing for the past and promising that they cared about (and were supporting) interoperable Web standards.
And this was an earnest sentiment, it wasn’t bullshit. I remember Chris Wilson—then Platform Architect of Internet Explorer Platform team—telling me he had personally printed out the entire CSS 2.1 spec and put it on the desk of each developer working on Trident, the browser’s rendering engine.
And IE7, for all of its faults, was an improvement over IE6. A few years later, IE8 was an improvement over that. And, a little later, IE9 gave us a completely reborn Internet Explorer, largely free of the layout and rendering quirks we had earned so much grey hair fighting. And so on. And so on. But all the while, the drumbeat from the IE team was this: Now with more standards support!
And it wasn’t just IE that was making this claim. Other browsers began to tout their support of one particular standard or another that the others didn’t in hopes of getting developers to pay more attention to them.
Some time before the launch of IE8, I remember having a conversation with Chris Wilson over drinks at a conference. We talked at length about the state of Web standards, browsers, and the like. During the course of our chat, he offered up his dream:
I’ll be happy when browsers stop competing on standards support and start competing on chrome.4
It stuck with me because what he was saying made a lot of sense: Standards-compliance should be a given; browsers should be competing on the extra stuff they offer.
Which brings me back to today’s announcement. Standards-compliance wasn’t mentioned5 by Joe Belfiore in his walkthrough of “Project Spartan”. Instead, Joe focused on the value adds in the browser: in-app note taking, a focused reading mode, cross-device synchronization, and Cortana integration.
This is a major milestone for IE in my opinion and it makes me wonder if we’ve finally reached the place that Chris dreamed about all those years ago. I certainly hope so.
I was told that, internally, decision-makers felt the browser was “done” and there would be no more advancements on the Web that would require a new browser. ↩︎
An apt metaphor, Firefox was originally Phoenix, then later Firebird, before eventually becoming Firefox. ↩︎
You may not realize it, but Opera Mobile predated even IE6. And it’s Opera Mini variant touts big numbers too: In April 2014, there were over 267 million Opera mobile browser users (244 million of whom used Opera Mini) and Opera Mini users viewed over 177 billion pages in that same month. (Source) ↩︎
The Chrome browser, from Google, did not exist at this time. By “chrome” he meant the window around a webpage—it toolbars, buttons, menus, and other browser-based functionality. ↩︎
The only phrases that even hint at standards compliance were “modern Web” and “a new rendering engine… that is compatible with how the Web is written today” (starting at around 59:05 in the video). ↩︎
Most of this is stuff you know: ::first-line
, ::first-letter
, and ::selection
. But what magic is this? ::spelling-error
and ::grammar-error
too! What?! Now I know I may be a little too excited about this, but it’s not because spelling and grammar errors are going to solve all of our design problems; I am excited about this because it points to us being granted more control over how more of the internals of browser operations are rendered to our users.
Obviously this is a double edged sword and could be abused to the detriment of the user experience, but it may also lead to someone coming up with clearer conventions for indicating spelling and grammar errors than the ones we’ve been using since the early days of Microsoft Word.
This has been a long time coming. The idea is pretty simple: Within a flow-type element (think p
, div
, etc.) you can apply the wrap-flow
property to children in order to control where content can go in relation to them. Here’s an example from the spec:
In this example, child A is set to wrap-flow: both
, allowing content to flow on both sides of it. Child B is set to wrap-flow: start
meaning content should only be allowed on the starting side of the element (based on text direction, in this case left to right). Child C is set to wrap-flow: end
meaning content can only appear after it in terms of text-direction. And child D is set to wrap-flow: clear
which, as you might expect based on your existing CSS knowledge, ensures no content appears on either side of it.
There is another, more thorough example in the Working Draft (scroll down a little bit), but it uses grid layout as well and is a little more complicated that I want to get into today.
On the flip side of the equation, there is the wrap-through
property which allows other elements to control whether or not they pay attention to the wrap-flow
property of another element. A wrap-through
value of “wrap” adheres to what they are calling the “wrapping context” of the parent element (which basically means it behaves as you’d expect, being a child element). Setting a wrap-through
value of “none” on the child element, however, would make it ignore the parent element’s wrapping context and allow this child’s content to flow through (hence wrap-through
) the excluded element.
It’s a lot to take in and (of course) still subject to change, but I just wanted to bring this concept to your attention as it is pretty cool stuff and is a nice complement to CSS Shapes, which landed early last year.
This one is pretty crazy and I am still tucking into it, but here’s the gist: You could provide a “package” or rolled up collection of files necessary to render your page—think CSS, images, fonts, etc.—in Streamable Package Format which would allow a browser to download them all in one go rather than having to request each resource individually. This could be really useful from a performance standpoint on high-latency connections as it cuts down on the number of round-trips the browser must make to the server before it can render the page. And, since the package can be streamed, we can optimize the contents to provide the most important bits first so the browser can go about rendering the document as quickly as possible.
This is a very cool concept with a lot of potential benefit to our users. Of course, as the document acknowledges, it’s also possible to abuse this and adversely affect performance by including a lot of unnecessary content. Kinda like that guy who made his whole single-page app into a Web Component. Just because you can do something doesn’t make it a good idea.
Still, there’s a lot of potential here and I’m excited to see where it goes.
Now the caveat: all of these are Working Drafts, not Technical Recommendations, so they are not final by any means. That said, they do point to a pretty interesting not to distant future.
]]>To address this need, I put together a tool that dissects CSS3 transforms so the transform process would be more clear for my students. I decided to throw it up on Codepen so you could learn from it as well. Why don’t you open that link and I’ll explain how it works. (Obviously, you’ll need to be in a browser that supports transforms for the tool to be useful to you.)
The tool starts you off with a simple three-function transform
that includes both translate
and rotate
functions. If you hit the “Show Me” button, the browser will draw a box and step you through the application of the transform
functions, one by one, animating their application so it’s easy to follow. This makes it pretty clear why an element ends up where it does.
Beside the “Show Me” button is the “Rearrange” button. It does exactly what you’d think: It rearranges the functions into a new, random order. If you press it and then press “Show Me” again, you can see if (and why) the element ends up in a different place. The original box remains ghosted out so you can see any differences.
Here’s a video of me playing around with the default transform
stack:
Have a play and let me know your thoughts. I hope you find this tool as helpful as I (and my students) have.
]]>As a programmer, I love the idea of being able to abstract reusable bits like colors, border widths, font sizes, and the like to obviously named variables. It’s a far more DRY approach and makes maintenance far easier.
Before I made the leap to using a CSS preprocessor, I was convinced we needed CSS variables, but I always wondered how we might make it possible without breaking one of the fundamental design principles of CSS: Forward and backward compatibility. Take a look at this example (which is based on the working draft spec) and I think you’ll spot the problem:
:root{
–foreground-color: #333;
–background-color: #fff;
}
body{
background:var(–background-color);
color:var(–foreground-color);
}
For a browser that understands CSS variables, the interpreted stylesheet would look like this:
body{
background: #fff;
color: #333;
}
But any browser that doesn’t understand the variables would never get the color values because browsers follow the rules of fault tolerance in CSS and ignore anything they don’t understand. The introduction of variables to CSS would effectively build a wall between older browsers and new ones. (For the record, as of this writing, only Firefox has implemented CSS variables).
In order to serve the broadest spectrum of devices, we’d have to provide a fallback like this:
:root{
–foreground-color: #333;
–background-color: #fff;
}
body{
background: #fff;
background:var(–background-color);
color: #333;
color:var(–foreground-color);
}
But that kinda defeats the whole purpose, right?
Preprocessors already give us this access to variables today (along with nesting, mixins, and programmatic structures like conditionals, loops, etc.). Here’s a SASS example:
$foreground-color: #333;
$background-color: #fff;
body {
background:$background-color;
color:$foreground-color;
}
The big difference here is that this document is a source file, it is not what is sent to the browser. This file is compiled by the preprocessor into actual CSS, which is what we send to the browser and is exactly what we wanted in the first place:
body{
background: #fff;
color: #333;
}
And it works on every browser that supports CSS, all the way back to IE 3.
With a preprocessor like SASS, Less, or Stylus, I get all of the maintainability benefits without sacrificing browser support. It’s a no-brainer. But even if that were not true, there’s another issue to consider: If I push CSS variables to browsers, they have to parse the CSS and substitute the variables before they can apply the styles.
Now I’m sure browser makers can find ways to optimize this process, but it’s bound to affect the rendering time. And not in a positive way. I don’t even want to think about how bad it would be on a mobile chipset, especially on a low-end device.
Honestly, I love using variables… in the source files I use with a preprocessor. Given the potential loss of browser support, the pointless fallbacks I’d have to use if I wanted to continue supporting older browsers, the existence of numerous preprocessor options that solve the abstraction problem in a backward- and forward-compatible way, and the fact that CSS variables would make browsers have to work even harder to achieve the desired result, I’m not convinced we need them.
CSS variables are a bad idea.
]]>