flow|state

Update

For the past two months or so, I’ve left off from my weekly blogging habit here to focus on some behind-the-scenes aspect of QuickUI. I post about those updates on the separate QuickUI blog. That blog is more technically-oriented, but I though it was worth sharing a roundup of those posts here:

• I’ve made a number of improvements to the QuickUI runtime, including a significant version update. One interesting new feature is support for creating UI in CoffeeScript (in addition to plain JavaScript).
• A developer asked for sample application code showing how to use QuickUI as the “View” in an application with an MVC (Model-View-Controller) architecture. That’s a great idea, and to date I haven’t had such a sample I could offer. Cozi’s Meal Planner is actually a Model-View-Presenter that uses QuickUI for the View, but the source for that application is proprietary. It’ll be useful to have an interesting MVC/MVP sample application that shows off how to use QuickUI; I’ll post back here when I have something worth looking at. Thanks for the suggestion, Chris!
• I continue to be interested in making sure the emerging Web Components spec is well-suited to the scenarios routinely faced by UI designers and developers, and have articulated a vision for how QuickUI and Web Components could co-evolve. This has included some time analyzing the QuickUI Catalog controls in light of the Web Components spec. On that note, I’m looking forward to a meeting with the spec’s author, Dimitri Glazkov, later this week.
• A designer friend suggested creating a new QuickUI screencast. The few QuickUI screencasts I’ve done in the past are now out-of-date, and my ideas about how to explain the value of component-based UI development have evolved, so it’s a good time for a new one.
• Along those lines, I’ve invested some time improving the framework documentation, including an overview of how QuickUI controls render themselves.
• I also continue to improve the QuickUI Catalog controls, although at a slower pace. The above work on the fundamentals and explaining them is taking precedence for the time being.

Thanks to those who have shared suggestions with me — they’re very helpful. If you take a look at any of the above and have feedback, please let me know.

From MacPaint to FiftyThree's Paper: Someday all our apps will be this great

Imagine, for a moment, that you’re living way back in the early 1980s, maybe 1984. You have access to a computer, and on that computer, you use a top-end DOS app like Lotus 1-2-3:

 

 

Then, one day, you see a marketing campaign for a new computer. Your eye catches on this image:

 

 

Your mind is completely blown. The user interface for this application, which you learn is called MacPaint, seems utterly unlike any application you’ve ever seen. In fact, the entire premise of this image speaks to a proposition that’s never before even occurred to you: a computer can be used to create art.

• • •

Looking back, it’s hard to convey now how stunning both this image and that proposition were at the time. When the original Macintosh was released, this was probably the first vaguely art-like computer-rendered illustration most people had ever seen. Before that moment, when (or if) the average person thought about a computer, they considered it a tool for crunching numbers or typing documents.

In retrospect, this Japanese woodcut was probably the most sophisticated illustration most people ever saw on an original Macintosh. As groundbreaking as the application was, it was simply impossible for the average user, even a fairly artistic person, to create something of this quality with a mouse. Drawing with a mouse in MacPaint was said to be, “like drawing with a bar of soap”. If you tried to create something like the above yourself, the results were laughable. You could indeed create interesting works in MacPaint, but only by relying on text, lines, polygons, and those paint bucket textures along the bottom. That is, you got the most interesting results with tools that were well-suited to software implementation and which produced effects you couldn’t easily achieve on paper.

The designer behind this image, Susan Kare, discussed it in an interview:

With the Japanese lady, [Apple developer] Bill Atkinson was experimenting with scanning, and Steve [Jobs] brought in an actual woodcut that he had bought: it was big and colorful, and that was one of the first things that we scanned. And I took the scan, which was kind of rough, and refined it to make the final illustration. It looks so crude now — in terms of scanning technology — but it seemed amazing at the time that you could get a “real” image into your computer.

The fact that this image started from a scan was both a surprise and something of a disappointment. Ah, no wonder we never saw illustrations like this — fundamentally, this was marketing! Not to detract from the groundbreaking impact of this work, but this image was clearly meant to suggest to users that they could create art freehand, using only the tools in MacPaint.

Nevertheless, MacPaint represented a watershed in application user interfaces that had broad impact far beyond its users or market. When such an event occurs, it’s possible to look at the app and say something remarkable: Someday all our apps will be this great.

The only reason the MacPaint woodblock image is no longer jaw-dropping to us today is because, within a relatively short time, nearly every application acquired a user interface that in many ways looked and worked as as well as (or, eventually, better than) the interface in MacPaint. Apps simply had to improve to stay competitive, and users everywhere reaped the benefits.

• • •

Such a moment has now happened again — or at any rate, it has now happened again to me. The moment came when I saw a post on Beautiful Pixels about Paper, an app by a company called FiftyThree:

 

 

Paper is beautiful, and I find it a joy to use. Like MacPaint before it, I think Paper represents a new watershed in user interfaces.

Earlier I’d tried Adobe Ideas, a vaguely similar sketch pad app. It’s a fairly typical touch-enabled iPad application, and follows many (but not all) iPad conventions. Judging by app store reviews of Adobe Ideas, some of its users love it, and find it very useful. I myself was underwhelmed. Adobe Ideas feels utilitarian, like a dead thing. Using it to create a sketch feels like work. After a few attempts, I stopped using Adobe Ideas.

Paper, in contrast, feels like something tangible and alive. It’s delightfully fun. Since I installed Paper, I look forward to using it every day. Paper’s interface is beautiful at every level. Zooming out from a drawing (above) shows a sketchbook (below left) containing multiple drawings, and zooming out further shows your collection of sketchbooks (below right):

 

 

 

A stunning amount of detail has gone into every aspect of Paper’s design. Just a sampling of the tiny details I’ve noticed:

• The “paper” background of a sketch isn’t pure white; it’s very slightly off-white. This lets the pure white ink and paint appear extra bright.
• If you’re in the middle of drawing and drag your finger (or stylus) over the tool palette, the palette automatically drops out of the way so you can continue your line into the space the palette had just been occupying.
• You can paint with white watercolor to lighten things. While Paper carefully models the physics of real inks and paints, in various places it breaks with those limitations to let you do things which useful but not possible in the real world. This seems to be done judiciously; the drawing tools still feel very much like their physical analogues.
• The tools respond to the speed of your movement — but not always the same way. The pen gets thinner the faster you go, which makes physical sense, but the calligraphy pen gets thicker when you go fast. This is another case in which the physical metaphor has been judiciously broken. I’m not sure of the precise rationale behind these differences, but the result feels right.
• As you flip through a sketchbook, not only do the pages animate in 3D, their shadows do as well. Paper is built on the OpenGL 3D library, but it probably still was a lot of work to get these effects to look this good and this smooth.

Surprisingly, Paper actually delivers on the original MacPaint premise: you can create beautiful art. I’m no artist, but I was able to quickly sketch the still life with fruit shown above, and the cat in the smaller image. It turns out you can add watercolors to pretty much any pen or pencil drawing in Paper and get something that looks pretty good. My children think so too — yesterday evening I had to read the Kindle edition of Angelmaker on my phone because I couldn’t pry the iPad away from my four year-old.

(Aside: Paper is free, but you’ll have to pay to get the watercolors. You should just bite the bullet and buy all the tools — you will in the end, anyway. I think Paper’s pricing model is as clever as their interface design.)

As amazing as the artistic results are, I don’t think they represent Paper’s greatest accomplishment. At the highest level, I think the best thing Paper has really done is let you feel like an artist. I haven’t regularly sketched anything since Drawing 101 in college, and now I find I’ve bought an iPad stylus so I can do more with Paper.

FiftyThree carries this message throughout out the Paper app, as well as through their site and brand. Everything about this product is designed to lead you to believe, “I am the kind of cool latter-day renaissance person who carries around a Moleskine notebook because my free aesthetic soul may encounter a beautiful scene I want to render as art. I am that awesome.” This is, in fact, the very image in the Paper promotional video: a guy wandering around New York City sketching stuff. The video is shot from first-person perspective. That guy is you.

I think the term “user experience design” is often overblown puffery — when I get to observe someone working through an app design problem, they’re usually focused on the feature set and interface. I rarely witness someone actually thinking directly about the experience their user will have. But with Paper, I think “experience design” is an apt term for what they’ve done. Maybe even that term sells it short. It could be called something like “self-conception design”.

But, wait! Here’s the best part. Someday all our apps will be this great.

Think about that. In the not-too-distant future, every bit of software you currently use (and maybe swear at) — an online store, the Settings area for your latest device, a random tool from an IT department, the production app you spend your workday in — all those things will someday be as beautiful to look at and joyful to use as this Paper app.

And those apps will make you feel great. When send a message, you will feel like a great communicator (or socialite). When you follow a treasure map to an out-of-the-way restaurant in a new town, you will feel like a great explorer. When you follow a recipe, you will feel like a great chef. And when you create a bit of software, you will feel like a great designer.

Make web menu bars more usable: open a menu on hover only if another menu is already open

The history of user interface design isn’t terribly long, but it’s long enough that designers who ignore it do so at their users’ peril. In the transition from clients apps to the web, a lot of UI history has been forgotten, ignored, or reluctantly set aside because its lessons were too expensive (if not impossible) to preserve in early browsers.

For example, it’s hard to find a web application with a menu bar as usable as the standard system menu bars in OS/X and Windows. Consider the basic tasks of opening and closing a menu in a menu bar. Last week’s post on popups listed a number of ways in which a user can cancel a menu: clicking outside of it (while not accidentally triggering whatever one clicks on), pressing Escape, resizing the window, scrolling, or moving the focus to a different window. Web implementations often overlook these aspects of closing a menu.

If we now turn our attention to the task of opening a menu, we find most web apps give this basic act similarly blunt treatment. The choices you’ll typically see in web menus are one of these:

• Menus open when the user clicks on a menu title. This is straightforward for a single menu, but problematic in a menu bar with multiple menus. Users need to scan a set of menus if they’re exploring their options, or if they’re hunting for a particular command. In these situations, having to click on each menu in turn feels clunky. And if the menu developer has done the fundamentally right thing in absorbing mouse clicks outside the menu (so the user doesn't accidentally trigger something when canceling the menu), the user must click twice to open up the next menu.
• Menus open as soon as the user hovers over a menu title. This feels responsive, and lets the user quickly scan a set of menus. On the downside, it can be incredibly distracting to have menus pop open when they’re unwanted. Consider a user who clicks in a text field, and then has to move the cursor away from the text field because the cursor doesn’t automatically disappear when they start typing. (Another bit of UI history that’s been forgotten!) Knocking the mouse out of the way, the user happens to end up parking the cursor over the menu bar, and now a completely unwanted, giant mega menu pops up, covering up their work surface. (That menu article suggests using careful timing to avoid irritating the user, but to me that seems like a band-aid on what’s fundamentally the wrong solution.) Open-on-hover does offer the ability to have a click on the menu title perform navigation, but as discussed in Why Hover Menus Do Users More Harm Than Good, users may not discover that they can click on the title like a link — if hovering into the title popped it up, then the user can easily conclude that the menu has already performed the only job it’s there for.

The odd thing is that a completely smooth way to finesse the problems of both these methods is right in front of the designer and developer, in the very same OS/X and Windows client applications they are likely using to design or code one of these problematic approaches.

Key attributes of menu riffing behavior

For ages both OS/X and Windows have used the following menu behavior:

1. When no menu is open, hovering over a menu title can provide hover feedback on the title (Windows does this), but does not pop up the menu.
2. Clicking a menu opens it. This required click prevents accidental menu invocation.
3. Once a menu is open, hovering into the title of another menu closes the previous menu and implicitly opens the new one. This lets the user quickly riff through a menu bar’s menus to familiarize themselves with their contents or to find a specific command.
   
   [UPDATE: A commenter correctly points out that client OSes actually open menus immediately on mouse down, instead of waiting for mouse up. This makes it possible to riff through menus with the mouse down. If I recall, Mac OS menus originally only worked on mouse down; letting go of the mouse while over a menu title closed the menu. Windows, in contrast, would keep the menu open even after the user released the mouse button, which was easier to use. The user didn't have to hold the mouse down throughout the whole menu exploration and command selection operation. This approach was eventually adopted by the Mac OS. But both Windows and OS/X still support mouse down opening and riffing of menus.]

To me, this resolution seems about perfect, and I wish all web app menus worked this way. In contrast, how often have you used one of the clunky always-click-to-open or twitchy open-on-hover web menu implementations and said to yourself, “I wish all my OS/X (or Windows) apps worked this way!”?

To be fair, simply knowing the UI history (or being very observant) isn’t enough — there’s still the question of cost. One could argue that Apple and Microsoft have greater control over the environment than a web site within the constraints of the browser, which is true, but I think that explanation falls short. The fundamental problem seems to be the economics of homegrown UI: for most companies, it’s hard to justify the return on investment to get these details right in order to make a really usable menu bar. (Which, if they get it right, their users won’t even notice.) Apple and Microsoft can each build a perfect menu bar once that many developers can benefit from, so it’s worth their taking the time to get it right.

Google Docs is one web app that has taken the time to sweat the details. Their document editing suite carefully follows the same menu riffing behavior described above: you open the first menu with a click, and subsequent menus with hover:

 

I’m not sure if Google acquired this finely-tuned menu through Writely or one of the other predecessors to Google Docs, or if they’ve more recently decided that a good way to displace with Microsoft Office is with great usability at a much cheaper price. Either way, it’s details like this that make Google Docs feels like such a reasonable replacement for a desktop application suite. (Thought not perfect yet: Google Docs gets the menu open behavior right, but gets points off for menu closing behavior because they don’t absorb background mouse clicks. And, as referenced above, it doesn’t hide the mouse when you start to type, the way most client text editors or word processors do.)

MenuBar control

I’ve added a MenuBar control to the QuickUI Catalog, along with the usual companions of Menu, MenuItem, and MenuSeparator classes. A Menu can be used on its own, or as part of a MenuBar. When placed inside a MenuBar, the menus will exhibit the riffing behavior described above.

I like the way Google’s visual style puts both the menu title and an open menu on the same seamless surface to visually connect the two regions, so I’ve used that style for a Menu’s generic appearance (the one you get if you don’t want to do any of your own styling).

Although the MenuItem and MenuSeparator classes assume a traditional vertically-oriented list of commands, use of those classes isn’t required, and the Menu class could just as easily be used to present commands in multiple columns or any other arrangement.

Implementation notes

The tricky bit here was making the entire MenuBar and its menus accessible to the mouse, while simultaneously absorbing any background mouse click outside the menu bar or its menus. By default, an individual Menu control supplies its own Overlay so that a Menu can be used on its own or in some other menu bar-like UI construct. The problem is that an Overlay behind a single Menu control will prevent the user from hovering into other menus in the menu bar. So the MenuBar creates its own Overlay control, and turns off the Overlays of the individual Menu controls. The result is the entire menu bar and its menus sit above a shared overlay. The user can hover from one menu to the next, and any clicks on the background overlay are absorbed and cancel the currently-opened menu.

As always, it’s my hope that delivering this behavior in an open, reusable component can eventually change the economics of web usability so that anyone can benefit from the UI design history baked into a component — whether they realize that history is there or not.

There must be 50 ways to close a popup: menus, dropdowns, tooltips, palettes, dialogs, and more

Apps often need to pop up something over the main UI; common examples would be menus and dialogs. Unfortunately, while apps need popups, documents don’t, and until recently HTML was relentlessly document-focused. It’s frustratingly difficult to do a popup well in a contemporary web app, and so it’s not surprising to see so many apps do them poorly or inconsistently.

As a case in point, consider the ways a user might want to dismiss a UI element which has popped up over the main UI. Depending on the specific form of popup, there are a surprisingly large number of methods the popup might support for leaving it:

1. Click outside the popup. This is the most common means to dismiss a lightweight popup like a menu. The user is saying, “I see you, popup, but don’t want to interact with you; let me get back to the main UI.” When the user clicks on the main UI in the background, a key question arises: what happens with that click? This isn’t an easy question to answer; see below.
2. Click inside it. Perhaps the user has hovered into an element that’s popped up a tooltip, and maybe the tooltip’s under the mouse. If the tooltip is nothing but static content, the user can generally click anywhere within the popup to dismiss it.
3. Make a selection. This is a special case of the above point. If the user’s dropped down a combo box and has clicked in an item in the resulting list, they’re not only making a selection, they’re also saying they’re done with the dropdown.
4. Click a button that explicitly indicates completion. Another special case of point #3. A classic example would be an OK button in a modal dialog, which is essentially a heavyweight form of popup.
5. Click a close box. A modeless dialog or persistent palette window often relies on a small “×” icon in the upper-right corner as the primary means to dismiss it.
6. Press Esc. Popups of many flavors can be dismissed by pressing the Escape key.
7. Wait. A tooltip or transient message may go away all on its own after giving the user time to read its contents.
8. Hover into another control that produces a popup. The classic example here is menu riffing in Windows or OS/X menu bar. The user must click a menu to open it, but once that first menu is opened, the user can open the next menu simply by hovering into it. (This aspect of menus is worth a closer look in a subsequent blog post.)
9. Move the focus to another window. Most forms of pop ups are temporary enough that the user doesn’t expect them to stick around. If the user opens a right-click context menu in Google Docs, and then switches to work in a different window, they don’t expect to come back to Google Docs later and find the context menu still open.
10. Press the ALT key. On Windows, the ALT key or (considerably more obscurely) Shift+F10 are used as the keyboard shortcuts to activate the menu bar (or, in some cases, the selection’s context menu). If the user already has a menu or other popup open, this generally dismisses the popup before activating the menu bar.
11. Scroll the page with the mouse wheel. Some apps handle this, some don’t. But if a tooltip or context menu was invoked from something that’s being scrolled out of view, there’s probably no reason for the popup to remain open.
    
    [… Are there other ways? There are a wide range of other user actions that could dismiss a popup, but the others I can think of close the popup as a side effect of a click outside the popup or a loss of window focus.]

Most web apps that create popups seem to consider only a small fraction of these cases. For example, it’s common to have a web app’s menu stay open even when the Escape key is pressed (point #6 above) or the tab or window has lost focus (#9 above).

Some of the above cases have complexities that get overlooked. Point #1 above — handling a click outside the popup — raises the question of what should happen with that outside click. The choices are: a) absorb the click so that it has no effect other than closing the popup, or b) let the click affect as usual whatever element outside the popup the user clicked on. On the web, the latter choice can be easier to handle, but this raises a significant usability risk: if the user clicks outside a menu, and just so happens to do so by clicking on a link, do they really intend to trigger the link’s normal navigational response?

As an illustration, suppose a Facebook user has dropped down the menu on the right side of their current toolbar, and then they decide to close the menu by clicking outside it:

 

Careful!

That click outside the menu isn’t just going to dismiss the menu—the click is also going to activate the partially obscured “app request” link. If the mouse were just a few pixels lower, the user would end up launching the process to create an ad.

Most OSes and client apps will absorb a click made outside a popup like a menu so that the user doesn’t accidentally trigger an unintended action. Web apps usually don’t absorb the click. It’s hard to know whether this is intentional or not. I think it’s simply a reflection of the fact that absorbing the outside click in a web app takes more effort. I personally think that effort is worth the trouble to avoid usability issues that can arise if, in the course of dismissing a popup, the user ends up accidentally triggering a background UI element. I think that work is even more worthwhile if it can be folded into a shareable component so that most designers and developers don’t have to ever think about this issue.

Related to the concept of a popup is that of an overlay. To help the user see a heavyweight popup like a modal dialog, many web apps display a “lightbox effect” or other visual treatment. This layer sits visually behind the popup but over the main UI in the background. This overlay is really a distinct UI element, albeit one whose existence is probably seldom noticed. The overlay may not even be visible — it may be entirely transparent! But a transparent overlay is precisely the means one would typically use to absorb clicks outside a popup: a click on the overlay prevents the click from reaching a UI element in the background.

The Popup control and its related classes

Over the past week, I’ve overhauled the Popup base class as part of work towards a new Menu control. One of my goals was to create a base class that handled more of the cases above automatically. For example, I wanted a Popup to absorb outside clicks by default so that most designers won’t have to even think about this, while still leaving the option of letting the outside click go through if the designer really wants that behavior. Similarly, I wanted the various Popup subclasses (like Dialog) and related classes to handle their respective situations better so that anyone using them has an edge in producing UI with good usability.

The base Popup class now gives the designer and developer the ability to smoothly handle many of the dismissal cases above: outside click, inside click, loss of window focus, pressing Esc, etc. Special cases like menu bar hover behavior can be addressed in subclasses (like the forthcoming Menu control).

A Popup control will normally create a companion overlay control to absorb outside clicks. This overlay is generally an instance of the Overlay class. By default, the first click on an overlay dismisses the popup and removes the overlay. A subclass called ModalOverlay can be used for modal dialogs that want to absorb all outside clicks (not just the first), so as to force the user to explicitly dismiss the dialog. The generic appearance of the ModalOverlay class includes a basic lightbox effect. A Popup can also be created with no overlay in situations where it’s important to let outside clicks have their normal effect.

A related class called PopupSource is available for the common case where a persistent UI element (a button, say) invokes an attached popup. PopupSource takes care of positioning the popup in relation to the button which invokes the popup. If space allows, the popup is shown below the button and left-aligned, but if this would cause the popup to extend beyond the viewport, the popup may appear above the button or right-aligned as necessary. PopupSource is used as the base class for ComboBox, so a dropdown produced by a combo box may actually drop up if there’s more room above the combo box and not enough below. This is standard behavior on client OSes, but rare in web sites that have created their own combo box-like elements.

Implementation notes

In dealing with popups, one naturally has to dive into the details of how browsers render one element on top of the other. In this study I was aided by an excellent summary of how DOM elements stack. Having read that, it now seems likely to me that any occurrence of the CSS rule, “z-index: 1000”, can also be read as, “z-index: I don’t really know how this works”.

Predictably, creating a general-purpose Popup class that works reasonably well in a wide variety of configurations on all the mainstream browsers entailed a substantial amount of testing and debugging. IE8 was particularly problematic in this regard.

Controls of the Week: HorizontalPanels and VerticalPanels for basic CSS3 flexbox layouts today

It’s really, really common in UI to place a panel on one or both sides of a main content area, on the left and right or on the top and bottom:

 

 

 

As ubiquitous as these layouts are, until recently it wasn’t easy to create them in HTML and CSS alone. You were either forced to hard-code the heights or widths of the panels, which is gross and hard to maintain — measuring the rendered dimension of a UI element is a task best left to the browser. You could write JavaScript to calculate the dimensions at runtime, but that’s a bunch of work many have avoided.

The CSS Flexible Box Layout Module, a.k.a. “flexbox”, is intended to address layouts like the ones above. For a general introduction to flexbox layout, see CSS3 Flexible Box Layout Explained. This feature hasn’t gotten as much use as it could; as shown on When can I use, it’s not supported on the current (as of this writing) versions of Internet Explorer. Moreover, the flexbox spec changed a while back; only Chrome supports the final spec.

To address older browsers, it’s possible to use a polyfill to support new CSS features. In this case, I wanted to create QuickUI controls to serve as a polyfill for flexbox layout. That is, these should take advantage of flexbox on browsers that support it. On older browsers, they should fall back to simpler flexbox-less CSS in cases where that is sufficient, and otherwise fall back to JavaScript-based layout.

Key attributes

The flexbox layout module can handle many layouts and needs beyond the two shown above, but the two above are common enough that they represent a good starting point.

• Each layout has a stretchable main content panel.
• A horizontal layout can have a panel on the left, right, or both. Similarly, a vertical layout can have a panel on the top, bottom, or both.
• The control needs to be able to handle arbitrary content in the panels. If the content changes, the layout should adjust in response.
• Each layout comes in two forms: one with a constrained height (in which the content is generally scrollable) and one with no height constraint (i.e., grows as tall as necessary). In practice, the unconstrained form comes up much more often in the horizontal layout. (In the vertical case, the unconstrained form is really just a stack of divs, so no special layout is necessary. However, controls such as TabSet come in both height-constrained and unconstrained forms, and it’d be nice to be able to position the tabs using a vertical layout in either case. So even the unconstrained vertical layout comes up in some, albeit rare, situations.)

HorizontalPanels and VerticalPanels controls

I’ve posted HorizontalPanels and VerticalPanels controls that address the layouts described above. They can each handle up to one panel on either side of the content area.

As browser implementations come up to snuff, the components can be updated to take advantage of native CSS flexbox support (including, eventually, the new syntax). You can build a UI using these layout components that will work today (as far back as IE 8), knowing that your UI will capitalize on flexbox support as it become more available.

Implementation notes

The HorizontalPanels and VerticalPanels controls derive from a base class called SimpleFlexBox, which sniffs out support for display: box and its variants. In testing, it seemed only WebKit’s flexbox implementation is worth using today. As of this writing, the Mozilla implementation seems too flaky overall to depend upon. And even on WebKit, I hit what looks like a bug preventing the use of automatic scroll bars in a height-constrained flexbox panel with horizontal orientation, which is a pretty common use case. This means HorizontalPanels can’t always use flexbox layout, even on Chrome. And while I’m interested in testing these controls on IE 10, Microsoft has tied the IE 10 preview to the Windows 8 preview, and I’ve already wasted too much of my life fiddling with Windows betas to care about trying Windows 8 before it’s ready. (Weren’t all the tying-IE-to-Windows shenanigans supposed to end with the DOJ consent decree?)

The height-unconstrained cases can be emulated on older browsers using other CSS (i.e., without doing layout in JavaScript), so again there’s no price to pay unless its necessary. If the only way to perform the layout is JavaScript, the control binds to a newly-overhauled pair of events in the QuickUI framework. There’s now a general-purpose layout event to which any control class can bind if it wants to be notified when the control’s dimensions have changed in response to a window resize. There’s a companion sizeChanged event a control can listen to for changes in the dimensions of its children. This is used by the SimpleFlexBox base class, for example, to listen for any changes in the size of controls in its side panels so it can determine whether it needs to adjust the size of the center content area. SimpleFlexBox only binds to these events in the cases where it needs to manually lay things out, so you’re only paying the price of those events when it’s necessary.

I did hit a weird cross-browser issue in IE9: when I view the VerticalPanels demo in IE9 under Large Fonts, the border for the main content area doesn't quite touch the border for the bottom panel. This can happen in IE9 because elements that size to text content can end up with fractional pixel heights. Since IE9 doesn't support flexbox, in the constrained height scenario SimpleFlexBox needs to examine the height of the top and bottom panels so it can adjust the metrics on the main content area. SimpleFlexBox requires on jQuery's height() function to do this, which turns out to always report integral pixel values. Under certain cases, then, it's possible to end up with a sub-pixel gap between the main content area and the panels — and the gap can become visible if the browser or display is scaling things up (as with Large Fonts). IE9 can report fractional heights via window.getComputedStyle(), but it doesn't seem worth this trouble just to support IE9 under various display edge cases. IE8 reports integral heights, and IE10 should support flexbox, leaving only IE9 with this issue. A simple workaround would be to avoid setting interior borders on the main content area if you're also setting them on the panels.

In any event, it’s nice to be able to wrap up a bunch of browser-dependent styling or code into a reusable component that can handle the details so the component user doesn’t have to. And, IMO, I’m not altogether sure that universal flexbox support will actually eliminate all need for controls like HorizontalPanels or VerticalPanels. Use of those controls in your code can arguably make it easier to clearly state your intent. While the CSS flexbox spec is very, um, flexible, the resulting CSS is not particularly easy to read. I preferred the Dock=“Left” syntax of Microsoft’s DockPanel control to the flexbox syntax, and have tried to mirror the former in designing the API for HorizontalPanels and VerticalPanels. Compare: to set the content of the left panel of HorizontalPanels control, you can stuff that content into a property called “left”. To achieve the same result in CSS3, you omit the “box-flex:” property to ensure the panel won’t stretch. I think the former is easier to read and maintain. Even once everyone has a flexbox-capable browser, these controls might still find use as more legible wrappers around the underlying CSS.

Like snapping together a skyscraper: Web components will catalyze a completely new ecosystem for creating UI

In just a few years, the ecosystem in which we create UI will change so dramatically that it will be hard to remember how we did things way back in 2012.

For a sense of perspective, consider a similar change that transpired over a much longer period of time in a different industry: home construction. If you were building a house hundreds of years ago, you might have directly built or overseen most of the elements that went into your house: the framing, the hearth, the chimney, the roof, the windows, doors, you name it. You built nearly everything yourself — there were hardly any components. Depending on where you lived, the only pre-built components you might have used would have been small and simple: glass from a glazier, bricks from a brickmaker, hardware from a blacksmith, and pipes or tiles from a ceramist. Even a glass window would have its surrounding parts — the case or sash, the wooden frame, the sill — measured, cut, and assembled on site and for a specific window. If you hired a craftsman like a carpenter or mason, everything they built for you would have been created on site specifically for your house.

Now build a house in a modern economy. The majority of your home’s elements are components assembled elsewhere by specialists and shipped to your construction site ready for final installation. When you design a house, you now spend a lot of your time looking through catalogs of these components. Most of those components come in standardized dimensions or configurations. Many are quite complex. You can buy an intricate multi-part casement window in a variety of window configurations as a single, complete unit that includes wood, metal, multiple layers of glass, glass treatments, hinges, locks, screens, and other hardware. You can find a similarly dizzying selection of pre-built roof joists, plumbing fixtures, or light sconces, or other components. If you want a component that someone doesn’t already offer for sale, you are either visionary or insane.

 

A tiny handful of configurations for window components (source: Window Express)

 

The componentization of the building industry means you can get a lot more house for a lot less money, and the resulting home can be better suited to your needs. Most of the factory-made components will be of better quality than what any one individual could make themselves on site. (It’s the site-built skylights that leak, not the factory-made ones.) And not only is the resulting building different; the component ecosystem brings about myriad new roles and industries.

Now consider software, where we’ve labored for years hand-crafting every element of the user experience like a medieval builder. The browser or OS gives us a tiny number of simple UI primitives; we must write nearly everything else in the UI by hand. For simple designs that are essentially fancy documents, one can use a visual editor of the Adobe DreamWeaver ilk, but you still have to roll up your sleeves. And any UI that affords any significant degree of interactively is created substantially in code on either the back end or front end. To the extent that UI code is “shared”, most often it’s actually copied and then hacked to fit, rather than implemented with a truly shareable component. If you did static analysis of the UI code for the 100 most popular web apps, I’ll bet you’d find that only a tiny percentage of that UI code is actually shared with another organization.

If only there were some standard for composing and extending web UI components, we’d be able to unleash a UI ecosystem that would transform the UI world as thoroughly as the physical building component ecosystem has changed home construction.

The UI field may actually undergo a bigger transformation, because the software world isn’t subject to the same constraints as the physical world. It is possible to create responsive UI components that change based on the device context, meta-controls that generate UI from more basic controls, adaptable components that change based on the user’s abilities and experience, and components that directly exploit third-party services.

With such tools in hand, it should be possible to create huge, complex interfaces in a fraction of the time it currently takes, and for far less money. You’ll be able to assemble the UI of a significant application very quickly, and get something interesting that in many ways actually works. It will be like snapping together building parts to create a skyscraper.

This transformation is still in the future, but it’s coming. One important step here is Google now taking the lead on a spec for web components that will standardize how components are defined and interact. A good summary can be found in Web Components Explained. (Years ago, Microsoft tried to promulgate a standard for HTML Components, but it never caught on.) While closure on the web component spec is still off in the future — and broad availability is, of course, even further away — this new world is coming.

This can’t happen soon enough. It will finally free us from having to waste such an ungodly amount of time attending to the design, coding, and testing of common user interface patterns, and let us move our attention up the value ladder to focus more on our own products’ domains.

This development will ultimately commoditize some large portion of the industry’s UI output. As with the building industry, commoditization of UI elements will catalyze the creation of new roles in the UX industry: specialists who create components, component integrators, component testing labs, standards groups, and many more people in more organiziations creating better UI because they can start with solid, usable components addressing many of their needs.

I’m excited by what this will mean for the QuickUI control framework. Google’s web component spec will eventually let the browser natively address the lowermost functions which QuickUI must currently perform in JavaScript. This will enable much better performance, better isolation and modularity, and faster adoption. It’s too early to say how QuickUI evolve in this regard, but I want to direct its evolution such that it will transition smoothly to the standard web component foundation when that becomes widely available. Among other things, I’d looking at how to evolve the open QuickUI Catalog of common UI controls so that they can someday be delivered as web components on the standard foundation. The goal is that someone using QuickUI controls today will find their investment preserved and profitable when the component future arrives.

If you’re interested in tracking Google’s work on the topic, they are posting announcements on Google+ on the Web Components page.

Control of the Week: Pinterest-style PackedColumns efficiently fills space with tiles of varying heights

Image-sharing site Pinterest is the current darling of the social media world, and the core of its user experience is its attractively-designed home page:

 

 

This page takes good advantage of available window real estate. As the user makes the window wider, the page re-lays out the columns of image tiles (or “pins”, in the parlance of the site) to take advantage of the extra width:

 

 

The page must accommodate a wide range of tile heights, as the photos have different aspect ratios, and the number of comments per pin can vary. If the page simply laid out the tiles in a strict grid, it would waste a great deal of space. To use the space more efficiently, the page employs a “packed columns” layout.

Key attributes

The packed columns layout algorithm is straightforward:

1. Divide available width by the standard item width to determine how many columns can fit.
2. Make all columns initially empty.
3. For each item in turn, add the item to the column which is currently shortest.

The simplicity of this algorithm is such that it’s been independently recreated multiple times. The algorithm has some nice properties:

• It’s fast.
• As much horizontal space is used as possible (while still showing entire items). If a user gives the site more width, they’re rewarded with more information.
• The arrangement is visually interesting.
• The positions of the first few items are stable.
• At any given page width, the overall heights of the columns will be roughly equivalent. If the user scrolls to the bottom, they won’t find an unbalanced amount of space under any particular column.
• The relative vertical position of any two items is preserved across resize operations. If item A appears above item B at one window size, then item A will always be above (or on the same row) as item B at any other window size. The user doesn’t need to understand this; it just means that if some interesting item is “near the top” before a resize, then after the resize the same interesting item will still be “near the top”.

The last point speaks to another benefit of the algorithm which doesn’t show up in Pinterest, but does show up in other applications: the consistent relative positions of items means you can offer users the ability to specify an order or prioritization for the items that affects (but doesn’t completely determine) where items end up. I used this years ago in the design for a home page for Microsoft Money, a personal finance application whose home page included a user-customizable set of home page modules. A Settings dialog let the user specify the priority of those modules by dragging the modules within a one-dimensional list. While the ultimate two-dimensional position of the modules depended on the window width and the modules’ current heights, the priority of any given module determined how close to the top of the page that module would end up. This limited degree of customization was sufficient to meet many users’ needs without having to create a full-blown customizable layout UI.

PackedColumns

I’ve added a PackedColumns control to the QuickUI Catalog. There’s a link to a demo that simulates the general appearance of Pinterest’s home page. (I initially centered the items in the demo the way Pinterest does, but turned centering off to make it easier to observe the layout behavior.)

Usage: Use PackedColumns to arrange a collection of child elements whose widths are fixed but whose heights vary substantially. If the heights are relatively consistent, users will likely find a traditional grid presentation easier to interpret and use.

Commentary

Given the simplicity of the algorithm, this wasn’t all that hard to code up. I expect it’s not necessarily the actual cost of a layout like this that deters sites from adopting it. Rather, it’s the current need to independently discover or reverse-engineer behavior like this that most inhibits its adoption. As design knowledge gets coded into controls, however, such UI should become more pervasive.

In essence, an ability to easily create and adopt create web components will lead to a commodification of user interface elements. Today Pinterest’s insight and ability to create a packed columns layout may confer a slight competitive edge, but someday commodification will quickly eliminate such edges. This will be true not just for UI elements that can easily be independently created, but for nearly anything. The day after a new site launches with a cool new UI trick, that trick will be copied and packaged up as an openly available and readily adoptable UI control anyone can use.

UI Control of the Week: Repeater control generates multiple copies of UI elements

User interfaces invariably entail a certain degree of repetition; they’re filled with vertical or horizontal sequences of UI elements that behave identically are are styled identically. Sometimes the elements in such a sequence vary only in their label, and sometimes even that doesn’t vary; the controls really are all exactly the same. As an example, if we go back to the first post in this series on UI controls, we find that Apple’s sliding pages with dots control contains a horizontal sequence of little dot buttons. The variant of this control on Apple’s web Store uses blue dots:

Those little dots along the bottom don’t contain any data, and so their DOM representation of each is essentially identical. (The blue selected state comes from a style applied with a class.) Sequences of completely identical UI elements like this are relatively rare in a production UI. During design and development, however, it’s pretty common to want to throw a bunch of placeholder controls into the UI. Early in the design process, a prototype’s toolbar might have buttons labeled, “Button 1”, “Button 2”, “Button 3”, and so on, until the team can work out exactly what commands they want to offer users there.

But, despite the repetition, creating a collection of buttons like that is generally a manual process: the designer or developer must manually create a set of buttons, and carefully give them each a unique, placeholder name. Alternatively, one writes a bit of throwaway script to generate a given number of controls, although that can take a few minutes to work up.

The recent post on placeholder controls pointed out that it can be worthwhile to have a UI control even if it’s only used during the design process; anything that saves time helps. Here, I think it’s interesting to have a control specifically for the task of generating repetitions in a UI. As with the previously-discussed ListBox, this is effectively a higher-order meta-control: a control that creates or manipulates other controls. This can be useful for mocking things up during design. And, per the Apple example above, it might even be useful in production UI.

Repeater

The QuickUI Catalog contains a Repeater control. Given a control class and a number, it will create that many instances of that class. If you create a Repeater and give it a dot button class and a count of 5, you’ll get:

 

 

With that in hand, you can easily bump the count up or down to get whatever number you need. If you want to see what things look like with 20 copies of the dot control, instead of doing a cut-and-paste of your UI code, you can just change the desired count to 20:

 

 

If you give the Repeater some content, each generated copy of the control will end up with that content. Here a Repeater has been told to create 5 instances of a simple button class and set their content to the text, “Button”:

 

 

For a bit of variety, you can also ask the Repeater to append an incrementing integer to the content:

 

 

This is another one of those controls that, now that I have it, I end up using quite a bit. When poking around with a layout idea, it’s great to be able to fill up the UI quickly with a sequence of elements.

Implementation notes

It’s easy enough to create a one-off bit of JavaScript that creates an arbitrary number of controls, but why rewrite that code every time you need it? By investing just a bit of time in creating a reusable component, even that simple bit of code has already been written for you.

The implementation of Repeater has become simpler over time as the QuickUI framework has gotten better at supporting the creation of meta-controls. These controls generally have one or more properties that accept a control class as a value. Creating such a property is easily done in a single line using a Control.property() declaration. A recent update to the QuickUI runtime makes it also possible to pass in arbitrary UI in Control JSON format, so you can use the Repeater control to generate n copies of some brand-new UI fragment containing a mixture of other controls.

As suggested above, a Repeater is incorporated into the implementation of the Catalog’s SlidingPagesWithDots and RotatingPagesWithDots (which adds automatic rotation) controls. Once the number of children (pages) is known, the control can simply pass that number to the Repeater’s count() property to generate the required number of dot buttons.

UI Controls of the Week: Making HTML check boxes and radio buttons work the way they should have in the first place

Here’s the current Sign In UI on a typical e-commerce web site (United Airlines, one of the largest airlines in North America) with a minor but common user interface bug:

 

 

The bug is this: the “Remember me” check box can only be checked by clicking the tiny 13 by 13 pixel square; clicking the text label for the check box has no effect. This minor but common bug appears on many web sites because an HTML <input> check box on its own can’t define a label. The label can only be defined by creating a separate <label> tag. I have no idea who came up with this arrangement, and can only imagine that this was intended to allow flexibility. It does allow, for example, a check box label to be placed above, under, or to the left of, a check box. But this flexibility comes at a cost: many web developers aren’t aware of the need for <label> tags, and so they end up with check boxes with static, unclickable labels. HTML radio buttons suffer from the same issue.

Of course, users have been long trained by client OSes that the text next to a check box or radio button should be clickable. It makes sense, after all, to give the user a large hit area (especially on a touch device). If the site above were to correctly define a check box label, the hit target would be 600% times as large as using the box alone, at no additional cost in screen real estate. Furthermore, the UI would be more accessible to a larger population, including vision-impaired people using screen readers.

The situation is improving, and a quick survey of some highly-trafficked web sites shows that many of them do correctly define labels for check boxes and radio buttons. But even some popular sites do not, or don’t do so consistently. Quantcast estimates the above United Airlines site gets about 1M U.S. visitors a month, and it’s fair to guess that some significant portion of those people are being driven through the faulty Sign In UI above.

The problem persists because here it’s harder to create a correct UI than an incorrect one. For the correct result here, the developer has to:

1. Hear about the need for the <label> tag and learn how it works.
2. Remember to use a <label>.
3. Set an ID on the <input> element.
4. Create the <label> element.
5. Type in the user-visible text.
6. Set the label’s “for” attribute to the input element’s ID.

In contrast, to create this check box the wrong way, the developer just has to:

1. Type in the user-visible text.

A check box created the wrong way looks pretty much like one created the right way, so it can be hard for the team creating the UI to spot the bug. And, of course, when the problem exists in UI that’s generally shown only to new users (like the UI above), team members will rarely be exposed to the bug themselves.

Usability experts can exhort the correct use of <label> tags until they’re blue in the face, but a real fix requires that it be easier to create a correct UI than an incorrect UI. Client OSes have made this easy for years, and I can probably count on one hand the number of times I’ve seen a check box in a client app in which the text was not correctly clickable.

Oh, and one more thing. On the web, it turns out that even if you do things the way you’re told to, your check box or radio button UI may still have a tiny bug. By default WebKit and Mozilla put an unclickable 3px margin around the input element. So even if you use a <label> tag in the recommended fashion, you still end up with a 3 pixel gap (highlighted below in red) between the input element and the label:

 

 

Clicks in this gap have no effect! This is a teeny tiny bug that nevertheless happens to show up in WebKit and Mozilla on nearly every web site. (IE takes care to leave no gap.) This probably means that on any given day thousands of users happen to click in that gap, and are puzzled that nothing has happened before they quickly click again. I noticed that one site, Gmail, carefully works around this very issue by overriding the margins on the check box and label to eliminate the gap. Once again, it seems the platform makes it harder to create a correct UI than an incorrect one.

CheckBox and RadioButton

I’ve added CheckBox and RadioButton controls to the QuickUI Catalog that implicitly associate a label with an input element, and close up the gap described above.

These aren’t particularly fancy or interesting components, but they’re nevertheless simple to use and solve the problem defined above. I wish HTML check boxes and radio buttons had always worked like this.

Implementation notes

Both CheckBox and RadioButton inherit from a LabeledInput base class that creates the automatic link between the label and the input element.

I originally implemented the LabeledInput base class as an inline div containing an input and a label element, and had some JavaScript explicitly link the two elements with a generated ID. But then I noticed something interesting on Gmail’s sign in page: the input element is inside the label element, right before the static text. I’ve never seen this approach documented on pages that describe the use of <label>. Every site seems to document the label appearing in the HTML immediately after the input. But putting the input inside the label seems to work in all the mainstream browsers. The advantage of this approach is that there’s no need to set the “for” attribute on the label; the label automatically binds to the input element it contains.

Taking another hint from Gmail, the LabeledInput class also sets margins so as to leave no gap between the input element and the adjacent text.

Finally, as an extra bonus, the RadioButton control solves an annoyance specific to HTML radio buttons. An HTML developer must manually designate an internal radio button group name for each radio button in the group that should work together (i.e., which should be mutually exclusive). This isn’t hard to do, but it’s still an extra step, and more work than should really be necessary. So, by default, if you don’t explicitly put a RadioButton into a group, it will automatically group itself with any siblings (with the same DOM parent) that are similarly ungrouped.

UI Controls of the Week: Quickly fill up a UI mockup with photos, placeholder text, and ads

When you’re designing a new UI, you often need to experiment with a variety of UI layouts in advance of having content that’s representative of what your UI will eventually display. This is a good thing — you don’t want to be burdened with the task of creating meaningful content when you’re focused on layout and navigation flow. In the exploratory stages of design work, it’s also important for you, or your design’s reviewers, to not get caught up too much in the generation of sample content.

This is why designers have long used Lorem Ipsum placeholder text to fill up a design. It looks like real text (which would not be the case if you simply mashed the keyboard), and you can generate an infinite amount of it to fill up any design you’re working on. Most designers also have a collection of placeholder images or advertisements they can use to fill up a design mockup. One service will dynamically serve up placeholder photos of kittens, although I’d imagine the conspicuous presence of zillions of kittens will be highly distracting for most web site designs.

Although end users never see these sorts of placeholders, they’re nevertheless an essential element in the software development process. I’ve yet to see placeholder components included in a UI library, but it seems eminently reasonable for these placeholders to be packaged up as reusable controls. Anything that cuts down on design time is money in your company’s pocket.

With that in mind, the QuickUI library now has several placeholder controls:

 

LoremIpsum

The LoremIpsum control generates an arbitrary number of paragraphs of Lorem Ipsum text. You can control number the number of sentences per paragraph. By default, the first sentence of the first LoremIpsum control starts with “Lorem ipsum dolor sit amet…”, but you can control that as well.

FlickrInterestingPhoto

The FlickrInterestingPhoto control grabs a photo from Flickr’s Interestingness collection for the previous day. You can pick one of Flickr’s standard image sizes, or you can use CSS to scale the photo to an arbitrary size.

I use Flickr for this control because it’s free, has a good API, has high-quality images, and the images will change each day. It’d be pretty straightforward to adapt the control to another photo service.

AdPlaceholder

Finally, the AdPlaceholder control creates a rectangle the size of any IAB standard ad unit, or you can specify an arbitrary size.

I’ve looked for a server that would serve up meaningful ad images, but haven’t found one. Some sites will give you a small set of ad placeholders, but they’re too boring to be convincing, and the small size of the sample set means you get too much repetition. An ad placeholder service would be quite useful. It would give advertisers free exposure, although the ad server would need to be rigged to not count such impressions as meaningful. All this means that it’s hard to provide a general-purpose ad placeholder control. It would be quite easy, on the other hand, to create an ad placeholder control that worked against a specific ad server and ad account.

 

Using placeholders like these let you quickly fill up a mockup. E.g., the demo for the PersistentPanel control uses all three types to block out a fairly interesting layout on the fly:

 

 

In practice, I’ve discovered that these dynamic placeholder controls deliver a substantial benefit over relying on static content: the random content forces me to cope with layout situations I might not expect or encounter until far later in the development process. Designers have a innate tendency towards perfection, and invariably pick sample content to make a layout look as appealing as possible. For example, a design for a window will typically show a set of content that perfectly fills the window, but as I noted long ago, such a design is probably too good to be true. Your team will end up evaluating a design according to a degree of theoretical perfection that will never be seen in production. By building mockups around dynamic content, you force yourself to recognize and adapt to a more meaningful range of text run lengths, picture aspect ratios, and so on.