I'm writing a CSS book
JavaScript archives Compatibility CSS coaching Thidrekssaga QuirksBlog Donations Politics Mobile About COH
QuirksMode
PPK sitemapcontact

A note on testing methodology

In his recent Feature testing CSS properties entry, Juriy Zaytsev (Kangax) discusses the possibility of detecting CSS support by means of JavaScript. Although he rightly points out that this method has its drawbacks, as far as I’m concerned he doesn’t go far enough.

This sort of testing should not be used at all. Ever. The methodology is plain wrong. Browser compatibility tests are to be done by hand. Any automated system is useless, because it will give false information.

Testing and testing

Before we continue we should make a distinction between performance, network, and compatibility testing. The first two can be automated — in fact, the whole point of performance tests is to measure the time a script needs to execute, which makes an automated set-up almost mandatory. Networking tests, too, can easily be automated (as Steve Souders has done).

Compatibility tests, however, cannot be automated. I’ve thought about this problem for quite a while since starting my mobile compatibility work, but a quick test last Thursday proved conclusively that testing CSS by means of JavaScript is just plain wrong.

CSS tests in JavaScript

In theory, testing CSS compatibility by means of JavaScript is quite elegant. Kangax gives this example:

 if (docEl && (s = docEl.style)) {
      return typeof s.borderRadius == "string"
        || typeof s.MozBorderRadius == "string"
        || typeof s.WebkitBorderRadius == "string"
        || typeof s.KhtmlBorderRadius == "string";
  }

See if there’s a property style.borderRadius and see if it contains a string (instead of undefined). If that’s the case, the browser (apparently) supports CSS border-radius.

Now I do not doubt that this approach works fine in the current crop of desktop browsers. It fails abysmally, however, on some mobile phones, as we’ll see in a moment.

Besides, this test does not prove anything about CSS border-radius. Instead it says something about JavaScript style.borderRadius.

Sure, these two properties are supposed to reflect the same bit of data, but what if they don’t? Suppose there’s a browser bug that doesn’t allow you access to style.borderRadius while border-radius works fine? Or vice versa? The test might return false positives or false negatives.

More in general, this testing methodology assumes that no browser will ever make a mapping mistake between CSS and JavaScript. That means that an entire category of bugs will never be discovered by such tests, and — worse — that the tests may generate false positives or negatives.

This is not just a random theoretical musing. I actually found such a bug. At least four mobile browsers will return a false positive when this testing methodology is used on font-style: italic.

If this methodology doesn’t work for one single browser, it is inherently untrustworthy. After all, any testing methodology must work everywhere, or it is worthless.

The font-style: italic problem

As I think I said before, I discovered that some basic CSS such as font-style: italic does not work in certain mobile browsers (Opera Mobile < 9 in mobile mode , Opera Mini, and Bolt). We’ll leave the whys and wherefores for another day, and concentrate on the aspect I tested last Thursday.

What I did then is change the test page a bit in order to include a simple JavaScript test:

<p style="font-style: italic" id="test">font-style: italic</p>
<script>
document.write(document.getElementById('test').style.fontStyle);
</script>

If a browser does not support font-style: italic, so the theory goes, style.fontStyle would be empty, too. The point is, it isn’t.

Two Opera Mobiles, Opera Mini, and Bolt all faithfully reported italic, even though they do not support font-style: italic (or, let’s put this more precisely, they do not use an italicised font for rendering the test paragraph).

These browsers do not support font-style: italic in any meaningful way, but an automated testing system based on JavaScript style.fontStyle would report that they do in fact support the CSS property. It would give a false positive.

This false positive conclusively proves that testing CSS compatibility by means of JavaScript does not work. (Remember, one single false result is enough to invalidate the entire approach.)

Testing CSS compatibility by means of JavaScript is methodologically unsound and should be avoided at all costs. CSS compatibility tests (and probably the JavaScript ones, too) must be done by hand, by creating a test page and looking at them. Automated systems are just not going to work.

This is the blog of Peter-Paul Koch, web developer, consultant, and trainer. You can also follow him on Twitter or Mastodon.
Atom RSS

If you like this blog, why not donate a little bit of money to help me pay my bills?

Categories:

Comments

Comments are closed.

1 Posted by unscriptable on 20 May 2009 | Permalink

Hey PPK, you're missing the whole point of Juriy's work. He is not trying to create some sort of automated CSS compatibility testing system. Rather, he's creating a baseline "CSS support detection mechanism" so we can remove more browser sniffing in our web apps.

How else would you propose we fix code like this:

if (isFF == 2)
/* implement a FF2 work-around for an inline-block bug */
else
/* do something the CSS3 way */

2 Posted by ppk on 20 May 2009 | Permalink

To me that's the same. CSS support detection === CSS compatibility testing.

And I have severe problems with using JavaScript to decide which CSS should be used. CSS problems are CSS problems and should be solved in CSS.

Anyway, it wasn't my intent to attack Kangax's work; it's an interesting idea that just happened to send my thoughts in the opposite direction.

3 Posted by Jonathan Snook on 20 May 2009 | Permalink

While I generally agree with your sentiment about using JavaScript to test for CSS feature support, I'm not entirely sure that the lack of an italic font means that there's no support for font-style:italic. I'd compare it to specifying something like font-family: Helvetica; Does the fact that Helvetica, in this overly simplistic font stack, mean that there's no support for font-family? I'd imagine that your not seeing font-style:italic because the system does not have an italic variant in which to use.

I think the larger message here is that when it comes to something that is reliant on display, the only way to reliably test it is by looking at it.

4 Posted by ppk on 20 May 2009 | Permalink

I've heard this argument before, and it might be true. Still, if a certain browser does not render an element with font-style: italic as italic, that's a bug, strictly speaking.

The really weird thing is that I've installed four browsers on my Nokia E71, which, as far as I can see, all use the same font. Nonetheless, two of them (Opera Mini and Bolt) do not support font-style: italic, while the other two (Skyfire and S60 WebKit) do.

So it's not just a question of which fonts the device allows you to use; it's more complicated than that.

5 Posted by kangax on 20 May 2009 | Permalink

Thanks for the tests, @ppk. They confirm my concerns about brittleness of `style`-based inference.

I don't have much time at the moment to comment on a post; just wanted to chime in and say that I have recently found similar discrepancies in Firefox 3.0.x. Firefox has `style.textShadow` of a string value, yet absolutely does not support it visually (you can see it yourself in the original test page).

This sucks.

I've been playing with alternative means since that, but haven't tested anything comprehensively yet. The idea is to check if computed style of an element is affected by the assignment of a specific value to a specific CSS property.

Something along the lines of:


var isSupported = (function(){
var getStyle = (function(){
var view = document.defaultView;
if (view && view.getComputedStyle) {
return function(el){ return view.getComputedStyle(el, ''); }
}
else if (document.documentElement.currentStyle) {
return function(el){ return el.currentStyle; }
}
})();
return function(prop, value) {
var el = document.createElement('div');
var orig = getStyle(el)[prop];
el.style[prop] = value;
return getStyle(el)[prop] !== '';
}
})();

isSupported('textShadow', '3px #000');

6 Posted by David Storey on 20 May 2009 | Permalink

I'm pretty sure it is related to if the font is available or not, as I know many phones do not supply an italic font, and Opera clearly does support font-style: italic in the browser engine. The end result is of course the same as an italic font is not being displayed, but on a different phone it might. Opera Mini is a bit different in version 4 and above as it generates its own font (I'm guessing it doesn't have a italic font).

7 Posted by fearphage on 20 May 2009 | Permalink

I disagree sharply that useragent sniffing is similar to css compatibility detection. Being an Opera browser user, I'm used to being shunned by the following code:

if (isIE) {
// do stuff
} else if (isNetscape) {
// do other stuff
}

There is no path for every other non-IE browser that doesn't have have Netscape as the navigator.appName although they fully support the functionality. Detecting if things work is a much better solution than checking the useragent.

In your italics example, you pointed out that Opera and Bolt support the italics font-style but just don't visually display it. So it is not a false positive. The support is there. How the useragent defines what "support" means somewhat irrelevant. For instance Opera supports animates gifs and animated pngs, but they can be disabled in the browser. This is not detectable via javascript to my knowledge but you get the point. The same could be said for the marquee and blink tags. Their conventional functionality can be disabled but that doesn't mean that support has been removed.

8 Posted by ppk on 20 May 2009 | Permalink

Let's keep this discussion simple: if I give a certain element font-style: italic, and the browser doesn't display it in an italic font, font-style: italic is not supported. Period.

That said, I'm fully aware this might be a device problem instead of a browser problem, but I just don't have enough information yet to determine whether that's the case (or even how to set up a test for this sort of bugs).

So more research is clearly necessary.

However, I don't buy the argument that the browsers I named "really" support font-style: italic when they just don't show the test element in an italic font. They don't; the lack of italicised fonts on the display proves it.

9 Posted by fearphage on 20 May 2009 | Permalink

Opera and firefox both provides means to globally override conventionally styles and functionality. I can add

* { font-style: normal !important; }

to my user-defined stylesheet and effectively disable italics and oblique font styling. It is up to the useragent how they handle the style. I think your definition of support is flawed. Because I don't see it does not conclude that it is not supported.

Another example:
I can disable javascript but that doesn't make it unsupported in my browser. It is simply disabled. Would you say I am adding and removing javascript support to and from the browser? Presumably the answer is no.

10 Posted by ppk on 20 May 2009 | Permalink

I don't do either in a test page. And that's what counts here.

11 Posted by Sander Aarts on 20 May 2009 | Permalink

@fearphage:
The definition of 'support' that you use only matters when you want to compare different browsers, not when a script has to behave in a certain manner based on on some condition.
If you turn off support for some features then these features are just simply not supported for the documents that are currently used.

12 Posted by Ricardo on 20 May 2009 | Permalink

what about making an automated test to render the page to an image, and compare that image with a reference image (pixel by pixel comparison excluding not styled 'input' elements) created by a browser that behaves properly with the testing property?
In this way, you should check manually just once browser (maybe more, if the browser you choose does not support the property to test)

13 Posted by David Storey on 20 May 2009 | Permalink

I understand what you are getting at, but I think you are framing the statement slightly incorrectly.

you are not really saying the browser doesn't support it, you are testing two things; the browser and the device (and the OS for that matter). The browser can support it, but not display it, as the platform doesn't give it the tools to do the job, i.e. in this case an italic font. You are really saying, for example,that Opera Mobile on x phone on Series 60 doesn't display (or support) italic fonts. The exact same browser on a different phone, or a different OS may display the font in italic. The browser clearly supports the CSS, just can't display it. It isn't a bug in the browser, like it isn't a bug that IE can't display a Helvetica font as Helvetica isn't installed on Windows.

I would say this difference is worse on feature phones. On smartphone OSs most phones on the same OS *should* have the same fonts installed. Server-based browsers are different as they can generate fonts instead of use what is available on the phone. Mini did this as many phones only had one font at one size (I'm looking at you RAZR)

14 Posted by David Storey on 20 May 2009 | Permalink

I'm not saying you are wrong that we shouldn't use JS to test CSS though. Just that it isn't really correct that the browser doesn't support something if it doesn't display it.

Automated testing can work though. We do huge scale automated testing of all sorts of things in our build system at Opera, as it would simply be impossible to run many thousands of tests by hand for every build. Obviously these are very advanced automate testing, and can do things like comparing if the display changes from a previous build.

15 Posted by ppk on 20 May 2009 | Permalink

David, I see what you mean but I use a simpler definition of compatibility. If an author writes font-style: italic he expects text to become italic. If that doesn't happen, I count it as a bug.

I totally agree that the device matters a lot (I *know* Opera supports font-style: italic) but that's not what's at issue here.

If a certain browser/device combination does not support font-style: italic my tables should note that fact clearly, regardless of whose fault it is.

16 Posted by Aaron Gustafson on 21 May 2009 | Permalink

It's an interesting problem to attempt to address, but, for better or worse, JavaScript is the only way we have of diagnosing what is supported and what isn't. What would be *really* helpful, of course, would be some sort of internal notification of whether setting the property will fail.

I'm curious if you've tried playing with eCSStender's isSupported() method at all in looking for property support. The method uses a different approach to Juriy's, whereby it attempts to set the property and then read back the value to ensure a match (more along the lines of what I'd hope an API within the browser would do). Opera may tell a different story to eCSStender::isSupported() than it does to getStyleProperty(), but I can't say for sure as I haven't tested it in Opera Mini/Mobile; I've only tested it against the desktop version of Opera.

17 Posted by Diego Perini on 21 May 2009 | Permalink

@ppk,
feature testing in Javascript is to test for support or presence of a functionality to be conditionally used by code, not to verify it's visual appearance or representation to users.

A visually impaired person can see different colors from those specified, or a monitor has a broken GREEN sync...we cannot test that, we can just test if the objects we manage accepts being modified or a method exists to be called and gives expected "non visual" results.

18 Posted by Michiel N on 22 May 2009 | Permalink

I get the feeling a lot of commenters are not really getting the point - and to be honest, PKK has a hard time getting the point across.

So - after slightly insulting both commenters and owner of this site - let's see if I can make things clearer :)

Two points:

1) PKK is not arguing against javascript support detection in scripts at all. In fact, he always is a strong advocate of support detection, instead of browser sniffing.
He is just arguing that you can't use js support detection as an automated test of CSS support.
If you use a script on you website that changes font-style to italic, by all means detect support for it with javascript. Just know you can't be 100% certain your end-user actually sees an italic font.
If you want to make a compatibility table for CSS - something only people like PPK are crazy enough to do - do it by hand.

2) PKK is not testing devices and he is not testing browsers. He is testing real-life cases of browsers on devices (the only thing that matters to an actual end-user). If something doesn't work, that is a bug of that specific combination. His example of four browsers on one device, two supporting font-style:italic; shows it's a lot more complicated than just pointing at browsers or devices.

19 Posted by David Mark on 9 June 2009 | Permalink

Forget about the test pages, PPK. You've missed the boat on this one.

You detect CSS properties with script when your script is going to set those properties. Otherwise, you risk augmenting host objects (always a bad idea.)

Also, the technique works brilliantly when used with understanding. It's been around forever, most notably it is the way to detect the right opacity style for older agents. IIRC, the idea originates with Richard Cornford (a very good source for cross-browser scripting techniques.)

Empirical evidence is pretty weak in comparison. And what would you do with it anyway? Chart the results? That's also why a lot of your chart comments are light on understanding.

Of course, I don't write test pages. YMMD.