I'm writing a CSS book
JavaScript archives Compatibility CSS coaching Thidrekssaga QuirksBlog Donations Politics Mobile About COH
QuirksMode
PPK sitemapcontact

2005: The year of data retrieval

With http://map.search.ch for a case study, Simon Willison announced, and Dave Shea confirmed, that 2005 is going to be the year of JavaScript, that our beloved language is going to hit the big time again, though, one hopes, in a more responsible way than in 1998. I fully agree, but I'd like to add a few comments, and try to narrow down the questions a bit.

The map

http://map.search.ch is an interactive map of Switzerland which uses a nifty but simple way to gather its data. As the user zooms and pans, it automatically loads the necessary images and displays them in existing or new layers. Since the interface works smoothly (always more difficult than it seems), this is a good example of how JavaScript can enhance web pages.

Note: enhance. The creators have gone to some trouble to allow old browsers access to the data, too. Although their solutions don't always work (my Netscape 4 wants to be resized before showing any images, but this browser was born a troublemaker; more seriously, Netscape 3 doesn't show any map at all), this doesn't affect the basic theoretical principle: this is a useful application that could work in all browsers.

XMLHTTP

Simon and Dave originally thought it used XMLHTTP, not an unreasonable idea when confronted with a working data retrieval system silently running in the background of a web page. One of the developers pointed out that the mapplication doesn't use it. He's right, and yet he's not totally right.

I haven't studied the source code closely, but it seems the mapplication requests the necessary images in the ancient and hallowed mouseover tradition ([someImgElement].src = 'pix/someImg.jpg'). Simple and efficient. (Incidentally, I was gratified to see my event target, mouse position and Find Position scripts right at the top of the source code.)

As we all know, ancient mouseover scripts preloaded their images, that is, they sent HTTP requests to the server to fetch all images that the page might conceivably need. Even though map.search.ch doesn't use modern XMLHTTP, Simon and Dave were correct after all: it uses an ancient and reliable subset. This subset can only be used to retrieve images, but that's quite good enough for a mapplication.

DOM, DHTML and data retrieval

I feel there's some confusion over terminology that ought to be settled first. Simon calls the map a DOM application, while Dave calls it a DHTML application. I disagree with them both and will call it a data retrieval application which uses a DHTML interface layer to present the data to the user.

Which forces me to define these three kinds of scripts:

  1. DHTML scripts change the presentation layer of a site (in other words, the CSS).
  2. DOM scripts change the structural layer of a site (in other words, the XHTML).
  3. Data retrieval scripts retrieve data which they pass on to an interface layer that uses DHTML and/or DOM scripts to present the data to the user.

Client side silent data retrieval scripts could very well become the hit of 2005, but there are a number of technical hurdles to be taken first.

Data retrieval techniques

Roughly speaking, I see three viable techniques for client side silent data retrieval:

  1. Import XML, a method I cobbled together back in 2000 and that works only in Explorer Windows and Mozilla. Although it appeared in Flanagan's Definitive Guide (4th ed), it never really took off. I'm not sure if it's fundamentally different from XMLHTTP, but it uses different JavaScript methods. I think we can write it off, but I'd like to give it one more moment in the spotlight before laying it to rest.
  2. XMLHTTP, originally invented by Microsoft but subsequently embraced by Mozilla and Safari (and it seems Opera's going to follow). Clearly better than importing XML for compatibility reasons, and maybe it's more flexible, too.
  3. Iframes. Yes, I consider iframes a valid data retrieval system. It's quite possible to load a page into an iframe, ramsack its element nodes for useful information and add the data to the main page. The iframe could even be invisible.

XMLHTTP is hot, but is it better than iframes for data retrieval purposes? At the moment I have no idea, but it seems that in some cases it's the worse choice, for now, at least.

Accessibility

When conducting some (inconclusive and unpublished) XMLHTTP experiments I tried to trick the browsers into downloading an HTML page instead of an XML document. It didn't work (though that might be due to my methods. Try to solve this one, if you can).

Update: Seems to have been my methods. XMLHTTP allows you to get an HTML file.

Why is this a problem, maybe even an argument in favour of iframes? Because of the accessibility of future data retrieval and presentation applications.

All data must be divided into chunks somehow, and a very simple way of ensuring the application's continuing graceful degradation is to store these data chunks in HTML pages. If the advanced application doesn't work, a click on the link or the area will simply load this HTML page and give any device access to the data.

If we could use these HTML pages for both layers of functionalities, if we could either show them as separate pages or niftily and silently include them in our advanced interface, we'd have ensured accessibility in the simplest possible way.

But XMLHTTP requests cannot request HTML pages, and therefore such a system cannot use it.

Of course this line of reasoning doesn't hold true for every project. If you've got the skills and the budget to arrange for server side scripts that transform the data into either HTML or XML, depending on the exact browser request, you don't have to worry about XMLHTTP's limitations.

For budgetarily challenged projects, though, iframe data retrieval may well be a good choice.

We've got plenty to think about in 2005.

This is the blog of Peter-Paul Koch, web developer, consultant, and trainer. You can also follow him on Twitter or Mastodon.
Atom RSS

If you like this blog, why not donate a little bit of money to help me pay my bills?

Categories:

Comments

Comments are closed.

1 Posted by dave on 11 January 2005 | Permalink

I hope I am not misunderstanding what you saying, I have no problem loading html, though grabbing the linked js and running a pseudo onload on the http requested content is a bit of a challenge.

using the code at :
http://jibbering.com/2002/4/httprequest.html

2 Posted by Chris Hester on 11 January 2005 | Permalink

DHTML = presentational, DOM = structural you say. But also, as your site shows, you can use the DOM to affect the stylesheets too.

3 Posted by ppk on 11 January 2005 | Permalink

Dave, do you have a link your code example? We'd all benefit to see how you XMLHTTPrequest an HTML page.

4 Posted by J van Velzen on 11 January 2005 | Permalink

Maybe two other RPC or DR techniques.
1. For 1-bit information: an image in combination with the onerror event.
2. Importing Javascript Scripts see: http://www.paranoidfish.org/boxes/2002/01/30/

5 Posted by Rick on 12 January 2005 | Permalink

I am working with XMLHTTPRequest on my site and furniture.rickmosher.net.
(Both under development - until I get a handle on this XMLHTTPRequest stuff)
I bring back things as simple
"Hello There" or as complex as div - table - form I don't yet have a comprehensive view of the results; but I'm not doing anything extraordinary for formatting.

6 Posted by ppk on 12 January 2005 | Permalink

Chris, I've been thinking about your comment for a while, and you're right.

Changes in the structural layer can cause changes in the presentation layer, but not vice versa. Therefore the structural layer is more fundamental and important than the presentation layer.

What does this teach us? I'm not sure yet, but it's something to keep in mind.

7 Posted by dave on 12 January 2005 | Permalink

Again, I am sorry if this isn't what you mean...
Here is the basic version of the code I use:

*****


var xmlhttp,alerted
/*@cc_on @*/
/*@if (@_jscript_version >= 5)
// JScript gives us Conditional compilation, we can cope with old IE versions.
try {
xmlhttp=new ActiveXObject("Msxml2.XMLHTTP")
} catch (e) {
try {
xmlhttp=new ActiveXObject("Microsoft.XMLHTTP")
} catch (E) {
alert("You must have Microsofts XML parsers available")
}
}
@else
alert("You must have JScript version 5 or above.")
xmlhttp=false
alerted=true
@end @*/
if (!xmlhttp && !alerted) {
// Non ECMAScript Ed. 3 will error here (IE or
// ECMAScript.
try {
xmlhttp = new XMLHttpRequest();
} catch (e) {
alert("You need a browser which supports an XMLHttpRequest Object.\nMozilla build 0.9.5 has this Object and IE5 and above, others may do, I don't know, any info [email protected]")
}
}

function RSchange() {
if (xmlhttp.readyState==4) {
alert(xmlhttp.responseText)
document.getElementById('content').innerHTML=xmlhttp.responseText
}
}
function go() {
if (xmlhttp) {
d=document;
// must be in same domain
xmlhttp.open("GET", "any.html",true);
xmlhttp.onreadystatechange=RSchange
xmlhttp.send(null)
}
}

The content will appear between here
and here.Do it
More information on xmlhttprequest object, including fuller examples.


8 Posted by dave on 12 January 2005 | Permalink

Sorry for the above post (#7). I tried to post some code, it showed up fine in the preview and then got stripped for display.

how does one post code here?

9 Posted by ppk on 13 January 2005 | Permalink

You don't. If you really want to show a code example, post a link to the page you created.

But I tried the code and you're right, it gets HTML pages without trouble. I wonder what I've done wrong in my original tests...

10 Posted by Adam Platti on 13 January 2005 | Permalink

A major event in "2005, The Year of Javascript" will be the release of a Yahoo mail client based on Oddpost's technology!

11 Posted by ppk on 13 January 2005 | Permalink

What is Oddpost? And is Yahoo going to imitate Gmail? They could do worse.

12 Posted by Rahul on 14 January 2005 | Permalink

Oddpost, http://www.oddpost.com/ , is something like Gmail on steroids. And has been around since long before Gmail.

But it only works on MSIE. Shame, shame? The makers defend themselves on http://unraveled.com/archives/2003/07/an_interview_with_ethan_diamond (see the reply to the second question).

In latest news, Oddpost has been acquired by Yahoo.

13 Posted by Angus Turnbull on 14 January 2005 | Permalink

All this XMLHttpRequest talk made me finally get organised and polish off a project of mine that's been ticking for a while: HTMLHttpRequest. See: http://www.twinhelix.com/javascript/htmlhttprequest/

It's an IFRAME remote loader at heart, but with some nifty tricks to detect load events in various browsers, so you can load and parse pure HTML files with no additional ONLOAD events in them. Plus it works in a few browsers XMLHttpRequest doesn't (IE/Mac, IE4, Opera 7, older KHTML versions, etc. etc).

IMHO, IFRAME scripting is underrated; as long as you don't need to send more information to the server than can be conceivably encoded in a GET request, it's better to rely on W3C standards like IFRAME+JS+DOM than a reasoanbly proprietary technology like XMLHttpRequest (despite it's reasonably broad support).

14 Posted by Marc van den Dobbelsteen on 14 January 2005 | Permalink

Just for your information:

Take a look at the Domapi DHTML application development framework at http://www.domapi.com.
This community is using rpc calls through the Domapi rpc-component for allmost than
2 years now. This components uses xmlhttp for sync. or async. data transfer between
the client and the server, and has some abstraction layers for databinding etc. It
can handle javascript arrays, JSON, XML etc.

15 Posted by Chris Hester on 14 January 2005 | Permalink

PPK, I wasn't meaning styles that the DOM would remove if they were tied to an element via inline CSS. Say when the element was removed, or the inline style was changed directly. Though that is another interesting issue. What I meant was the direct editing of the linked stylesheet itself. (Er, unless this only works using computed styles the browser generates based on any stylesheets loaded.) It's not something that works cross-browser (if at all) but does show how the DOM isn't tied to structure only. As we know, a change of stylesheet can radically alter a page. 'Display:none' might be used, for instance. So wouldn't that mean presentation was able to alter structure?

16 Posted by ppk on 14 January 2005 | Permalink

"As we know, a change of stylesheet can radically alter a page. 'Display:none' might be used, for instance. So wouldn't that mean presentation was able to alter structure?"

As far as I'm concerned, No. The structure remains the same, one of the elements just happens to be hidden.

But I admit this is a grey zone that allows for more than one interpretation.

17 Posted by Scott Schiller on 15 January 2005 | Permalink

I've been using XMLHTTP on my site for a while, adding functionality where supported: Dynamically loading article content without URL switching.

Each article link element (pointing to say, /content/foo/ ) has an event handler attached dynamically via JS, which intercepts the click and retrieves the content via XMLHTTP instead of loading a new URL - appending a parameter so only the minimal HTML is retrieved rather than a full site template. eg. /content/foo/index.php?xmlhttp=true

The XMLHTTP .responseText (HTML data) is written via DOM (.innerHTML if you aren't using app/xhtml+xml,) to the "content" area after that.

I found also that inline script calls in this loaded content can also be run via an eval() on the innerHTML of script elements within the xmlHTTP.responseText (ie. getElementsByTagName('script');)

It might not be the best method, but it seems to work reliably for me.

The benefit is browsers that don't support XMLHTTP just load a new page, new URL etc. as usual, where browsers that do support it can load new content quicker.

The downside is the "back" button breaks (although you could theoretically do some iframe trickery to try to restore this.)

I'm all for adding cool features in the browser via script, but I think it should always be a "value-added" kind of thing.. Not a required technology providing functionality which the site would otherwise fail without.

Quick PPK plug: I'm a big fan of this site. ;) Keep it up.

18 Posted by Steve Yen on 15 January 2005 | Permalink

I've just published a library called the JavaScript Templates or JST. It let's you do PHP/ASP/JSP style programming but runs in your web browser.

Sick, yes. But I hope folks find it useful. And, doing the JST project helped me re-evaluate my formerly low opinions about DHTML and JavaScript technology.
I think these 2005 predictions will be right on.
http://trimpath.com/project/wiki/JavaScriptTemplates

19 Posted by EQzE on 18 January 2005 | Permalink

I have to second J van Velzen's second method. Using the DOM to import an external, possibly dynamic script works just as well (if not better) than IFrames in some situations. If you're using DOM-compatible methods especially (no innerHTML) then writing out a serialized JS array of element data is much easier, and more compatible, than trying to import the nodes from the IFrame into the parent document (since document.importNode() doesn't even work consistently). I used this in a dynamic calendar script months before I had heard about XMLHttpRequest.

20 Posted by Simon on 21 January 2005 | Permalink

I've had good luck with Javascript Remote Scripting [ http://www.ashleyit.com/rs/ ] which automates the process of using an iframe to do remote scripting. On a recent project I used JSRS to get XML data from Perl scripts which were then processed by JavaScript to update the interface. Works in more browsers than XMLHTTPRequest, most notably IE5/Mac.

21 Posted by Brent Ashley on 26 January 2005 | Permalink

My favourite really lightweight and cross-platform method is to create an Image object and set its source to a script, passing a parameter in the url, then have the script send back an empty page but set a cookie along the way with the response.

see http://www.ashleyit.com/rs/rslite/ for an implementation.

22 Posted by Jury on 30 January 2005 | Permalink

The image object method would be cool, if it worked in the browser with "Images OFF" setting enabled. Unfortunately, it doesn't.

23 Posted by Ross Lawley on 24 April 2005 | Permalink

Does anyone know how to import javascript functions or linked files via xmlhttprequest? The URL posted above is no longer useful.

I'm currently retrieving xhtml and placing it to my page using:
"element.innerHTML = theRequest.responseText;"

Which imports the html fine - but I don't have access to the jabvascript functions in the script block (they are visible via the Dom Inspector)

Anybody have ideas how to get round this??