Ninjas, samurai, and artists

Lately I haven’t been posting as much on this blog. My professional responsibilities have shifted, and much as I still love the issues of file formats, I don’t have as much time to give attention to them. There are still general programming issues that are worth blogging about, though, and I’ll occasionally address these issues here, hopefully along with occasional file format posts.

cover thumbnail, Secrets of the JavaScript NinjaThis weekend I borrowed a book from the company library called Secrets of the JavaScript Ninja. It’s a better book than I expected from the title.

In an inspired error, the cover shows not a ninja but a samurai, with colorful armor, a banner, and a long sword. A ninja is a hit-and-run assassin; he shows up out of nowhere, attacks, and vanishes. A samurai is a dedicated soldier, the Japanese equivalent of a knight, and he follows a code of honor. The software development world has too many ninjas. The samurai is a better, if not ideal, model.

From the title I was expecting a cookbook, one of the many books that provide formulas to follow but no deep understanding. Instead, Secrets of the JavaScript Ninja is about understanding the language. Such books are even rarer for JavaScript than for most programming languages. I’d always tended to think of JavaScript as a half-baked derivative of Java, one where features such as classes, packages, and inheritance were left out to cut it down to a scripting language. This book, though, shows that it’s really quite a different language, a very powerful one in its own right. I still think the language has serious problems, the biggest being lack of standardization, but reading through the book, I’m learning how to think in JavaScript’s terms and to make use of features which other languages don’t have.

It’s not the language I want to talk about here, though; it’s the approach to any language or software technology. Too many programmers don’t have any deep understanding of their craft; they just have a bag of tools that they expect to solve problems for them. The worst can’t do much more than run a Web search for the code they need or beg on Stack Overflow for a solution to their problem. Once I actually had to fix some PHP that consultants from a big-name company had pasted from a website and didn’t know how to adapt to the problem at hand — and I don’t even know PHP! Those are your ninjas.

Even the samurai isn’t a great metaphor. They were a part of Japan’s entrenched feudal culture, and their opposition to capitalism promoted the warlike mindset that culminated in Japan’s role in World War II. Our metaphors should be based on creativity, not war and violence. We should think of ourselves as architects, sculptors, artists. We learn a craft and master the tools that go into it. There’s a real psychological affinity; I know a lot more software developers who are skilled musicians than are skilled fighters.

When I see a book called Secrets of the JavaScript Artist, then I’ll be pleased.

Update: I just noticed that the book itself says: “Ninjas were chosen for their martial arts skills rather than for their social standing or education. Dressed in black and with their faces covered, they were sent on missions alone or in small groups to attack the enemy with subterfuge and stealth, using any tactics to assure success; their only code was one of secrecy.” Just what you want in a programmer, right?

Posted in commentary. Tags: , , . Comments Off

In MS Word, the bullet bites back

There’s nothing new about Microsoft’s ignoring standards and ruining compatibility, but knowing the details is useful. One case I just learned about, from Mark Mandel, is the way it does bullet lists. This applies to the old Word DOC format on Mac OS X.

A 2008 OpenOffice Forum discussion explains the problem. If you create a bullet list in Word and import it into OpenOffice, the bullets are turned into something odd-looking. The file doesn’t use Unicode bullets, but instead uses the Microsoft Symbol font, which has its own nonstandard encoding. This applies only to bullets generated by list styles, not to ones you type in. On Windows, OpenOffice will display the files correctly, since it has access to the needed fonts and mapping.

Apparently the issue can also be manifested when creating a DOC file with OpenOffice and importing into Word, though I’m not clear on how that happens.

The problem is that Word 97/2000/2002 isn’t fully Unicode-compatible, mapping Unicode characters to the 8-bit encodings that its fonts need. This has presumably been fixed in the more recent versions that use DOCX (Office Open XML), but DOC is still widely used as an interchange format, so it’s an important issue. It’s also an illustration of the risks of using undocumented interchange formats.

Tools come and go, effort must be ongoing

In a comment on a JHOVE bug, I said offhandedly that it’s approaching the end of its life. This caused a certain amount of concern in Twitter discussions. Andy said that software tools are one of the best ways to “preserve specific, reproducible knowledge about processes.” I don’t think dropping support of a rather dated tool is a big concern, though, as long as the code doesn’t vanish.

A software application is good for a certain number of years before it needs to be either left as legacy code or completely rewritten. Throwing out code and starting over takes a lot of effort, but it can result in much better code. I started on JHOVE in 2003 as a contractor to the Harvard University Libraries. After a few years it became clear that some of the design decisions weren’t ideal. Its all-or-nothing approach and its tendency to give up after the first error have long been obvious problems. The PDF module is a kludge built on a crock, and that’s without even talking about its profiles. The TIFF module, on the other hand, has a fair amount of elegance.

JHOVE2 was supposed to be the successor to JHOVE. Its creators learned from JHOVE and produced a better design. What they didn’t have was enough time and money to cover all the formats that JHOVE covered. I’ve continued to work on JHOVE because I know it inside and out. Someone else could pick up the work, but it might make more sense for a newcomer to the code to join the JHOVE2 effort instead. However, Maurice noted on Twitter that there hasn’t been much activity lately on JHOVE2 issues.

Both JHOVE and JHOVE2 were funded under grants. When the grant money ended, progress slowed down. The one-time grant model is the wrong way to fund preservation software. It’s an ongoing effort; new formats arise and old ones change, and there are always bugs to fix. What I’d like to see happen is for major libraries in the US to create an ongoing consortium for preservation work, similar to the Planets project in Europe. Or better yet, a consortium bringing together libraries all over the world. It wouldn’t take a lot from any individual institution. Its job would be to maintain information, preservation tools, test suites, and so on, on an ongoing basis. Instead of rushing to create a tool and then leaving it to freelancers like (formerly) me to maintain, it would support maintenance of tools for as long as it made sense and creation of new ones when it’s appropriate.

My voice isn’t enough to call anything like this into existence, but I can hope.

Posted in commentary. Tags: , , . Comments Off

Rescuing Macintosh Files

On Wednesday, September 4, 2013, I talked with a small gathering of the Mac Tech Group at MIT on “Rescuing Macintosh Files.” There was a good discussion, with several people contributing valuable points.

The computer presentation which I used is available as a Powerpoint or OpenOffice document. The PowerPoint one had some problems at MIT with displaying all the images, so if you have a choice the PowerPoint one may work better.

Posted in commentary. Tags: , , , . Comments Off

TIFF/EP vs. Exif

I just discovered today that there are two different TIFF tags called “FocalPlaneResolutionUnit.” Tag 41488 goes by this name and is part of the Exif tag set. Accepted values for it are:

  • 1 = No absolute unit of measurement
  • 2 = Inch
  • 3 = Centimeter

Tag 37392 is a TIFF/EP (Electronic Photography) tag (working draft, final version not available online), also used in other raw formats, including DNG. Its accepted values are:

  • 1 = Inch
  • 2 = Metre
  • 3 = Centimetre
  • 4 = Millimetre
  • 5 = Micrometre

Recently I was sent a TIFF file, as a JHOVE issue, that had a tag 41488 with a value of 4. JHOVE correctly, but perhaps confusingly, reported that the fFocalPlaneResolutionUnit tag had an invalid value.

There are other tags in TIFF/EP that are equivalent, or nearly, to Exif tags. In some cases their values are identically specified, sometimes not. The Exif SubjectLocation tag is numbered 41492 and always has two shorts for its value, giving an X and Y value. The TIFF/EP counterpart is tag 37396, which can also have three shorts (specifying a circle) or four (specifying a rectangle).

I don’t know how this came about, but it’s something to watch out for in software that deals with both Exif and TIFF/EP tags. Some software may accept the EP extensions for Exif tags, but there’s no guarantee this will work.

Posted in commentary. Tags: , . Comments Off

Google Reader is gone (yawn)

feedAs of today, Google Reader is gone. (Correction: It goes away at the end of today. You still have time to export your feed list.) When its termination was announced, some writers declared it meant the end of RSS feeds. From what I’m seeing today, the attempts at panic have died away, replaced by a realization that RSS and Atom are well-understood feed formats and that lots of alternatives exist. Tristan Louis writes for Forbes:

While the death of the most popular RSS reader on the internet could have been seen as something that would represent a grave danger for RSS as a standard, for openness as a concept, and for heavy news consumption, the inverse has been true, as it only solidified RSS’ position in the world as the format for news delivery. Reader was a good product but one can hardly call it a great product and its demise will help rectify some imbalances it created in the market.

Hopefully dedicated users backed up their feed collections as an OPML file. If not, all they have to do is start collecting feed URLs again.

There’s no need to use a website at all to manage your feeds. On my iPod, I use Free RSS Reader, a simple, straightforward reader, though unfortunately it’s no longer being updated. On my main computer I use Sage, a Firefox extension.

A few columnists got a temporary boost in readership and a long-term loss in credibility by proclaiming the demise of RSS. The rest of us are still fine.

Posted in commentary. Tags: , , . Comments Off

Audio and video in HTML5

I’ve been studying up on streaming audio and video and related issues, so lately I’ve been playing with the <audio> and <video> tags in HTML5. It’s possible to put them to good use, but there are more issues than their proponents will readily admit.

A good piece of news is that both tags do exactly the same thing except for their appearance. You can play video with the audio tag and vice versa, and they implement the same DOM model. (Of course, you won’t see anything interesting if you use <audio> for video.)

The main limitation is that these tags support only progressive streaming, which differs from “true” streaming in some important ways. Progressive streaming means downloading a file and starting to play it almost immediately, rather than after it’s finished downloading. Its disadvantages are that the bit rate can’t be adjusted while playing, you can’t keep the file from being grabbed in its entirety with a simple HTTP call, and the download continues to completion even if the user pauses the player. These aren’t always significant problems, but they mean that the new HTML5 tags aren’t the full replacement for Flash which they’re sometimes claimed to be.

There’s enough interest in true streaming that various parties have developed protocols to do it over HTTP. These include HTTP Live Streaming from Apple, HTTP Dynamic Streaming from Adobe, Smooth Streaming from Microsoft, and Dynamic Streaming Over HTTP from MPEG (which its proponents insist isn’t a protocol). There are more details on streaming on my website.

The other problem with the HTTP tags is that there’s no one encoding that all major browsers support. This problem is well known on the video side, but I was surprised to discover it’s even true for audio. The current version of Firefox doesn’t natively support MP3 in the audio tag, and the QuickTime plugin isn’t used in this case (or at least I can’t get it to work). The reason for this is software patents. There’s a good discussion of the state of MP3 with Firefox on Stack Overflow.

You can specify several <source> elements within an audio or video element, and the browser will try each one in turn till it finds one it can play. Two formats or at most three will cover all major browsers. For audio, including both an MP3 and an Ogg Vorbis version should cover all the bases; for video, MP4/H.264 and Ogg Theora should do it, though you might want to add WebM.

Specifying the type attribute as the MIME type of the file (e.g., <source src="song.mp3"
helps the page to load faster, since the browser can determine without examining the file if it can play the file in principle. Make sure, however, to use only the canonical MIME types. From experimentation with various browsers, these include:

  • audio/mp4
  • audio/mpeg
  • audio/ogg
  • video/mp4
  • video/ogg
  • video/webm

If you specify application/mp3 rather than audio/mpeg for an MP3 source, the browser may decide it can’t play it even though it really can.

Another issue is the AV API for HTML5. There’s a pretty decent DOM API to go with the audio and video tags, allowing you to override the player controls and dynamically change content. Some implementations (e.g., Mozilla’s version) have added private extensions. Some people want more power, so there are third-party plugins and JavaScript libraries such as MediaElement.js that extend the API.

It’s a minefield, except that the mishaps come from the absence of an earth-shattering kaboom. Still, using the HTML5 tags is much simpler than Flash or HTTP streaming.

Posted in commentary. Tags: , , . Comments Off


I’m surprised I only got a complaint about this recently. Using JHOVE to validate XHTML files is often painfully slow. In fact, using anything to validate them without caching or redirection of DTDs would be painfully slow. The DOCTYPE declaration brings in the standard XHTML DTD, and it in turn brings in lots of other DTDs. These all have URLs on As you can imagine, this is a lot of traffic converging in one place, and the response is often very slow.

JHOVE has a remedy, but it turns out not to work in this case. In the configuration file, you can declare local copies of schemas and DTDs to be loaded by the SAX entity resolver. This looks something like this:


Unfortunately, there are some problems in JHOVE 1.9. The HTML module processes XHTML files by passing them to the XML module. In this case, the module doesn’t get the parameters that the config file declared for it. In JHOVE 1.10, I’ll fix this by having the HTML module pass its own parameters to the XML module. At present, JHOVE’s processing of XHTML files makes no use of the configuration file’s instructions to the entity resolver.

There’s another complication. The XHTML DTD invokes other DTDs, and JHOVE has to get every one of those in turn. Some of them have relative URLs to other DTDs; these break when they’re redirected to local files. Even making local copies of all the files doesn’t work, as JHOVE doesn’t handle the relative URLs correctly within the file system, and making them work would require changing some existing assumptions. The best fix for the user is to get JHOVE 1.10 when it’s ready (version 1.10B2 doesn’t have the XHTML fix yet) edit all those files so that all the URLs are absolute.

This is a big chunk of work, and I haven’t tested the approach fully. Any ideas on how this might be better handled would be appreciated.

The future of WebM

Yesterday I posted about the WebP still image format, expressing some skepticism about how easily it will catch on. Its companion format for video, WebM, may stand a better chance, though. Images aren’t exciting any more; JPEG delivers photographs well enough, PNG does the same for line art, and there isn’t a compelling reason to change. Video is still in flux, though, and the high bandwidth requirements mean there’s a payoff for any improvements in compression and throughput. The long-running battle among HTML5 stakeholders over video shows that it’s far from being a settled area. Patents are a big issue; if you implement H.264, you have to pay money. Alternatives are attractive from both a technological and an economic standpoint.

With Google pushing WebM and having YouTube, there’s a clear reason for browser developers to support it. YouTube plans to use the new WebM codec, VP9, once it’s complete. I haven’t seen details of the plan, but most likely YouTube will make the same video available with multiple protocols and query the browser’s capabilities to determine whether it can accept VP9. If the advantage is real and users who can get it see fewer pauses in their videos, more browser makers will undoubtedly join the bandwagon.

Posted in commentary. Tags: , , . Comments Off

An eye on WebP

Google has been promoting the WebP still image format for some time, and lately Facebook has added its support. It’s hard to displace the well-entrenched JPEG, but it could happen. It supports both lossy and lossless compression, and Google claims it offers a significant advantage in compression over PNG and JPEG. Google says it’s free of patent restrictions; the container is the familiar RIFF. The VP8 lossy format is available as an IETF RFC; a specification for the lossless format is also available.

The container spec supports XMP and Exif metadata. Canvas width and height can be as much as 16,777,216 pixels, though their product is limited to 4,294,967,296 pixels. As far as I can tell it doesn’t support tiling, though, so partial rendering of huge images in the style of JPEG2000 may not be practical.

Chrome, Opera, and Ice Cream Sandwich offer WebP support, but not many other browsers do. Facebook’s offerings of WebP images have resulted in complaints from users whose browsers can’t read the format. The Firefox development team is starting to warm to it but hasn’t committed to anything yet. Internet Explorer hasn’t even reached that point.

It’s still early to make bets, but WebP increasingly bears watching. I’ve initiated a page for updates and errata for Files that Last with some updated information on WebP. (When I wrote the book, I couldn’t find the lossless spec.)


Get every new post delivered to your Inbox.

Join 41 other followers