Coverage and Profile Information Gathered from -D

The disassembly information provided by SpiderMonkey's -D switch is much richer than the plain coverage data I gathered with my trace hook. However, the disassembly is printed straight to stdout which makes it more difficult to separate from test output and harder to parse. So, I wrote a small patch which makes -D take the filename to write the disassembly to instead of stdout.

http://pastebin.mozilla.org/1296772

I need to get my situation with the mozilla-central repository figured out so I can create a branch and commit. In the meantime, there's the small patch.

Then, I rewrote my coverage_parser.py script in dom.js to parse the -D output and was able to generate nice coverage files, including displaying the number of total bytecodes executed on each line, and nice profile files, sorted from the lines which executed the most bytecodes down to those that executed the least.





With the test suites we scraped together from various places, we have almost 50% coverage of dom.js right off the bat.

Dumping Bytecode with SpiderMonkey

While trying to implement my code coverage tool, SpiderMonkey's -D flag was brought to my attention.

You need to have a debug build of SpiderMonkey. You can find instructions on how to get the source and build here.

For the given input file foo.js:

var a = 1 + 1;
var b = 2 + 2;
var c = a + b;

Running with the command-line switch -D gives:

$ js -D foo.js
--- PC COUNTS foo.js:1 ---
loc counts x line op
----- ---------------- ---- --
main:
00000:1/0/0 x 1 bindgname "a"
00003:1/0/0 x 1 int8 2
00005:1/0/0 x 1 setgname "a"
00008:0/0/0 x 1 pop
00009:1/0/0 x 2 bindgname "b"
00012:1/0/0 x 2 int8 4
00014:1/0/0 x 2 setgname "b"
00017:0/0/0 x 2 pop
00018:1/0/0 x 3 bindgname "c"
00021:1/0/0 x 3 getglobal "a"
00024:1/0/0 x 3 getglobal "b"
00027:1/0/0 x 3 add
00028:1/0/0 x 3 setgname "c"
00031:0/0/0 x 3 pop
00032:1/0/0 x 3 stop

--- END PC COUNTS foo.js:1 ---

Useful and interesting.

Code Coverage Reporting in JavaScript

Now that I am working on dom.js, I need to learn an entirely new environment and all the tools that go with it. JavaScript is also fundamentally different from other scripting languages because usually the environment it is executing in is the browser rather than the command line. However, dom.js is designed to be used in environments where a native DOM does not already exist, such as in Node.js or in SpiderMonkey. Since it's such a unique project many of the existing tools don't really apply.

I went looking for code coverage tools that we could use to determine how much of the dom.js code was being exercised by the test suites we have in place right now. Several coverage tools exist for JavaScript, as discussed on StackOverflow here: http://stackoverflow.com/questions/53249/are-there-any-good-javascript-code-coverage-tools

For example, the ffhrtimer project (http://hrtimer.mozdev.org/) is a nice firefox extension that provides high resolution timers and UI to display JavaScript code coverage, but it can't easily be integrated into the SpiderMonkey command line js and only runs on FireFox 3.0.

JSCoverage is interesting (http://siliconforks.com/jscoverage/) but it requires source-level translations on the javascript in order to record coverage information. It makes this easy by providing a web server that automatically translates javascript that it serves as well as a proxy that translates any javascript that passes through it, but this does not really fit into our model where we are testing from the command line.

js-test-driver (http://code.google.com/p/js-test-driver/wiki/CodeCoverage) looks good but it also is designed to work in a browser environment.

Finally, JSChiliCat (http://jschilicat.sourceforge.net/) is getting closer because it allows running of tests without a browser being involved, but it also requires Rhino for running the tests. dom.js needs the HEAD version of SpiderMonkey since it uses extensions to JavaScript which are not widely implemented, Proxies and WeakMaps.

So, I looked at the way ffhrtimer gathers coverage data and modified SpiderMonkey to have a --coverage switch which installs a simple hook that prints out the current javascript filename and line number for every line of execution.

JSTrapStatus
CoverageHook(JSContext *cx, JSScript *script,
jsbytecode *pc, jsval *rval, void *closure)
{
const char* filename = JS_GetScriptFilename(cx, script);
uintN lineno = JS_PCToLineNumber(cx, script, pc);

printf("CoverageHook %s %d\n", filename, lineno);

return JSTRAP_CONTINUE;
}

Here's how this hook is installed as a callback:

JS_SetInterrupt(cx->runtime, &CoverageHook, NULL);

Now, the question is how best to store the data for use by an analysis tool. ffhrtimer used an in-memory data structure to keep track of which lines in which files had been visited, but for simplicity I think I am going to use the code above, writing the files and line numbers to a file 'coverage.out' for post-processing with a python script.

I started work at Mozilla yesterday. It has been quite a whirlwind. I'm sitting next to Brendan Eich and working with David Flanagan. David's JavaScript: The Definitive Guide was the reference I turned to when I seriously started with web programming in 2000, and when Brendan Eich wrote JavaScript I was working on server-side scripts hosted in LambdaMOO and had no idea what I wanted to do with my career. Life is strange. If I could go back in time and tell my 1995 self where I am now I don't think I would believe myself.

I'm having a lot of fun learning more about the history of JavaScript and the projects we are working on. After being so deeply embedded in the Python world for so long, it feels refreshing to venture into completely alien terrain. Some things are familiar and some things are incredibly strange. It feels very natural overall; I think if something like WebSockets had existed in 2000 I never would have discovered Python and would have stuck with JavaScript and the LambdaMOO programming language. Python was class oriented; JavaScript was prototype oriented like LambdaMOO. I needed some intermediate glue language to handle JavaScript's inability to use plain old socket objects though, and thus my love affair with Python was born.

The first project I am helping with is dom.js, a project whose aim is to implement the common browser DOM APIs in pure JavaScript. This project will be useful for a server-side implementation of the DOM for use in node.js and will also be useful as a DOM implementation for Narcissus which is just straight up JavaScript written in JavaScript.

Woah, man. Meta. I love it.

Finally there is Zaphod, which is a FireFox extension which installs Narcissus as the default JavaScript interpreter, useful for rapid prototyping of changes to JavaScript itself.