Writing custom extensions for the Remarkable JavaScript Markdown parser

We were presented with the task of finding a Markdown parser written in JavaScript about the same time the CommonMark drama was well underway. Each of us had our requirements for our ideal parser:

I looked into Marked at first, and found the extension support difficult to the point where I ended up forking it to rewrite some functions to allow for the creation of new rules at the cost of a couple hundred milliseconds. I was planning on making it a marked wrapper in the end, but after some team discussion we decided to try another parser out. Remarkable had only recently been released at the time but development was moving quickly, it passed the CommonMark test suite, and seemed to fit a more modern style of JavaScript writing. It also was faster than Marked, though I haven’t fully understood how, even after reading the code for quite a while. It also obviously supported extensions, addition and manipulation of rules that would be far easier to handle and maintain than an entire fork. There were no clear examples on how to create new plugins or extensions for Remarkable though. So having worked on creating some and having a measure of success it is time to share the knowledge.

A Simple Extension: Open Links in a New Window

The simplest ways of modifying the Markdown parser is to change the rendered output. Open up the rules file from the Remarkable source and search for the current rule renderer you want to modify. In this case we are looking for link_open. That is what we want to extend to add our own code, note the output string is the first part of a complete anchor tag. These rules are exposed through a property named renderer on a new Remarkable object.

var markdownParser = new Remarkable();
// open links in new windows
markdownParser.renderer.rules.link_open = (function() {
  var original = markdownParser.renderer.rules.link_open;
  return function() {
    var link = original.apply(this, arguments);
    return link.substring(0, link.length - 1) + ' target="_blank">';
  };
})();

Through a bit of JavaScript closure we have saved off the original link_open function as the variable original which we then use to get the normal Remarkable anchor tag output. Then we append our target property to open all links in new windows. This function replaces the one on our current instance of the Remarkable object. For my purposes I have extended a single Remarkable object and reuse it throughout my application.

Another Simple Extension: Adding Images to Lighbox

This is another rule rendering extension example where we want to include any Markdown images into a lightbox. The lightbox library I use looks for anchor tags with a data-lightbox attribute wrapping an img element. Again we look into the rules file from Remarkable and find the rules.image is the renderer we want to extend to add this functionality.

var markdownParser = new Remarkable();
// add images to lightbox
markdownParser.renderer.rules.image = (function() {
  var original = markdownParser.renderer.rules.image;
  return function(tokens, idx) {
    var href = Remarkable.utils.escapeHtml(tokens[idx].src);
    var imgOutput = original.apply(this, arguments);
    var anchorStart = '<a href="'+href+'" data-lightbox="markdown">';
    return anchorStart + imgOutput + "</a>";
  };
})();

We’ve had to do a bit more work here since the anchor tags href is a link to the image used by the lightbox library. This exposes a little of the Remarkable inner workings. We get an array of tokens, and our current index that we can use to get information gathered in earlier steps. Specifically we want to get the image src. We leverage the existing Remarkable utility functions exposed in the utils property to get a safe href string. Then we get the original image function output, and return it wrapped in our new anchor tag.

A True Remarkable Plugin

Remarkable has a use function to include new plugins. The architecture isn’t well documented however, and this requires a great deal more work. By no means am I suggesting my example is the best practice, but it does work. Use expects a function that takes the current Remarkable instance, and an optional options parameter. This function assigns adds itself to the rules list and to the parsing functionality. So far we’ve only looked at rendering. The parser goes character by character to match a complex rule set. For example to you can add emphasis to text by using asterisks or underscores. The emphasis parser looks for a sequence of either, but not exceeding 3 characters, then continues down the string it has been given looking for a matching closing sequence of characters. This type of code isn’t often seen outside of parsers so may look alien.

Lets make a smiley emoticon plugin as an example. We’ll define the syntax as a colon followed by an exclamation mark followed by a colon :!:. That markdown will be rendered as a smiley GIF. Our parser function will check each character in the sequence to make certain we finish a completed smiley before adding it to the token list to be rendered.

var parse = function(state) {
  // I get my character hex codes from the console using
  // "0x"+"[".charCodeAt(0).toString(16).toUpperCase()
 
  var pos = state.pos;
  var marker = state.src.charCodeAt(state.pos);
 
  // Our pos starts at 0, so we are looking for :!:
  // marker starts with the character at 0
  // Given state.src :!:
  // We are here:    ^
 
  if (marker !== 0x3A/* : */) {
    return false;
  }
 
  pos++;
  marker = state.src.charCodeAt(pos);
 
  // Given state.src :!:
  // We are here:     ^
 
  if (marker !== 0x21/* ! */) {
    return false;
  }
 
  pos++;
  marker = state.src.charCodeAt(pos);
 
  // Given state.src :!:
  // We are here:      ^
 
  if (marker !== 0x3A/* : */) {
    return false;
  }
 
  state.pos = pos+1;
 
  if (state.pos > state.posMax) {
    state.pos = state.posMax;
  }
 
  // Having matched all three characters we add a token to the state list
  var token = {
    type: "bangSmiley",
    level: state.level,
    content: marker
  };
 
  state.push(token);
 
  return true;
};

Making our renderer is much more straight forward. The render will be called when the token comes back up in the rendering process. The render function we create will pass back the string to be rendered. The check you see at the end for options.xhtmlOut is to comply with the Remarkable option for outputting XHTML.

var render = function(tokens, idx, options) {
  var smileyString = '<img src="http://upload.wikimedia.org/wikipedia/commons/thumb/3/30/Binette-typo.png/200px-Binette-typo.png" alt="" width="20" height="20"';
  smileyString += options.xhtmlOut ? '/>' : '>';
 
  return smileyString;
};

Finally we combine both our new parser and render functions to pass into the Remarkable.use function and complete our plugin.

// var parse = code above
// var render = code above
 
var bangSmiley = function(md) {
  md.inline.ruler.push('bangSmiley', parse);
  md.renderer.rules.bangSmiley = render;
};
 
var markdownParser = new Remarkable();
markdownParser.use(bangSmiley);

This Markdown:

[new window](http://github.com)
 
![lightbox](https://assets-cdn.github.com/images/icons/emoji/unicode/1f44d.png)
 
:!:

Becomes:

<p><a href="http://github.com" target="_blank">new window</a>
</p>
<p>
 <a href="https://assets-cdn.github.com/images/icons/emoji/unicode/1f44d.png" data-lightbox="markdown"><img src="https://assets-cdn.github.com/images/icons/emoji/unicode/1f44d.png" alt="lightbox">
 </a>
</p>
<p><img src="http://upload.wikimedia.org/wikipedia/commons/thumb/3/30/Binette-typo.png/200px-Binette-typo.png" alt="" width="20" height="20">
</p>

This is just scratching the surface of writing plugins for Remarkable, and writing a parser in general. The smiley example above is an inline rule, there are also block and core rules. Hopefully these examples get you started and you can come up with new and creative ways to extend Remarkable and Markdown syntax to suit your own needs.

Here is a Gist of the full code: https://gist.github.com/barretts/8677348c6e77c2b3ea80

The poor state of Android JavaScript debugging

Cross-platform mobile web development is heating up and the the performance level is continuing to increase as new an more powerful devices are released. One of the allures to this style of mobile development is the single code base you can use across multiple platforms. With wrappers like Cordova and modern web frameworks you can get up and running super quick, but you incur a technical debt in debug time that you won’t know you need until it breaks in an unexpected way. It will break if you work on a sufficiently complex application.

When you start in cross-platform you’re probably thinking about hitting the most devices possible with the least amount of development time. Google reports, as of this posting, that Ice Cream Sandwich and KitKat account for 17.7% of Android installs (http://developer.android.com/about/dashboards). Those devices have it easy because they are coming with Chrome for Android installed. Chrome for Android features the future of debugging mobile web applications with the new remote debugger you can read about https://developers.google.com/chrome-developer-tools/docs/remote-debugging. That is wonderful for the future, but the other 82.3% of devices don’t get this new browser, and that means you don’t get those fancy new debugging features.

88294The first option for debugging on Android is a hosted JavaScript file that is served by a tool called Weinre. The quick way to give Weinre a try is to use the version hosted by PhoneGap at http://debug.phonegap.com. Just load up the script on that page in your app, but you’ll have to use Safari, Chrome didn’t work for me though it did for my colleague. Again poor planning for the extra Android debug efforts lead to confusion and short patience. From here you can see current CSS, DOM and execute JavaScript, but you can’t set any breakpoints. It is very slow, think slower to use than the Android ARM emulator is, and will cause frustration for developers.

Another option that you can use is jsHybugger which is an instrumentation library. This is the solution for proper JavaScript debugging, but the instrumentation does change your code when enabled and deployed to your device. I was personally met with race conditions when the JavaScript files were loaded in a large application and felt uneasy about relying on it for accurate debugging. We suffered from the same poor planning I am advising you don’t do in your project. By the time we needed to debug the app our patience was short, and we didn’t give it enough time or planning.

In the end, to debug your cross-platform mobile web application I would use a combination of both jsHybugger and Weinre. When you start your project and are thinking about all the speed of development improvements just don’t forget about debugging your program. You’ll want to allow for more time debugging your application on Android than you would on iOS. Eventually this won’t be as much of an issue, and it isn’t an issue at all of you only support Chrome for Android, but for now you’ll need to work with more than one debugging solution to deploy a successful cross-platform mobile web app.

How to Hide Your JavaScript Source Code (Don’t!)

Just don’t. If you are trying to focus on hiding your trade secret sauce that you’ve written into your new single-page JavaScript application you’re just wasting your time. If you’ve coded the entire framework and all your libraries by hand then no one will be able to understand your code enough to steal pieces, they will do an entire copy of your website. The odds are however that you’ve leaned on some existing libraries, that more of the code in your final product was written by someone else entirely. That’s completely okay! For years now as developers we’ve stood on the shoulders of giants to create our finished work. Great libraries like jQuery, frameworks like AngularJS, layout frameworks like Bootstrap all give us a great and well tested path of least resistance to launch.

The idea that you’re able to protect your code through minification or obfuscation isn’t new. ActionScript decompilers started coming out much to the dismay of Flash developers and shortly after that the ActionScript obfuscators hit the market. Minification is the process of removing all extra spacing and lines from a file or set of files for the reason of saving space. Obfuscation is the process of making your code illegible, even going so far as to encode strings that are later decoded. Obfuscation can lead to introducing hard to debug issues, and only protects your strings from a first glance as it is simple enough to unencode the entire document and retrieve said strings.

It’s the strings that can allow anyone to see in your code what your trade secret sauce is. What combination of libraries you used in a project help make it unique, but the interconnecting code that brings them together is what make it a product. Each library has a certain set of strings in them that act as markers, version numbers, unique variable names that will allow those libraries to be identified even in a minified state. You can’t hide the recipe for your JavaScript application. So stop trying! There are also restrictions on certain libraries that ask for attribution, or that they remain as part of open source projects.

View source is the spirit of the web, being able to look at how the documents and applications are structure is part of how the World Wide Web became so popular. That low barrier to curiosity where you could simply inspect the client side code being delivered to you allowed for a generation of tinkerers to be born. Those same tinkerers went on to build the libraries that you have now brought together to make your application. Don’t focus on hiding what you’ve created. Focus on making the best product possible and be proud of the libraries and frameworks you’ve used to create your product. Without a herculean effort, no one will be able to discern a gem of code that you’ve wrote, pull it out and then profit off it. Stop worrying and get back to making sweet JavaScript applications.