Jim Driscoll's Blog

Notes on Technology and the Web

Archive for the ‘tools’ Category

Hosted WordPress Markdown support is badly broken

leave a comment »

Just a quick update to let users of Hosted WordPress know that the recently advertised Markdown support is badly broken if you’re trying to post source code. (If you’re actually reading my blog regularly, you may have noticed some odd formatting problems come through – I’ve finally figured out what’s causing them.)

The Problem

The standard version of Markdown let’s you specify code blocks by just indenting by 4 characters, while the WordPress version of Markdown claims to support the format “language` for syntax highlighting, for instance by saying:

```java
java code goes here
```

Despite this being very explicitly described in their docs the parser they’re using still strips out all &, <, and > characters within the source code in either of those blocks. Not cool – if I’m testing for greater than or equal to, all my code is silently deleted when I post.

No problem, there’s an alternate syntax that’s WordPress specific – just use shortcodes. To use short codes, you say:

[ code lang="java"]
java code here
[/code]

This prevents the stripping out of the < and > characters, so that’s nice, but there’s still one more problem that you need to deal with, if you’re using a product that uses the WordPress API to post, instead of the website directly: &, < and > are escaped when read out by the API. So that means that if you have any of those characters in your source code block, they’ll be escaped into your editing program, and if you try to post an update, this incorrect escaping will be visible to your users.

I use MarsEdit, but the WordPress official iPhone app also reportedly does this. And since these special characters are only converted on one side of the round trip, if you make more than one edit, this conversion happens multiple times, leaving you with code that looks like this:

java code containing  &amp;amp;amp;amp;amp;gt; here

The workaround

There is a workaround, but it is a bit unpleasant. The following code will display correctly, as well as survive roundtripping with the API:

<pre class="brush: java">
List&lt;Integer&gt; l = new List&lt;&gt;();
</pre>

So, if you’re using the website to post, and never use any other way, then just use the short codes. If you are using the API, or even may consider using it, the workflow you will need to adopt to deal with this is as follows:

Create your post in Markdown, as usual, and mark your code blocks with `language. Then, right before posting, search and replace “`language with <pre class=”brush: language”>. Then search and replace &, >, and <, and replace them with &amp;, &gt;, and &lt; – which should work around all the bugs in place, and should also continue working once they fix the bugs as well.

I’ll be reporting these issues to WordPress, but to say I’m disappointed would be an understatement.

Update (Jan 11, 2014):

WordPress has partially fixed the issue. Here’s the current situation:

List<Integer> - works, auto converted to next line

List&lt;Integer&gt; - works, no conversion

Set<List<Integer>> - doesn't work, converted to Set&lt;List&gt;

```java
List<Integer> - works, autoconverted - (added \ to avoid additional rendering)

```

[ \code lang="java"]
List<Integer>  - doesn't work (added \ to avoid rendering here), still has round trip problem
[\/code]

List&lt;Integer&gt; - doesn't work, "tag" stripped

List&amp;lt;Integer&amp;gt; - doesn't work, auto converted, but displayed as though it wasn't!

<pre>
List<Integer> - doesn't work, "tag" stripped
</pre>

<pre>
List&lt;Integer&gt; - works, no conversion
</pre>

List&lt;Integer&gt;  - doesn't work, "tag" stripped

So, if you use >, < in normal text, it works, but it’s auto converted on save to their entity representation. Annoying, but bearable – and if that was all it had done out of the gate, I probably wouldn’t even have found that annoying.

If you use those chars in a code block via triple-backtick, that works. Yay! But it’s converted on save as well.

And if you use the entity in regular text, that works, and isn’t auto converted. Additionally, my original fix continues to work (by wrapping in pre and escaping).

But that’s the end of the good news. Shortcodes still don’t work, and still have the roundtrip problem.

If you use indentation to mark your code blocks, wordpress will see a “tag” and strip it. Same if you use a pre tag and don’t escape.

Much worse, though, is if you use single back tick to show inline code: if you don’t escape the code, it stripts the “tag”, but if you do escape the code, it displays the raw escaped code to the user, so there’s no way to do this:

List&lt;Integer&gt;

and have it work.

The real problem, of course, is that WordPress is trying to re-use their HTML safety code for their Markdown support, when they really, really shouldn’t have. The logic I think they should be using is this:

First, scan for all tags using their safety code. If an allowable tag is found, then that’s left in. If it is not allowable, then escape it as part of rendering. The stopgap solution of escaping gt and lt in the saved code is probably acceptable, if that escaping on render is too expensive, but it’s hard to see how it could be – they’re already doing rendering on display anyway, since they have to convert the rest of document. But really, there shouldn’t be any code, anywhere that strips tags in the WP Markdown task flow.

Sigh. But I’ve been where they are, and I understand that it’s probably not as easy as that.

Anyway, nice to know they’re working on it.

Please keep in mind that there are certainly other render bugs other than the one’s I’ve pointed out here – it’s not hard to imagine other combinations of Markdown that tickle their “safe HTML” library.

But at least if you’re going to just use the triple back tick code block, that should work.

Further update: (Jan 11, 2014):

If you include text with <S> in your code markup (for instance as a Java Generic Type), the entire markup, escaping, and special handling of WordPress will… mangle… the code it saves to the CMS. So, avoid that. Looks like another way to tickle the safe HTML library. (Hey, I did promise there would be more of these bugs.) While the <S> tag isn’t escaped in code markup in Markdown (per the spec, it’s treated as a valid HTML tag), the complete mangling is certainly a bug – you essentially leave the “fenced” area, and the code is passed through the safe HTML filter.

And don’t even ask about using the html shortcode

Another bug – guess what happens if you try:

```html
<nav>
```

Yep, it doesn’t work. The nav tag is stripped. Though tags like div and span work fine. The workaround to use the short code with html works, but roundtripping does still has the same problem of doing repeated substitutions. So, there’s no way to do this. Frustrating.

A note about the broken formatting

At some point, the formatting of this page was broken by WordPress markdown parsing changes – it used to work. I really can’t figure out how to fix it, so I’m leaving it as is. Sorry about the mess.

Advertisements

Written by jamesgdriscoll

January 7, 2014 at 6:24 PM

Posted in tools

QEDServer: A Sample REST endpoint for your sample Web projects

leave a comment »

One last tool to mention.

While reading Web Development Recipes by Brian Hogan, I came across a neat little tool which I’m going to use for my next few example programs.

I wanted to learn Web front-end development – I’ve already done a little work here and there with REST, and didn’t feel like revisiting that just to get up and running.

There’s a very popular (and useful!) site which lists most of the major MV* component libraries, as well as sample programs, called TodoMVC. The way they get around that is to not use REST at all, and keep all the data on the client.

But that doesn’t seem very satisfying as a solution either – one thing you want to evaluate is how these frameworks handle the same kind of data and use cases which you’d actually use in production, just at a smaller scale.

QEDServer solves that problem pretty neatly. Written as a self contained REST server, QEDServer provides a sample database, REST endpoints, and a /public directory which you can use to model possible solutions to the common Thin Server problem.

Neat little tool.

There are, of course, some pretty serious limitations. The REST API isn’t what I’d design – there’s no capacity, for instance, to get all records in a table, or choose a range. Everything’s paged at 10 records per GET. Additionally, there’s no way to get aggregate information: i.e., total number of expected records in a query. But these are the same kinds of limitations that you might very well face in production, so I decided to just roll with it.

One last limitation, as well. Because it’s a Java process, I found it used up a fair bit of memory, and when running the server, plus an IDE, plus 2 (or even 3) browsers, my poor little 4GB memory Mac was really starting to strain under the load. But it was time to upgrade anyway, so I did so, and my shiny new Mac has no issues with it. But if you’re on a resource constrained system, you may want to drop back to the same route the TodoMVC guys took…

Written by jamesgdriscoll

December 28, 2013 at 11:30 AM

Posted in REST, tools

IDEs for the Modern Web

with 3 comments

As with the discussion of Thin Server, if you’ve already got a favorite IDE for the Modern Web, feel free to skip forward. It’s unlikely that I’m going to change your mind.

A programmer without tools is a sad little thing. Reduced to coding in vi like some sort of caveman, or worse, edlin, like an animal.

So, first order of business was to do a quick survey of what kind of IDEs folks were using for modern Web Development.

If you’re a Java programmer, you probably use Eclipse, IntelliJ IDEA, or even Netbeans (which is actually my favorite for small projects). And while these can tackle the job of web programming, I wanted something purpose built for the task – I didn’t intend to do any Java work as part of this process, and so there was no need for any of the nice features those tools give you.

I wanted to pick a few different tools with different workflow characteristics, and I didn’t want to spend any money up front, though I’d be happy to license whichever I liked, as long as it was in the sub-$100 range. Quickly, I was able to narrow down the field to three, and at this point, I’ve tried all of them for about two weeks each.

The Three Candidates

  • WebStorm, a “does everything” IDE from JetBrains
  • Sublime Text 2, a visual editor which relies more on outside processes than built in functions
  • Brackets, an editor built entirely in JavaScript, extensible in JavaScript as well

Brackets

I picked Brackets first, because it was entirely free and open source – meaning, in part, that I’d never have to pay for it even if I liked it. Also, Brackets had also gotten a lot of positive buzz at the HTML5 Dev Conference, and I certainly wanted to see what the fuss was about.

Brackets is currently under very active development, and was at Sprint 31 at the time I tried it. (It’s already in Sprint 34 at the time of this writing.)

At first, I have to say I really quite liked it. It’s surprisingly small and fast, and since it’s extensible in JavaScript, there are any number of user written plugin which you can adopt into your workflow. In particular, I immediately wanted to add code folding, as well as jshint and htmlhint support, as well as additional themes. There was even a code completion plugin (and although it was pretty primitive, without inferences, it generally got the job done).

With these plugins installed, Brackets was a rather useful lightweight IDE, with active syntax highlighting, code folding, as well a color theme I could actually read.

But… it’s obvious that Brackets is still in it’s early days. The plugins would sometimes interfere with each other, and bulk editing operations weren’t really fully there yet.

Additionally, Brackets offered integration with Chrome for debugging – and when it worked, it was wonderful – but it would often get wedged, and require a restart of Chrome and Brackets to resume.

So while I’ll definitely be watching Brackets closely over the next year, it’s not for me at this time.

WebStorm

Reportedly an outgrowth of IDEA’s popular PHPStorm tool, WebStorm will be familiar to anyone who’s done work in IntelliJ IDEA. The interface is pretty much the same, the inspectors are the same, and so on. So, if you love working in IntelliJ, then the odds are good that you’ll love working in WebStorm.

I have to admit that I never really loved working in IntelliJ. While there are an endless number of knobs you can fiddle with to get your desired behavior, it’s always been a common problem for me that I’ll end up burning 30 minutes of precious coding time trying to find the right option.

As one example, an errant right click in my HTML document resulted in the HTML inspector displaying an alarming number of (incorrect) errors in my code. Apparently, I’d accidentally changed how the inspector operated. You’d think that that wouldn’t take long to find and correct… but as usual, it was about 20 minutes of looking.

WebStorm also offered a builtin build system, which again is very similar to the one that’s found in IntelliJ. While this was handy when I was just starting out, it began to chafe a bit when I tried integrating external build tools – which, I was reminded, was one of the things that frustrated me about IntelliJ as well.

I did run into one significant bug. Like Brackets, WebStorm integrates with Chrome to do debugging, via a Chrome plugin. The plugin allows you to use the very nice IDEA debugging interface, and never need to leave your IDE.

In principle, this is awesome. In practice, not so much. Like Brackets, this communication could easily become wedged, and require a restart to resume. In the end, I ended up just using Chrome’s most excellent debug tools right in the browser, and you know what? Staying inside the IDE is overrated. I lost almost nothing (except frustration) by having to swap back and forth between the two toolsets.

In the end, I certainly liked WebStorm more than Brackets – it was stable and almost bug free. But IDEA’s insistence on controlling the entire build infrastructure led me to consider using what was for me a surprising choice – Sublime Text, an editor which was almost the polar opposite of the IDEs I’ve been using for the last 10 years.

Sublime Text

In the third stage of my project (which I’ll detail at length in a later post), I moved to using a Yeoman setup with Grunt (a make-like task runner). Rather than learn how to integrate WebStorm into the build system that Yeoman sets up for you as part of installation, I opted to try a real departure from the classic “everything in the tool” IDE.

Sublime Text is a text editor, not an IDE, and I have to admit to a serious bias against using text editors for development. I was slow to move off of them (I was a vi guy for the longest time) when IDEs first came out, but I just couldn’t imagine giving up inline error highlighting. Because, when you think about it, what’s an IDE for? Build, debug, error highlight, edit. Build was being handled (quite well) by grunt. Debug was already being done in browser… and Sublime Text is truly a wonderful text editor.

Sublime Text won me over. Want to edit every matching phrase in a file? It’s only a couple of keystrokes (with no annoying GUI). In fact, you could say that about almost everything in Sublime – no annoying GUI. Nothing that gets in your way.

And inline errors? Grunt handles that. There’s a pretty simple configuration that will run all your files through jshint/jslint and htmlhint on every save, as well as automatically running your tests, all running as a background watch process. Just watch the terminal output in a separate window.

So much to my surprise, I’ll probably be shelling out for a Sublime license. Not the outcome I was expecting.

YMMV

Of course, that might not be your conclusion. If you like the idea of extending your IDE’s behavior in JavaScript, then Brackets is something you should check out. I’ve got every confidence that they’ll work out their bugs.

And if you like IntelliJ, definitely try out WebStorm, you’ll almost certainly like that too.

Next, let’s look at one last tool, then I can start to code.

Written by jamesgdriscoll

December 22, 2013 at 10:26 AM

Posted in JavaScript, tools