Jim Driscoll's Blog

Notes on Technology and the Web

Archive for December 2013

QEDServer: A Sample REST endpoint for your sample Web projects

leave a comment »

One last tool to mention.

While reading Web Development Recipes by Brian Hogan, I came across a neat little tool which I’m going to use for my next few example programs.

I wanted to learn Web front-end development – I’ve already done a little work here and there with REST, and didn’t feel like revisiting that just to get up and running.

There’s a very popular (and useful!) site which lists most of the major MV* component libraries, as well as sample programs, called TodoMVC. The way they get around that is to not use REST at all, and keep all the data on the client.

But that doesn’t seem very satisfying as a solution either – one thing you want to evaluate is how these frameworks handle the same kind of data and use cases which you’d actually use in production, just at a smaller scale.

QEDServer solves that problem pretty neatly. Written as a self contained REST server, QEDServer provides a sample database, REST endpoints, and a /public directory which you can use to model possible solutions to the common Thin Server problem.

Neat little tool.

There are, of course, some pretty serious limitations. The REST API isn’t what I’d design – there’s no capacity, for instance, to get all records in a table, or choose a range. Everything’s paged at 10 records per GET. Additionally, there’s no way to get aggregate information: i.e., total number of expected records in a query. But these are the same kinds of limitations that you might very well face in production, so I decided to just roll with it.

One last limitation, as well. Because it’s a Java process, I found it used up a fair bit of memory, and when running the server, plus an IDE, plus 2 (or even 3) browsers, my poor little 4GB memory Mac was really starting to strain under the load. But it was time to upgrade anyway, so I did so, and my shiny new Mac has no issues with it. But if you’re on a resource constrained system, you may want to drop back to the same route the TodoMVC guys took…

Written by jamesgdriscoll

December 28, 2013 at 11:30 AM

Posted in REST, tools

IDEs for the Modern Web

with 3 comments

As with the discussion of Thin Server, if you’ve already got a favorite IDE for the Modern Web, feel free to skip forward. It’s unlikely that I’m going to change your mind.

A programmer without tools is a sad little thing. Reduced to coding in vi like some sort of caveman, or worse, edlin, like an animal.

So, first order of business was to do a quick survey of what kind of IDEs folks were using for modern Web Development.

If you’re a Java programmer, you probably use Eclipse, IntelliJ IDEA, or even Netbeans (which is actually my favorite for small projects). And while these can tackle the job of web programming, I wanted something purpose built for the task – I didn’t intend to do any Java work as part of this process, and so there was no need for any of the nice features those tools give you.

I wanted to pick a few different tools with different workflow characteristics, and I didn’t want to spend any money up front, though I’d be happy to license whichever I liked, as long as it was in the sub-$100 range. Quickly, I was able to narrow down the field to three, and at this point, I’ve tried all of them for about two weeks each.

The Three Candidates

  • WebStorm, a “does everything” IDE from JetBrains
  • Sublime Text 2, a visual editor which relies more on outside processes than built in functions
  • Brackets, an editor built entirely in JavaScript, extensible in JavaScript as well

Brackets

I picked Brackets first, because it was entirely free and open source – meaning, in part, that I’d never have to pay for it even if I liked it. Also, Brackets had also gotten a lot of positive buzz at the HTML5 Dev Conference, and I certainly wanted to see what the fuss was about.

Brackets is currently under very active development, and was at Sprint 31 at the time I tried it. (It’s already in Sprint 34 at the time of this writing.)

At first, I have to say I really quite liked it. It’s surprisingly small and fast, and since it’s extensible in JavaScript, there are any number of user written plugin which you can adopt into your workflow. In particular, I immediately wanted to add code folding, as well as jshint and htmlhint support, as well as additional themes. There was even a code completion plugin (and although it was pretty primitive, without inferences, it generally got the job done).

With these plugins installed, Brackets was a rather useful lightweight IDE, with active syntax highlighting, code folding, as well a color theme I could actually read.

But… it’s obvious that Brackets is still in it’s early days. The plugins would sometimes interfere with each other, and bulk editing operations weren’t really fully there yet.

Additionally, Brackets offered integration with Chrome for debugging – and when it worked, it was wonderful – but it would often get wedged, and require a restart of Chrome and Brackets to resume.

So while I’ll definitely be watching Brackets closely over the next year, it’s not for me at this time.

WebStorm

Reportedly an outgrowth of IDEA’s popular PHPStorm tool, WebStorm will be familiar to anyone who’s done work in IntelliJ IDEA. The interface is pretty much the same, the inspectors are the same, and so on. So, if you love working in IntelliJ, then the odds are good that you’ll love working in WebStorm.

I have to admit that I never really loved working in IntelliJ. While there are an endless number of knobs you can fiddle with to get your desired behavior, it’s always been a common problem for me that I’ll end up burning 30 minutes of precious coding time trying to find the right option.

As one example, an errant right click in my HTML document resulted in the HTML inspector displaying an alarming number of (incorrect) errors in my code. Apparently, I’d accidentally changed how the inspector operated. You’d think that that wouldn’t take long to find and correct… but as usual, it was about 20 minutes of looking.

WebStorm also offered a builtin build system, which again is very similar to the one that’s found in IntelliJ. While this was handy when I was just starting out, it began to chafe a bit when I tried integrating external build tools – which, I was reminded, was one of the things that frustrated me about IntelliJ as well.

I did run into one significant bug. Like Brackets, WebStorm integrates with Chrome to do debugging, via a Chrome plugin. The plugin allows you to use the very nice IDEA debugging interface, and never need to leave your IDE.

In principle, this is awesome. In practice, not so much. Like Brackets, this communication could easily become wedged, and require a restart to resume. In the end, I ended up just using Chrome’s most excellent debug tools right in the browser, and you know what? Staying inside the IDE is overrated. I lost almost nothing (except frustration) by having to swap back and forth between the two toolsets.

In the end, I certainly liked WebStorm more than Brackets – it was stable and almost bug free. But IDEA’s insistence on controlling the entire build infrastructure led me to consider using what was for me a surprising choice – Sublime Text, an editor which was almost the polar opposite of the IDEs I’ve been using for the last 10 years.

Sublime Text

In the third stage of my project (which I’ll detail at length in a later post), I moved to using a Yeoman setup with Grunt (a make-like task runner). Rather than learn how to integrate WebStorm into the build system that Yeoman sets up for you as part of installation, I opted to try a real departure from the classic “everything in the tool” IDE.

Sublime Text is a text editor, not an IDE, and I have to admit to a serious bias against using text editors for development. I was slow to move off of them (I was a vi guy for the longest time) when IDEs first came out, but I just couldn’t imagine giving up inline error highlighting. Because, when you think about it, what’s an IDE for? Build, debug, error highlight, edit. Build was being handled (quite well) by grunt. Debug was already being done in browser… and Sublime Text is truly a wonderful text editor.

Sublime Text won me over. Want to edit every matching phrase in a file? It’s only a couple of keystrokes (with no annoying GUI). In fact, you could say that about almost everything in Sublime – no annoying GUI. Nothing that gets in your way.

And inline errors? Grunt handles that. There’s a pretty simple configuration that will run all your files through jshint/jslint and htmlhint on every save, as well as automatically running your tests, all running as a background watch process. Just watch the terminal output in a separate window.

So much to my surprise, I’ll probably be shelling out for a Sublime license. Not the outcome I was expecting.

YMMV

Of course, that might not be your conclusion. If you like the idea of extending your IDE’s behavior in JavaScript, then Brackets is something you should check out. I’ve got every confidence that they’ll work out their bugs.

And if you like IntelliJ, definitely try out WebStorm, you’ll almost certainly like that too.

Next, let’s look at one last tool, then I can start to code.

Written by jamesgdriscoll

December 22, 2013 at 10:26 AM

Posted in JavaScript, tools

Thin Server Architecture

leave a comment »

As I mentioned previously, I’m looking at the new ways of developing Web Applications that have turned up in the last few years. If you already know about Thin Server Architecture, feel free to skip ahead.

Single Page Applications were already the Next Big Thing back in the mid-2000’s – serve a single page for your web application, then as much as possible (in practice, almost always) simply serve deltas back to the user to avoid page reloading. The advantage you gain from doing this is a much more snappy use experience, with a page that starts to feel a lot like a native app. However, for many (i.e. most) of these initial SPAs, much of the page rendering logic still resided on the server.

Thin Server Architecture takes this a step further. By moving page rendering logic onto the client, you gain additional advantages – using REST means that you can take advantage of proxied requests. Also implicit in this design is that you’re moving state management to the client as well. Moving state management and page generation to the client can radically reduce server load, which immediately gives significant advantages for scalability. Remember that, in total, your users have way more CPU than you do.

Essentially, what you’ve done is keep only database-centric tasks on the server. Authentication (login identity), Authorization (data view permissions) and Validation are the three most commonly cited. The server becomes a thin shell on top of the database. Even validation is done client side, with server side validation done primarily to protect data integrity, not as an otherwise critical part of the control flow.

Is that an oversimplification? Certainly. Just as you can move critical business logic into your database in stored procedures, you can do the same thing with Thin Server. And for some things, like cascading changes to maintain data integrity, this would make real sense.

In addition to page view logic moving to the client, many newish HTML5 APIs also provide additional capabilities that you can exploit. You can store data on the client (or just cache it) via localStorage. You can use history.pushState to update the browser’s URL to add bookmarkability (real URLs, not just anchors! But of course, there’s no support for this in IE8). This works rather well, up to a point.

Where this becomes problematic is when users do deep linking of those URLs that were generated from history.pushState. While it’s true that clients, in total, have more CPU than you, for many of them (especially those on mobile devices), the amount of CPU they can bring to bear at any given moment is limited. The last thing you want to do is to do all this work to get a server that can shovel pages out the door with sub 50ms response times, only to have the client’s phone browser spend 3s rendering it, as it plays through a bunch of JavaScript to build up the correct state.

The obvious solution is to do an initial page render on the server, serve that, and do all further renderings on the client. Additionally, while you’re rendering the client web page, you can seed all your JavaScript functions with the JSON state that they’ll need to spin up, enabling everything to get up and running after the initial load. This hybrid approach is certainly going to be considered best practice (if not already), and you can already see folks referring to it as such – though it does seem as though it might be easiest to do with a framework that spanned both client and server using the same language and libraries… Which is a strong argument for node.js.

Certainly something to watch.

Another buzz phrase that’s popular right now is API First Design. The rise in popularity of mobile devices has led to a conflicting need – there are now effectively two web platforms to design for, a Mobile Web, with severe memory, bandwidth and resource constraints, and a Desktop Web, where those restrictions only apply modestly, if at all. (This is known as the “web first” vs. “mobile first” argument.) The API First design movement says that you should design your REST API first, around the information you actually wish to convey and manipulate. Once you have that, you can design for anything you want – desktop web, mobile web, an application, or even open things up to a third party. Frankly, the idea makes a lot of sense, and I’m reminded of the reason why Google Plus bellyflopped on it’s first outing. Lack of a defined API can be crippling to a product in surprising ways.

Hopefully, this was helpful in understanding this new Web world we’re living in.

Next, let’s look at what folks are using for IDEs these days.

Written by jamesgdriscoll

December 20, 2013 at 10:58 PM

Posted in JavaScript, REST, web

A Visit to the Modern Web

leave a comment »

My previous post covered how I came to be a bit of a time traveller to the world of modern Web technology.

This time, let’s talk more about the world as I left it.

The Web Development World in 2008

Single page applications (SPAs) were all the rage. The various JavaScript libraries were duking it out. GWT was compiling Java into JavaScript. There was quite a bit of tension between the SOAP and REST camps, and JSON was still kind of a new thing (IE added support, finally, in 2008). Ruby on Rails had reached Maximum Hype, and subsided, and everyone was writing their own Rails-like framework in their own favorite language.

On the Java side, there was still a notable tussle between varying page definition frameworks, like Tapestry, Wicket and JSF – all server side technologies, though increasingly, support for SPAs was being rapidly added. IE6 was still a (rapidly vanishing) thing, and support for things like WebSockets was forecast to not really be fully adopted for years. For client side work, libraries were primarily downloaded off a website and incorporated into existing build frameworks (maven, for instance, if you were a Java shop), and testing was done with tools like Selenium. Relational databases were pretty much the only game in town, though some bleeding edge early adopters were using other technologies (the term NoSQL wasn’t even in common use until 2009).

Contrast that with what I’m seeing now… Warning: Opinions ahead.

The Web Development World of late 2013

JSON Won

As a complete non-surprise, JSON appears to have won out over XML as a data-interchange format. It’s just so much simpler for the simple case, and with native support in pretty much every available browser, I don’t see XML used anywhere client-side except for the DOM. From my recent observations, it seems to be rapidly bleeding into the server side as well. And speaking of non-surprises…

REST Won

As a Web partisan, I can’t say I’m especially surprised by this one. Yes, I know that (insert name of shop here) is actively using SOAP and loves it, just as there were always shops who swore by the control that CORBA gave you. But really, for mainstream adoption, REST won, and won big. This leads directly to the next point…

SPAs are pretty much all Thin Server

Most SPAs that I’ve seen described for startup are pretty much all based on the Thin Server architecture. Stateless REST calls, transmitting JSON to the client. The advantages for scaling are simply so large, and the performance benefits that you can get from proxied requests is also so large, that it’s the default architecture that’s adopted by most new sites. As a (former) stateful framework guy, I’d argue there’s a place for that architecture too, but there’s no denying which way things are moving. But as a downside to this, you really need a new way to organize your JavaScript on the client. More (much more) on that shortly. (Also, if you’re not sure what Thin Server really is, I intend to go over that in a later post.)

jQuery won

While other libraries such as Dojo and ExtJS are certainly still around and in wide use, jQuery as the main mechanism for DOM manipulation is just ubiquitous. (Trust me, at the time, this wasn’t as obvious an outcome as it now seems.) The other frameworks that it competes with generally try to do a lot of other things in addition to DOM manipulation, and for the client side web, small wins. Which brings me to a second point – for the mobile web, even jQuery seems to be too large for some purposes, and an even newer jQuery compatible library, Zepto, appears to be ascendant. It’s both modular and much smaller than jQuery – but it gets that partially by ignoring IE (they only just added support for IE10, and earlier versions of IE are not supported, and apparently never will be). Which leads to one of my more surprising discoveries…

IE just doesn’t matter like it used to

It’s more than a little early to say: “ding-dong the witch is dead” – IE8 still has at least a 10% market share – but rather surprisingly, I’m seeing more and more libraries as I keep exploring that simply don’t bother to support it. And not just IE8, which is certainly on it’s way out the door, but IE in general seems to be something developers are increasingly caring less about. (Zepto, for instance, used to bluntly say they didn’t support IE at all, and only recently added IE10 support). And for some uses, that makes total sense – if you’re targeting mobile, the number of mobile IE browsers is microscopic. Sadly, Microsoft still can’t get it together well enough on the standards support, even in IE11, and you constantly run across the phase “works in all browsers except IE”. Maybe next year.

JavaScript is cool. Really cool.

It was while reading Crockford’s “JavaScript: the Good Parts” that I first really understood that JavaScript was actually a pretty nifty little language. But back then, I didn’t have very much company. Today, everyone is writing in JavaScript3. In fact, they prefer it – so much so, that…

Full Stack JavaScript is a thing.

You can write your client side code in JavaScript using an IDE written (and extended) in JavaScript, utilizing JavaScript libraries downloaded and managed by JavaScript, build with a JavaScript program scripted in JavaScript, minified, checked and tested (with a headless server!) via JavaScript, and deploy to a JavaScript server which visits a database which is extended in JavaScript. Crikey.

Preprocessors are a big deal.

For a language that’s getting so much love, there sure are a lot of projects to hide it. CoffeeScript seems to be the most popular by far, but besides that, there’s Dart, TypeScript, and dozens more. They add static typing, async processing, and various other language extensions.

It’s not just JavaScript that’s a target for preprocessors – CSS is also the target of preprocessors. The big three are Less, Sass, and Stylus. They add additional structural features to CSS, such as methods and mixins.

You want to use the Node.js ecosystem

Pretty much everything you’ll want to do to build and maintain a modern website is available to you via the node.js ecosystem. It’s package manager, npm, is so ubiquitous that it’s possible to argue that if you aren’t on the npm registry, your software may as well not exist. And though many of the open source tools on there aren’t more than a year or two old, they’re mostly of commercial quality. This is a big enough topic that it deserves a separate post.

All the cool kids are using a NoSQL database

The structure of a NoSQL database makes them much more suited to use in an on-demand cloud instance. Their general lack of transaction support doesn’t mean they’re a good fit for every task, but when you’ve got a massive amount of mostly static data to serve, they’re just the thing. (They’re also a great thing for searching really large datasets as well.) More about that in a future post as well, though I don’t yet feel comfortable talking at length on this topic yet.

There’s a JavaScript library for everything. In fact, there’s 10.

Want to using templating on your web client? Pick from any of a dozen (though there are certainly some that are the most popular). Need to manipulate arrays (or fix this to make sense)? You’re covered. A lot. Want to test your code? You’re spoiled for choices. If you want to do it in JavaScript, someone’s already written a library for you. And the odds are good that it’s small, fast and covers most corner cases already. Did I say we’re in the early adopter phase? We may have already crossed the chasm to early mainstream.


So, that’s where we are – a changed world of Web Development. Next, let’s talk about the architecture that’s becoming the default choice for large scale web apps – Thin Server.

Written by jamesgdriscoll

December 16, 2013 at 5:32 PM

Posted in JavaScript, web

Catching up with the Modern Web

leave a comment »

I’ve been doing work on various Web technologies since Clinton was President. I’ve been involved in Internet technologies since… before that. Throughout the late 90’s and most of the next decade, I worked on Web technologies in one fashion or another, either as an engineer or a manager. We worked on, and invented, any number of technologies whose names you’d recognize (though I usually had a peripheral role).

My last project for Sun, implementing the Ajax front end for JSF‘s reference implementation, was heavy JavaScript work. Because we didn’t want to have any dependencies on external libraries, it was done as close to the metal as you can get in DOM programming. (And since IE6 support was, IIRC, required, it was quite an adventure.)

But then, as Sun spiraled downward toward it’s eventual dissolution, I ended up leaving hard core Web technologies to work on solutions for a somewhat smaller (though still pretty large) set of customers – and though I still worked making Web based solutions, I began to focus primarily on more backend problems like DSLs and metaprogramming.

Gradually, I stopped paying attention.

Every now and then, something would peep through my blinders. Since I was paying attention to language developments, I heard about Node.js, CoffeeScript, and Dart (though I admit to not being exceptionally impressed by any of them on first hearing). I attended a number of talks by Douglas Crockford, and I had some vague notion that the Thin Server model was taking over. I heard, often through friends, that NoSQL was a thing now. There were offhand references to client-side MVC here and there, as well as more esoteric things like Hypermedia and HATEOAS.

As part of my job, I began working in JavaScript somewhat heavily again, mostly instrumenting a JavaScript component (CodeMirror, which I can’t say enough kind things about, nice stuff). Almost on a lark, I decided to attend the local HTML5 Developers Conference – one of the major advantages of working in Silicon Valley is that such things are readily available to you if you’re willing to take the time to seek them out.

The conference was extremely eye-opening. While I was napping, a whole new ecosystem had opened up around the world of Web Development. As is usual in technology, after a brief period of consolidation and extension of the latest greatest tech (server side Java), a whole new way of doing things was being born.

In the month since, I’ve immersed myself as much as possible in this new world, and I love what I’ve found.

If, like me, you haven’t been paying as much attention to new developments as you should, it’s time to start. Things have settled down a little bit, but are still in the early adopter phase of the adoption lifecycle — though I believe that they’re rapidly crossing the adoption chasm.

Don’t have a month or two to burn to learn all this new stuff? Stick around – I intend to report what I’ve found.

Written by jamesgdriscoll

December 14, 2013 at 9:47 AM

Posted in JavaScript, web

DSL Slides from GR8 Groovy Conference

leave a comment »

Has it really been a year since I posted here?  Time to catch up.

I presented on the topic of DSLs at the GR8 Conference in Minneapolis during July.

As always, you can find the slides up on Slideshare:

Turtle Graphics in Groovy featuring the Napili program that I’ve blogged about here.

Groovy in the Cloud was a talk about what I actually do for a living, it was a refreshing change to actually be able to discuss it.

While the GR8 Conference was interesting, what really caught my attention was a talk entitled Grails for Hipsters – it was a pretty wide survey of the current state of web development.

All I can say is, wow, have things changed. In only three years, pretty much the entire landscape of web development has completely transformed.

Partly inspired by that talk, I attended The HTML5 Developer’s Conference in San Francisco in October. While phrases like “blew my mind” are certainly overdone, my mind was utterly blown.

More on that in my next blog entry. After a couple years of DSL work, it’s time to revisit the Web.

Written by jamesgdriscoll

December 7, 2013 at 10:45 AM

Posted in DSL, Groovy