Jim Driscoll's Blog

Notes on Technology and the Web

Archive for the ‘REST’ Category

Thin Server the Hard Way (Model layer)

leave a comment »

This post is part of a series examining the Modern Web. Currently, I’m trying to assess pain points in creating a Single Page application, and to do that, I’ve created a simple application which does inventory management. You can find the (somewhat sloppy) code on Github, if you want to follow along.

After my previous post looking at routing, today we’ll examine how to handle the model layer of our single page app.

Architecture

The first step on something like this is to decide on an initial architecture. QEDServer has a fairly simple REST API for fetching and updating records in the database – the only mildly tricky part is to format and pass it parameters. So a fairly lightweight wrapper around the jQuery AJAX API is probably all that’s required. Let’s model what how it’d be used, in this case for fetching a page of results containing the names of Categories for our products:

        qedModel.getCategories({
            page: currentPage,
            search: search,
            error: onFetchError,
            success: onFetchSuccess,
            complete: onFetchAlways
        });

Implementation

The REST API we’re using already has a paging facility (you pass it a URL param of ?page=number) as well as search (passing a URL param of ?q=searchterm), so the JavaScript API wrapper we need to develop only needs to smartly handle parameter building for the URL, which is a pretty simple task. We’ll also want to have success, error and complete callbacks, which are handled by the jQuery AJAX API already. So, the finished code is really pretty simple:

        getCategories: function getCategories(params) {
            var error = params.error || null;
            var success = params.success || null;
            var complete = params.complete || null;
            var page = params.page || 1;
            var search = params.search;
            var endpoint = '/categories.json?';
            if (!!search) {
                endpoint = endpoint + '&q=' + search;
            }
            endpoint = endpoint + '&page=' + page;
            $.ajax({
                type: 'GET',
                url: endpoint,
                success: success,
                error: error,
                complete: complete,
                dataType: 'json'
            });
        }

A thin wrapper around the AJAX API, which simply builds a URL and returns the results in the supplied callbacks.

Much like building a front controller, this was trivially easy, but it didn’t have to be – there’s other features we could add in here, the most important of which are cacheing and pagination. Rather than always using the server to handle pagination and fetching values from the server every time, it would be far, far better to prefetch large chunks of data from the server, and let the client model layer handle the caching. But because of the design of the QEDServer REST API, which only allows fetching of 10 records at a time, that would have been counter productive in this case. And cacheing server data on the client opens up the problem of currency – making sure that the client data and server data are in sync. This implementation mostly just hand-waves that away.

Pain Points

In my simple implementation, there really were no pain points, but as I mentioned, I’d hope that any client side Model layer would handle cacheing and pagination as well as offer some hints on how to implement data synchronization.

So, with the Model and Front Controller out of the way, it’s time to move into the View Layer, which presented far more problems. But first, we’ll detour into what capabilities Bootstrap provides.

Written by jamesgdriscoll

January 25, 2014 at 11:44 AM

Posted in ajax, HTML5, REST, web

Thin Server the Hard Way (Basic Architecture)

leave a comment »

As I mentioned in my previous post, I’m looking to create a basic Thin Server Architecture without any particular framework, mostly to see what the pain points are. Knowing what problems these MV* frameworks are trying to solve is going to be critical in further evaluation.

So, with that as the goal, I created a simple Thin Server front end around the REST endpoints provided by QEDServer.

I’ve set up a Github repository to hold all the Thin Server clients I’m writing, and made this project a subdirectory in the repo. To use this subdirectory, download QEDServer, clone the Github repo, and symlink QEDServer public directory to point to the subdirectory. I do recommend you try this out if you’re going to read further, since it’ll show clearly what kinds of nifty responsive layout options you get from Bootstrap, if nothing else.

The app is really just two monolithic blocks – a rather large HTML file, and a very large JavaScript file. There are also two other files – a load indicator gif (generated from a lovely website devoted to that purpose), as well as a tiny css file to augment the base Bootstrap setup.

As a side note before we go any further, I do feel the need to mention that I’m not particularly proud of this code – it served it’s purpose, and it’s mostly bug free, but as I continued on, I started a number of refactorings which I never quite finished. For that reason, I almost didn’t release it – but in the end, I decided it was worthwhile as a launch point for a discussion.

So, on the to code.

HTML file

The HTML file (about 250 lines) can be thought of being divided into about 4 separate sections. I’ll outline them in summary, and then go into detail in a later post. For added maintainability, I went with an Unobtrusive JavaScript approach, meaning that there’s no JavaScript in the HTML at all, just script tags which act on the HTML once loaded.

Header info

The header info contains the loading information for all the base JavaScript and CSS. Since I wanted to keep this simple, I just used CDN loaded versions of all the libraries. Note that some additional shims are required to get things running for IE8, but that’s not something I was interested in testing out – like most people, I can’t wait for that browser to die in a fire.

Nav bar

This section contains the Navigation and branding that’s used throughout the application. The markup is pretty simple, and really shows off what kinds of things that Bootstrap can do. That’s the topic for a whole separate post.

Swappable Divs

Underneath the navigation section are all the different “pages” which will be visible to our users. Since only one will be visible at any given time, I opted to place all of them in separate divs, and then switch between them via hide/show. This is a very performant way to do things, but has two drawbacks – you take a hit on initial load, and for large apps, it will become utterly unwieldy.

Cloning area

Similarly, I have a separate cloning area for sections of HTML code which I’ll be copying and placing into the app, completely wrapped in a hidden div. Again, this is a simple, fairly performant way to do things, at a cost in initial load and maintainability.

JavaScript file

At almost 800 lines, this ended up being a pretty big lump of code. I opted to use the module pattern (with import and export) to organize things. I further divided up the code into several modules, to do a proper division of responsibilities – and since it was going to be evaluating MV* frameworks, it seemed to make sense to use a similar structure.

View layer

The first module just controls the view layer (in conjunction with the HTML, meaning that that doesn’t fit properly into an MVC pattern). A variety of functions handle displaying the different pages via show/hide on the divs in the page, as well as event handlers, an alert messaging system, and other action functions for adding and deleting products. I refactored this code a number of times (and actually stopped in the middle of my last refactoring), but never really got the kind of cleanliness that I wanted. In particular, I really wanted to separate out the binding between the rendered HTML and the functions that acted on it that I felt would be necessary for maintainable code. Of the three layers I tackled, this one was the one that left me the most dissatisfied.

Front Controller / Router

By using onhashchange as a listener for the routing code, it was pretty easy to come up with a basic routing mechanism. With the exception of some annoying boilerplate that I had to write for parameter handling, this proved to be some of the easiest code to write for the whole app. While complexity would grow linearly with the complexity of the app, it doesn’t look like this is really something that other MV* frameworks are going to address – but maybe they’ll surprise me. It would be nice to have something that handled history.pushState automatically, but since QEDServer doesn’t really handle that, I didn’t take a stab at it.

Model layer

A thin ajax layer across the REST interface of QEDServer was, like the router code, rather easy to write (once jQuery’s excellent ajax support was added in). Since it’s only a thin layer, it does expose the underlying data structure of the REST server to the View layer, but you could just as easily change it to massage the returned data to any desired format, in the event that the returned data changed it’s structure. While there was a tiny bit of boilerplate, this doesn’t seem to be that big a deal. Adding in CORS support looks easy, but I didn’t try it, since QEDServer didn’t support it. Like the router code, this looks like complexity will grow linearly with the complexity of the REST API being modeled, and it’s not clear how a framework could really improve on things. Again, I’m hoping somebody will surprise and delight me by proving me wrong.

Initialization and utility methods

The end blocks contain a variety of utility and initialization methods. There was a fair bit of code in here that really felt like reinventing the wheel, and in the end, I even wrote a really primitive templating solution – I’ve no doubt that this is tackled in any number of other places (and more recent efforts have certainly proved that out).

Pain points

I was more than a little surprised by what I found. The two bits I thought would be hard, routing and Ajax retrieval from REST, were handled pretty easily, while the part I thought I knew how to do, view manipulation, proved to be a pretty tough nut to crack.

  • Templating is a pretty large need.
    • Balancing initial load time with performant changes
    • Better code organization for large applications
  • A simple setup for routing would be convenient, but not especially critical
  • Removing some boilerplate for REST endpoint wrapping would be a nice-to-have
  • View state management looks to be critical.
    • Binding both data values and entire views to HTML tags
    • Binding event handlers
    • Managing view transitions and lifecycle

I’ve already ported this application to the first MV* framework that I looked it, Backbone, but it’s probably worthwhile going through a few more items on this first effort first before diving in. Look for that next.

Written by jamesgdriscoll

January 22, 2014 at 6:41 PM

Posted in JavaScript, REST

Thin Server the Hard Way (Getting Set Up)

leave a comment »

After the diversion I just had with Java 8, time to get back to describing some features of the Modern Web.

There are any number of MV* client side frameworks out there. I’ve already mentioned TodoMVC, where you can find an extensive list, as well as sample code for each.

But before you evaluate tools, it always pays to know what pain points you’re trying to solve. So, with that in mind, I decided my first task would be to do a Thin Server application the hard way, using only jQuery to manipulate the DOM and handle the data.

Now, I didn’t want to not learn anything new during this exercise, so to keep in interesting, I added in one new dependency, Bootstrap.

So, here’s the recipe list I started with to develop the application:

Tools in Use

  • Brackets, which I wanted to evaluate as an IDE
  • QEDServer, which provides default REST endpoints, as well as a public directory to serve files
  • jQuery, because why on earth would you use the built in DOM APIs if you didn’t have to?
  • Bootstrap, to make the site look pretty

So, not quite starting at the bare metal, but close enough.

jQuery

I’m not going to go over jQuery at all in describing my solution. Even though it was the first time I used it for writing anything more than a few lines, I think it’s pretty likely that anyone reading this will almost certainly know it. And if you don’t… There are any number of books out there on it, but before you invest in a book, just check out the API Documentation. It’s a small API, and quite restrained in what it’s trying to accomplish. Essentially, it’s for DOM manipulation, AJAX requests, and handful of utility functions to make those two functions easier. If you already know much about the browser DOM, it’ll take you a weekend to get up to speed. If you don’t know the browser DOM… then that’s the problem you need to solve, not learning jQuery.

Bootstrap

There are a number of “website boilerplate” projects out there. Besides Bootstrap, Foundation and HTML5 Boilerplate seem to be the most popular – but here’s a list of 10, if you’re interested.

I picked Bootstrap simply because it’s the most popular one right now, and they handle a bunch of things out of the box that I thought would be quite tricky for me to do on my own.

Bootstrap provides you with a set of CSS and JavaScript libraries that you can use as a starting point for your pages, and by default, their look and feel is both clean and modern. Additionally, adding simple effects (like collapsing navigation bars, popups, and closable alert boxes), can be done almost entirely declaratively. Like jQuery, I found it to be incredibly simple to use, and rock solid – I only found one bug during the process, and that was certainly because I was doing something that was an unexpected use of the product.

Also, because it’s the most popular framework, I was able to find lots of information on the web about various setup questions, including the fix for the bug I found.

Bootstrap’s only requirement was that jQuery be included in the same page, so that also worked out pretty well for me.

Conclusions

I’d certainly recommend both libraries as something that a beginner should start out with – they’re both self contained and easy to learn.

Next up, I’ll out line the architecture that I used to create my first Thin Server application

Written by jamesgdriscoll

January 20, 2014 at 3:02 PM

QEDServer: A Sample REST endpoint for your sample Web projects

leave a comment »

One last tool to mention.

While reading Web Development Recipes by Brian Hogan, I came across a neat little tool which I’m going to use for my next few example programs.

I wanted to learn Web front-end development – I’ve already done a little work here and there with REST, and didn’t feel like revisiting that just to get up and running.

There’s a very popular (and useful!) site which lists most of the major MV* component libraries, as well as sample programs, called TodoMVC. The way they get around that is to not use REST at all, and keep all the data on the client.

But that doesn’t seem very satisfying as a solution either – one thing you want to evaluate is how these frameworks handle the same kind of data and use cases which you’d actually use in production, just at a smaller scale.

QEDServer solves that problem pretty neatly. Written as a self contained REST server, QEDServer provides a sample database, REST endpoints, and a /public directory which you can use to model possible solutions to the common Thin Server problem.

Neat little tool.

There are, of course, some pretty serious limitations. The REST API isn’t what I’d design – there’s no capacity, for instance, to get all records in a table, or choose a range. Everything’s paged at 10 records per GET. Additionally, there’s no way to get aggregate information: i.e., total number of expected records in a query. But these are the same kinds of limitations that you might very well face in production, so I decided to just roll with it.

One last limitation, as well. Because it’s a Java process, I found it used up a fair bit of memory, and when running the server, plus an IDE, plus 2 (or even 3) browsers, my poor little 4GB memory Mac was really starting to strain under the load. But it was time to upgrade anyway, so I did so, and my shiny new Mac has no issues with it. But if you’re on a resource constrained system, you may want to drop back to the same route the TodoMVC guys took…

Written by jamesgdriscoll

December 28, 2013 at 11:30 AM

Posted in REST, tools

Thin Server Architecture

leave a comment »

As I mentioned previously, I’m looking at the new ways of developing Web Applications that have turned up in the last few years. If you already know about Thin Server Architecture, feel free to skip ahead.

Single Page Applications were already the Next Big Thing back in the mid-2000’s – serve a single page for your web application, then as much as possible (in practice, almost always) simply serve deltas back to the user to avoid page reloading. The advantage you gain from doing this is a much more snappy use experience, with a page that starts to feel a lot like a native app. However, for many (i.e. most) of these initial SPAs, much of the page rendering logic still resided on the server.

Thin Server Architecture takes this a step further. By moving page rendering logic onto the client, you gain additional advantages – using REST means that you can take advantage of proxied requests. Also implicit in this design is that you’re moving state management to the client as well. Moving state management and page generation to the client can radically reduce server load, which immediately gives significant advantages for scalability. Remember that, in total, your users have way more CPU than you do.

Essentially, what you’ve done is keep only database-centric tasks on the server. Authentication (login identity), Authorization (data view permissions) and Validation are the three most commonly cited. The server becomes a thin shell on top of the database. Even validation is done client side, with server side validation done primarily to protect data integrity, not as an otherwise critical part of the control flow.

Is that an oversimplification? Certainly. Just as you can move critical business logic into your database in stored procedures, you can do the same thing with Thin Server. And for some things, like cascading changes to maintain data integrity, this would make real sense.

In addition to page view logic moving to the client, many newish HTML5 APIs also provide additional capabilities that you can exploit. You can store data on the client (or just cache it) via localStorage. You can use history.pushState to update the browser’s URL to add bookmarkability (real URLs, not just anchors! But of course, there’s no support for this in IE8). This works rather well, up to a point.

Where this becomes problematic is when users do deep linking of those URLs that were generated from history.pushState. While it’s true that clients, in total, have more CPU than you, for many of them (especially those on mobile devices), the amount of CPU they can bring to bear at any given moment is limited. The last thing you want to do is to do all this work to get a server that can shovel pages out the door with sub 50ms response times, only to have the client’s phone browser spend 3s rendering it, as it plays through a bunch of JavaScript to build up the correct state.

The obvious solution is to do an initial page render on the server, serve that, and do all further renderings on the client. Additionally, while you’re rendering the client web page, you can seed all your JavaScript functions with the JSON state that they’ll need to spin up, enabling everything to get up and running after the initial load. This hybrid approach is certainly going to be considered best practice (if not already), and you can already see folks referring to it as such – though it does seem as though it might be easiest to do with a framework that spanned both client and server using the same language and libraries… Which is a strong argument for node.js.

Certainly something to watch.

Another buzz phrase that’s popular right now is API First Design. The rise in popularity of mobile devices has led to a conflicting need – there are now effectively two web platforms to design for, a Mobile Web, with severe memory, bandwidth and resource constraints, and a Desktop Web, where those restrictions only apply modestly, if at all. (This is known as the “web first” vs. “mobile first” argument.) The API First design movement says that you should design your REST API first, around the information you actually wish to convey and manipulate. Once you have that, you can design for anything you want – desktop web, mobile web, an application, or even open things up to a third party. Frankly, the idea makes a lot of sense, and I’m reminded of the reason why Google Plus bellyflopped on it’s first outing. Lack of a defined API can be crippling to a product in surprising ways.

Hopefully, this was helpful in understanding this new Web world we’re living in.

Next, let’s look at what folks are using for IDEs these days.

Written by jamesgdriscoll

December 20, 2013 at 10:58 PM

Posted in JavaScript, REST, web