Saturday, January 24, 2015

Integration Testing a Restful Endpoint with Request and Jasmine and Node.JS

What is it for?

I am building a simple restful API using Express over MongoDB in order to support a mobile application.  Even though the endpoint was quite straightforward, I wanted to ensure it is properly tested.  In creating this test, I wanted to fulfill some simple objectives
  • Integration style test - completely independent of the implementation
  • Setup and Teardown of the database.
  • Fast, simple, as little complexity as possible

What Am I Testing?

The API is simple
../rules/clientId/ruleId (get) - Retrieve a specific rule for a specific client
../rules/clientId (get) - Retrieve all of the rules for a specific client
../rules (post) - Store a rule
../rules/clientId/ruleId (delete) - Delete a specific rule for a specific client

Why Use Jasmine? Why not use Frisby.js?

Ah tool choice, such a fraught area.  Jasmine is well established and  there is a great node port of Jasmine here https://github.com/mhevery/jasmine-node that is reasonably active.  All you Mocha folks and Karma fanatics, pls comment and tell me why framework X is better than Jasmine :)

Frisby.js seems to have a lot of search engine traction but It doesn't support setup/teardown (https://github.com/vlucas/frisby/issues/91) and seemed to force me into a certain way of doing things which I didn't like.

Enough already let's do it

The full gist is here if you want to look at it...
https://gist.github.com/mdausmann/759ad37fe4f9e77244da

Setup

To setup, I use beforeEach with the done option because I want to asynchronously move on to the test once the MongoDB call is finished.  To setup for the tests, I insert a single 'rule' document containing just ID information.
 
    beforeEach(function(done){
        MongoClient.connect("mongodb://" + config.mongo_db_address + ":" + config.mongo_db_port + "/" + config.mongo_db_name, function (err, db) {
            var collection = db.collection('rules');
            collection.insert({"clientId": "test","ruleId": "test1"}, function (err, result) {
                db.close();
                done();
            });
        });
    }); 

Teardown

To teardown, I issue a global delete for all rules belonging to the test client.  This is a bit crude but as in all integration style testing, I don't have the luxury of simply making tests transactional.  Note that again, I am using the Async form and using a done method to indicate that things can move on.

    afterEach(function(done) {
        MongoClient.connect("mongodb://" + config.mongo_db_address + ":" + config.mongo_db_port + "/" + config.mongo_db_name, function (err, db) {
            var collection = db.collection('rules');
            collection.remove({ "clientId": "test" }, function (err, result) {
                db.close();
                done();
            });
        });
    });


Update: I am now using uri: not url: and also using the json:true parameter in the request options.  this means I don't need to JSON.parse the bodyString like I was before.. much cleaner.

Testing the Get

Testing the getter method is simple.  I simply use the request component to retrieve the body string from the endpoint, use JSON.parse to convert it to an object littoral and then assert its values.

    it("should retrieve a specific rule", function(done) {
        request({
            uri: "http://localhost:3000/api/rules/test/test1",
            json: true
        }, function(error, response, body){
            expect(body.clientId).toEqual("test");
            expect(body.ruleId).toEqual("test1");
            done();
        });
    });

Testing the Post

For the Post, I really should check the addition to the DB but I cheat a bit by making a nested get call and verifying the return result (I use the same Kluge for delete).

    
    it("should store a rule", function(done) {
        request({
            method:'POST',
            uri: "http://localhost:3000/api/rules",
            json:{"clientId": "test","ruleId": "test2"}
        }, function(error, response, bodyPost){

            request({
                url: "http://localhost:3000/api/rules/test/test2",
                json:true
            }, function(error, response, body){
                expect(body.ruleId).toEqual("test2");
                done();
            });

        });
    });

Thats it, hope this helps.  I certainly thought this would be easier than it was and had dead ends with Frisby.js (no setup/teardown) and fiddling about with Request but this works.  All comments and suggestions accepted.

Michael









Thursday, December 25, 2014

The key problem that AngularJS solves (and React does not)

Software exists to solve problems for people.  Software frameworks exist to solve problems with developing software.  Reflecting on the hype and excitement around React, I thought it was a good time to think about the key problem solved by the AngularJS 1.x framework and contrast that with the React framework.

Web design and coding are different
HTML and CSS are fundamentally about visual layout, look and feel.  To be good at these, you need a sense of design and a lot of patience.  Coding data access, UI interaction, callbacks and form validation is a completely different discipline and there are not many people that are productive at both of these.

AngularJS solves this problem by providing a clean separation between the two disciplines.  A developer can take a completed HTML/CSS layout and 'tag it up' to make it interactive, a designer can refactor the visual layout of a page without messing about with the code.

React does not solve this problem at all and requires you to 'cut up' the HTML/CSS and emit it to the framework from the code. Inevitably, this means that to be productive at coding and refactoring your React solution, you need to be productive at both disciplines.

Not every developer is a Front End developer
I have introduced AngularJS to development shops with no specialist front end developers at all. We needed to use a consultant to build the initial framework and build some components for us but that having been done, my team of backend Java developers were able to crank out a lot of UI and were very productive at it.

This worked because we were able to use a stock standard CSS approach (we used Bootstrap) and the HTML could be safely cut and pasted from templates and already existing applications and let our coders do coding.  The separation of concerns inherent in AngularJS made this possible.

React code is spaghetti code

Just going through the tutorial, I found myself hunting around in the code, looking for where a particular piece of code or feature was.  It might just be unfamiliarity with the approach but the code just looked like spaghetti to me and I have serious concerns about productivity and the cost of refactoring.

MVC was created for a reason
The MVC pattern and all of it's variants work because it offers a clean separation between the layers, a separation of concerns.  React is smashing these things back together and this seems like a backwards step to me.

Monday, September 29, 2014

How to NOT Learn Coffee Script in 90 Minutes

So I am tired of looking at reading otherwise reasonable articles on angular.js or atom.io and finding myself bamboozled by this crazy ruby-esque nonsense that is coffeescript.  Time to learn.

My train ride home is about 90 minutes.  that should be enough.  Lets go

8:19 - start (http://jashkenas.github.io/coffee-script/)

Installation

Michaels-MacBook-Pro:learn_cs michael$ npm install -g coffee-script
npm http GET https://registry.npmjs.org/coffee-script
npm http 200 https://registry.npmjs.org/coffee-script
npm http GET https://registry.npmjs.org/mkdirp
npm http 200 https://registry.npmjs.org/mkdirp
/usr/local/share/npm/bin/coffee -> /usr/local/share/npm/lib/node_modules/coffee-script/bin/coffee
/usr/local/share/npm/bin/cake -> /usr/local/share/npm/lib/node_modules/coffee-script/bin/cake
coffee-script@1.7.1 /usr/local/share/npm/lib/node_modules/coffee-script
└── mkdirp@0.3.5

done.

console

done.. I guess I'm ready (8:27).  where to start...

8:33, I have written some functions in coffeescript and javascript (using a node console)  the parentheses feel more solid and familiar but the coffescript is more fluid to write and I am digging that I can invoke a function without parenthesis (cube 8.... cool)

Michaels-MacBook-Pro:learn_cs michael$ coffee
coffee> square = (x) -> x * x
[Function]
coffee> square(3)
9
coffee> cube = (x) -> square(x) * x
[Function]
coffee> cube 8
512
coffee>

8:39 oops, first problem, coffee command line can't do multiline functions (node commandline can do this, i guess because it knows about brackets n stuff...)

coffee> fill = (container, liquid = "coffee") ->
[Function]
coffee> "filling the #{container} with #{liquid}"
ReferenceError: container is not defined
  at repl:1:22.....

Ok so i can switch to multiline (thanks stack overflow) with ctrl-v (http://stackoverflow.com/questions/10491849/i-cant-write-multiline-codes-in-coffeescript-interactive-moderepl)

but it doesn't work!

coffee> fill = (container, liquid = "coffee") -> "filling the #{container} with #{liquid}.."
[Function]
coffee> fill "cup"
'filling the cup with coffee..'
------> shit = (container, liquid = "coffee") ->
....... "filling the #{container} with #{liquid}.."
ReferenceError: container is not defined

its now 8:55 that was seriously annoying and I have burned 17 minutes on it.  poor form.  switching to an IDE for mutiline i guess.

YAML style littorals are cool...

kids =
  brother:
    name: "max"
    age: 11
  sister:
    name: "ida"
    age: 9

console.log(kids)

Michaels-MacBook-Pro:learn_cs michael$ coffee test.cs
{ brother: { name: 'max', age: 11 },
  sister: { name: 'ida', age: 9 } }

But If I accidentally mess up the whitespace in the file.. my code breaks!

kids =
brother:
name: "max"
age: 11
sister:
name: "ida"
age: 9

console.log(kids)

Michaels-MacBook-Pro:learn_cs michael$ coffee test.cs
/Users/michael/Workspace/learn_cs/test.cs:2:9: error: unexpected newline
brother:
        ^

even single spaces can mess up the meaning of my littoral..

brother:
   name: "max"
  age: 11

Michaels-MacBook-Pro:learn_cs michael$ coffee test.cs
{ brother: { name: 'max' },
  age: 11,
  sister: { name: 'ida', age: 9 } }

9:10.. having serious concerns about cs now. It seems flakey

9:20.. have never had a problem with reserved words and I don't mind typing 'var'

9:32.. writing some code now, ifs and thens.

9:37.. running out of time.  I didn't get splats , or comprehend comprehensions.
 
Overall, I don't like it. 

Javascript is straightforward, the same as Java is straightforward, no surprises.  I don't mind ruby, i get that it is beautiful and all that.  I think coffeescript is trying to be ruby but doesn't quite make it.  It allso seems to try to please javascript developers but doesn't quite do that either. 

All in all it's a bit of a hot mess IMO and I will be continuing to avoid it, along with all the other things that compile to Javascript like GWT (Yuck) and Typescript (Yuck also).

Michael

Sunday, August 17, 2014

App Idea: Support your personal learning with revision reminders

Being myself in the throes of re-learning Java, Maven and IntelliJ (don't ask) I found this article resonated a lot with me...

http://archive.wired.com/geekdad/2012/01/everything-about-learning/

In particular, this statement..

 If you study, wait, and then study again, the longer the wait, the more you’ll have learned after this second study session.

This is 100% me.  I study, understand and then do nothing and all the good learning evaporates away.  I will come back to the topic much too late and have to start from scratch again.  I know I should set reminders and come back to material sooner but that never happens in practice.

Then it struck me that perhaps a simple app might assist here.  The pitch might go something like this..

"You know how when you're self learning, you first forget to revise the material and then you forget the learnings?

Well what we do is make it easy for you to post entries each time you learn a key concept and then play them back to you at exactly the right time to maximise your learning retention.

In fact Research has shown that if you study, wait and then study again, the longer the wait, the more you'll have learned."

It wouldn't be a hard app to build.  

Capturing the learning entries and making it 'easy' would hard to get right, I thought maybe use a 'share' type functionality might work except you are sharing it with yourself.

The 'exactly the right time' thing is also challenging.  I thought maybe you could ask the user a question for each revision, something like "Too soon | Just Right | Too Late" to help train the replay algorithm to the users retention profile

There are also opportunities to do other cool stuff with the data like maybe you could 'cram' a particular topic by pulling together all of the entries for that topic.

Just a thought.

Michael

Tuesday, July 01, 2014

2 Things I learned by reading the AngularJS Source Code

I'm busy working on a curriculum for my favourite front end Javascript framework AngularJS.  To really understand it, I have been stepping through and reading the AngularJS Source.

This codebase is awesome! well written and well documented.  Over and above understanding AngularJS, this is what I have learned along the way.

1) Doing more things faster with for
I have always just used for for the basic pro-forma like this:-

for (var j=0; j < myArray.length; j++) {
//do stuff with j

But the for loop can do so much more, check this out from the AngularJS source...
       
    // iterate over the attributes
    for (var attr, name, nName, ngAttrName, value, nAttrs = node.attributes,
        j = 0, jj = nAttrs && nAttrs.length; j < jj; j++) {

Wow! here, they are initialising all sorts of variables that will be used in the loop, not just the iterator variable 'j'

Also, note that they are 'caching' the length of the array in the variable 'jj'.  Turns out that this is a great optimisation technique and makes the loop go faster because you don't need to re-evaluate the length property on each loop.

for more details on optimising loops, check out these links
http://stackoverflow.com/questions/5349425/whats-the-fastest-way-to-loop-through-an-array-in-javascript
https://blogs.oracle.com/greimer/resource/loop-test.html

2) Assignment returns a value.. Use it

Look at this..
 
if (directiveValue = directive.scope) {

This looks like a mistake at first.. usually you use '==' in your if statements however this is definitely on purpose.  The assignment returns a value.  In this case, it returns the value of directive.scope to the 'if' statement.  'if' will evaluate any object to true (unless it is undefined) so this statement has both sets the directiveValue to the directive.scope object and checks if the directive.scope is defined.. elegant!

There are some more good examples of this here..
http://www.quirksmode.org/blog/archives/2008/01/using_the_assig.html

Tuesday, May 27, 2014

3 Great Reasons to use BDD

Through using and explaining BDD, I have become convinced that there are 3 main facets to the BDD approach.  Understanding the incremental benefit and cost of these facets is key to guiding you on your BDD journey.

1) A common vocabulary for describing behaviours

Using the 'Given, When, Then' pro-forma for describing behaviour is the first, easiest to introduce and IMO, most important aspect of the BDD approach.

It is easy to get across to people without resistance because it is so simple, almost trivial.  When I have introduced it to even skeptical teams the response has been oh, ok, thats how you do it, lets go.

The commonality of the approach is key.  Everybody in the team can use this language to describe the intended behaviour of the system.  Your BA folks don't need to write separate specifications, your QA folks don't need to write separate test cases and your Developers don't need to write separate integration tests (but more about that in a moment).  The Product owner can also see and understand exactly what he is getting without pouring over additional artefacts.

In short, your whole team can use this approach to simultaneously talk about quality, behaviour, user focused design and all agree on one description of the behaviour before moving on.  This simple approach is enormously powerful and is compelling enough to use even without using the other 2 facets.

2) Automated test creation

A raft of clever tooling (JBehave, Cucumber etc...) has been created that allow you to take the textual descriptions of behaviour that you get from using 'Given, When, Then' and turn them into automated 'black box' tests.  This usually involves writing some 'glue' or 'steps' code between the behaviour and the test.

Adding this to your BDD approach gives you the additional benefit of creating an automated test that both allows you to prove that the software works (and close a story in an Agile context) and give you an automated behavioural regression test that you can run over and over as the software evolves.

Note that you don't need to introduce this tooling and approach in addition to using the vocabulary aspect.  Indeed it does come with an additional cost.  Turning the behaviours into tests invariably takes some additional effort.  You need to weigh this additional effort with the double benefit of story closure and automated regression.

Using this facet of BDD requires careful consideration of the balance between cost and benefit and a firm eye on the medium and long term.

3) A 'Test First' Development methodology

This third aspect is BDD as an analog to TDD.  TDD or 'Test First' development is a development discipline where you don't write any code until you have a test to cover the code you are about to write.  If you are using TDD, your development process looks a bit like this...

Write unit test -> Test goes 'red' -> write code until test goes 'green'

When you overlay BDD into this methodology, your process might look a bit like this...

Create 'black box' Test -> Black box test goes red -> Write unit test -> unit test goes 'red' -> write code until unit test goes 'green' -> repeat unit test cycle until black box test goes green.

The key benefit of this approach is that your development team never writes any unnecessary code i.e. it prevents 'over engineering'.  The flip-side of this benefit and indeed the additional cost of this approach is related to refactoring.  Because the developer is encouraged to write 'just enough' foundational code to support the current behaviour, as you introduce new behaviours, there is often a need to refactor those foundations.

This is by far the most controversial aspect to BDD.  While most developers will agree that TDD is the best approach, there is no such consensus on 'test first' for BDD.  Many developers want to design the foundations of their solutions with a broad vision of what it needs to achieve with quotes like "If I need to build a 3 story house, I need bigger foundations than a single story house.  If I know this up front why not build the larger foundations first".





Tuesday, January 15, 2013

AngularJS $scope vs Controller Inheritance

I wrote this article because I was confused about what 'Controller Inheritance' actually was.  First let's look at $scope inheritance.

Example1 - $scope inheritance


Note) The ChildCtrl is nested within MainCtrl in the html.  AngularJS responds to this by ensuring that the $scope passed to ChildCtrl inherits properties and methods from the $scope passed to and decorated by MainCtrl.

Also Note) AngularJS does not introduce any inheritance (prototypical or otherwise) between the actual controllers themselves.

If you want your controllers to actually inherit properties and/or methods (that are not exposed on the $scope), then you can do this perfectly well by employing standard, non-AngularJS javascript prototypical inheritance between the controllers.

Example 2 - Rolling your own Controller Inheritance


   Note) I changed for the way I create my controllers from an anonymous approach utilising the controller method to a named function approach.  This is necessary because you need the named function in order to access and extend its prototype.

Also Note)  I forced the ChildCtrl to inherit from the main MainCtrl using prototypical inheritance. I was then able to do is extend the $scope of the ChildCtrl with a method that made use of the inherited method.