Node.js

First class functions in Java 8

Raoul-Gabriel Urma

This is a guest post from Raoul-Gabriel Urma, a PhD candidate in Computer Science at the University of Cambridge. His research centers on programming languages and software engineering. Raoul has written over 10 peer-reviewed articles and given over 20 technical talks at international conferences. He has worked for large companies such as Google, eBay, Oracle, and Goldman Sachs, as well as for several startup projects – and has recently written a book on Java 8 in Action: Lambdas, Streams and functional-style programming by Manning.

Raoul also teaches a course on modern development with Java 8 at Skills Matter, alongside Jim Gough & Richard Warburton. At the end of this course, you will be ready to use Java 8 on your day job and be familiar with the cutting edge programming approaches which allow you to write more flexible and concise code. You can find out more information about the course here, or head to the Skills Matter page to book your place now!


Java 8 adds functions as a new form of value. What does this mean? Let’s look at a simple example.

Suppose you want to filter all the hidden files in a directory. You need to start writing a method that given a File will tell you whether it is hidden or not. Thankfully there’s such a method inside the File class called isHidden. It can be viewed as a function that takes a File and returns a boolean.

However, to use it you need to wrap it into a FileFilter object that you then pass to the File.listFiles method as follows:

File[] hiddenFiles = new File(".").listFiles(new FileFilter() {
   public boolean accept(File file) {
      return file.isHidden();
   }
});

Ouch, that’s pretty obscure! We already have a function isHidden that we could use, why do we have to wrap it up in a verbose FileFilter object?

In Java 8 you can rewrite that code as follows:

File[] hiddenFiles = new File(".").listFiles(File:: isHidden);

Wow! Isn’t that cool? We already have the function isHidden available so we just “pass” it to the listFiles method. Our code now reads closer to the problem statement.

The use of File::isHidden is a rather special case of a new feature called method references in Java 8. Given that methods contain code (the executable body of a method), then using methods as values is like passing code around.

I hope this brief post has sparked some interest in Java 8! You can find a longer explanation in this 10min animated video we put together:

 


 

Five top Javascript Skillscasts from the Skills Matter archives

 

Gary Short: End to End Javascript (login required)

Gary Short, End to end Javascript

From the 2012 Progressive .NET Tutorials in London Gary Short, Technical Evangelist for Developer Express, looks at how Javascript as a language was beginning to infiltrate all three layers of application development – from JQuery in the front end, through NodeJS in the middle tier, to Map/Reduce functions on back end databases like CouchDB.

Gary looks at some of the gotcha’s within the language before going on to build an exemplar, 3-tier application in Javascript. By the end of the session he demonstrates a much better appreciation for the language and the power it has on all application tiers.


In the Brain of Carlos Ble: Behaviour Driven Rias with Javascript

Carlos Ble came along to Skills Matter a year ago for an In The Brain talk looking at a whole range of topics: why JavaScript is getting so popular, Executable Specifications with Cucumber for the RIA, the Passive View Pattern, event oriented programming and how to test-drive it, test driving promises, technical debt on purpose and the role of the “server side”.


Tamas Piros: An Introduction to AngularJS

In this talk organised by the Mean Stack user group, Tamas Piros came along to Skills Matter HQ with a presentation suitable for everyone (experienced and brand-new), with an introduction to AngularJS describing the basics such as data-binding, expressions and directives.


Jonathan Fielding: Building better experiences with Responsive Javascript

From the archives of another Mean Stack meetup, Jonathan Fielding goes beyond media queries showing what CSS can offer and moving into the realms of responsive Javascript.


Damjan Vujnovic: Backbone JS Unit tests

While most of us will agree that unit tests are an important aspect of development, actually implementing them seems to be a different story. Damjan takes you through your first steps with unit testing in Javascript in this mini workshop, hosted by the Backbone.js London user group.


FullStack – the Node and JavaScript Conference

fullstack-tri-blog

This year sees us host our first ever FullStack Conference, bringing together the best in the worlds of NodeJS, Javascript and hackable electronics.

Join world leaders such as Douglas Crockford (creator of JSON), Cian O’Maidin (founder and CEO at nearForm) and Karolina Szczur (Smashing Magazine) for two days jam-packed with talks, demos, and coding.

Book your ticket now!

 

Build a Search Engine for Node.js Modules using Microservices (Part 2)

This is the second part of a three part guest post by Richard Rodger, CTO at nearForm & a leading Node.js specialist. Richard will be speaking at FullStack, the Node and JavaScript Conference on 23 October.

Following on from last week’s post in which Richard took us through his experience of building a search engine using microservices, here you will go through the steps to test your microservices and deploy them, in this example using Docker.

The final part will be available next week, after Richard’s In The Brain talk on the 22 September, building a Node.js search engine using the microservice architecture.


Testing the Microservice

Now that you have some code, you need to test it. Actually, that’s where you should have started. No Test-Driven Development brownie points for you!

Testing microservices is nice & simple. You just confirm that message patterns generate the results you want. There’s no complicated call flows or object hierarchy tear-ups or dependency injections or any of that nonsense.

The test code uses the mocha test framework. To execute the tests, run

$ npm test

Do this in the project folder. You may need to run npm install to get the module dependencies if you have not done so already.

First you need to create a Seneca instance to execute the actions you want to test:

  var si = seneca({log:'silent'})
        .use('jsonfile-store',{folder:__dirname+'/data'})
        .use('../npm.js')

Create a new Seneca instance and store it in the variable si. This will be used for multiple tests. This code keeps logging to a minimum ({log:'silent'}), so that you get clean test result output. If you want to see all the logs when you run a test, try this:

$ SENECA_LOG=all npm test

For more details on Seneca logging, including filtering, read the logging documentation here.

The next line tells Seneca to load the seneca-jsonfile-store plugin. This is a Seneca plugin that provides a data store that saves data entities to disk as JSON files. The plugin provides implementations of the data message patterns, such as role:entity,cmd:save, and so forth. The folder option specifies the folder path where the JSON documents should be stored on disk.

Finally, load your own plugin, npm.js, that defines the messages of the nodezoo-npm microservice. It may sound fancy having “plugins”, but all they really are, are just collections of message patterns and a way to make logging and option handling a bit more organised. If you want to know all the details there’s some Seneca plugin documentation.

Let’s test each message pattern. Here’s the role:npm,cmd:extract pattern:

  it('extract', function( fin ) {

    si.act(
      'role:npm,cmd:extract',
      {data:test_data},

      function(err,out){
        assert.equal('npm-version-verify-test',out.name)
        assert.equal('0.0.2',out.version)
        fin()
      })
  })
  

The seneca.act method submits a message into the Seneca system for processing. If a microservice inside the current Node.js process matches the message pattern, then it gets the message and performs its action. Otherwise the message is sent out onto the network – maybe some other microservice knows how to deal with it.

In this case everything is local, as you would wish for a unit test. The variable test_data contains a test JSON document. The test case verifies that the expected properties are extracted correctly.

The seneca.act method lets you specify the message contents in more than one way as a convenience. You can submit an object directly or you can use a string of abbreviated JSON or both. The following lines are all equivalent:

si.act( { role: 'npm', cmd:'extract', data:test_data}, ... )
si.act( { 'role:npm,cmd:extract, data:{...}', ... )
si.act( { 'role:npm,cmd:extract', {data:test_data}, ... )

Here’s the test case for the role:npm,cmd:query action. It’s pretty much the same:

  it('query', function( fin ) {
    si.options({errhandler:make_errhandler(fin)})

    si.act(
      'role:npm,cmd:query',
      {name:'npm-version-verify-test'},
      function(err,out){
        assert.equal('npm-version-verify-test',out.name)
        assert.equal('0.0.2',out.version)
        fin()
      })
  })
  

Finally you need to test the role:npm,cmd:get action. In this case, you want to delete any old data first to ensure a test that triggers the role:npm,cmd:query action. This is the important use-case to cover when a visitor is looking for information on a module you haven’t retrieved from npmjs.org yet.

  it('get', function( fin ) {

    si.make$('npm').load$('npm-version-verify-test',function(err,out){

      if( out ) {
        out.remove$(do_get)
      }
      else do_get()

      function do_get() {
        si.act(
          'role:npm,cmd:get',
          {name:'npm-version-verify-test'},
          function(err,out){
            assert.equal('npm-version-verify-test',out.name)
            assert.equal('0.0.2',out.version)
            fin()
          })
      }
    })
  })
  

The do_get function is just a convenience to handle the callbacks.

Running Microservices

In production you’ll run microservices using a deployment system such as Docker or tools that build on Docker, or one microservice per instance or some other variant. In any case you’ll definitely automate your deployment. We’ll talk more about deployment in a later part of this series.

On your local development machine it’s a different story. Sometimes you just want to run a single micro-instance in isolation. This lets you test message behaviour and debug without complications.

For the nodezoo-npm service there’s a script in the srvs folder of the repository that does this for you: npm-dev.js. Here’s the code:

var seneca = require('seneca')()
      .use('jsonfile-store')
      .use('../npm.js')
      .listen()
      

When you run microservices it’s good to separate the business logic from the infrastructure set up. This tiny script runs the microservice in a simplistic mode. It’s very similar to the test set up code except that it includes a call to the seneca.listen() method.

The listen method exposes the microservice on the network. You can send any of the role:npm messages to this web server and get a response. Try it now! Start the microservice in one terminal:

$ node srvs/npm-dev.js

And issue a HTTP request using curl in another terminal:

$ curl -d '{"role":"npm","cmd":"get","name":"underscore"}' http://localhost:9001/act

How does the micro-server know to listen on port 9001? And how does the seneca-jsonfile-store know where to save its data? Seneca looks for a file called seneca.options.js in the current folder. This file can define options for plugins. Here’s what it looks like:

module.exports = {
  'jsonfile-store': {
    folder: __dirname+'/data'
  },
  transport: {
    web: {
      port: 9001
    }
  }
}

If you want to see everything the microservice is doing, you can run it with all logging enabled. Be prepared, it’s a lot.

$ node srvs/npm-dev.js --seneca.log.all

Although the default message transport is HTTP, there are many others such as direct TCP, redis publish/subscribe, message queues, and others. To find out more about message transportation see the seneca-transport plugin.

There’s also an npm-prod.js script for running in production. We’ll talk about that in a later part of this series. The srvs folder for this microservice only contains two service scripts. You can write more to cover different deployment scenarios. You might decide to use a proper database instead. You might decide to use a message queue for message transport. Write a script for each case as you need it. And there’s nothing special about the srvs folder either. Microservices can run anywhere.

Next Time

In part three, you’ll build microservices to query Github, to aggregate the module information, and to display it via a website. If you’re feeling brave go ahead and play with the code for these services, it’s already up on Github!

Don’t forget, Richard will be speaking at Skills Matter HQ in London next week, covering the topics in these posts. Register for your free place here! The concluding part of this post will be published here next week, be sure to come back then!


FullStack: the Node and Javascript Conference

FullStack

Join Richard Rodger, along with many other world-leading experts and hundreds of Javascript and Node enthusiasts at our first ever FullStack Conference. Come along to experience two days jam-packed with talks, demos, and coding!

Build a Search Engine for Node.js Modules using Microservices (Part 1)

Richard Rodger

This is a guest post from Richard Rodger, CTO at nearForm & a leading Node.js specialist. He is a technology entrepreneur and wrote the seminal book on Node: Beginning Mobile Application Development in the Cloud. This is the first of a three-part blog, with the second to follow next week and the last post after his In The Brain talk at Skills Matter on the 22 September, building a Node.js search engine using the microservice architecture.

Richard will also be speaking at FullStack, the Node and JavaScript Conference on 23 October.

In this posts Richard will take us through his experience of building a search engine using microservices and how using a microservice architecture can be used to isolate features and create separate components that can be independently developed, deployed and maintained. Part one will introduce you to microservices and start you building your components and modules. Next week, we’ll cover testing the microservice!


Finding the right Node.js module is hard. With over 90,000 modules (and counting!) it can be tricky to pick the right one. You’ll often come across multiple modules that solve your problem. They are often incomplete, abandoned or just badly documented. It’s hard to make the right choice and you have to rely on rules of thumb. When was the last commit? How many stars? How credible is the author?

It would be great if there was a search engine for Node.js modules that ranked them using these heuristics. I built one in late 2012 that tried to do just that. It was a miserable failure! The quality of the results was pretty poor – see for yourself at nodezoo.com. Search engines are harder than they look, let me tell you. Recently I decided to try again. This time, I wanted to use a microservice architecture. This lets me iterate many aspects of the engine, so that it can get better over time.

This blog post is the first in a series covering nodezoo version 2.0. I’ll also be talking about this at my upcoming In The Brain talk on September 22nd. In this blog post you’ll see how a microservice architecture can be used to isolate the features of the search engine into separate components that can be independently developed, deployed, and maintained – developer heaven!

Let’s focus on the information page for each module. After you get your search results, you should be able to click-through to a page that shows you all the useful meta data about a module, such as the number of Github stars, so that you can compare it with other modules. So today we’ll build out one of the services needed to collect that information and display it.

Why Microservices?

The search engine will use a microservice architecture. Let’s take a closer look at what that means. Microservices are small, independently running processes. Instead of a single large, monolithic application running all your functionality, you break it apart into many separate pieces that communicate with each other over the network.

There are quite a few informal “definitions” of microservices out there:

…developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API
James Lewis & Martin Fowler

…a system that is built up of small, lightweight services, where each performs a single function. The services are arranged in independently deployable groups and communicate with each other via a well defined interface and work together to provide business value.
David Morgantini

…a simple application that sits around the 10-100 lines-of-code mark
James Hughes

And even a “ManifestNO”
Russell Miles

Of course, I have my own definition! It’s really the effect on humans that matters most. Microservices are just more fun. I prefer:

An independent component that can be rewritten in one week.

This gets to the heart of microservices. They are disposable code. You can throw away what you have and start again with very little impact. That makes it safe to experiment, safe to build up technical debt and safe to refactor.

What about all that network traffic, won’t it slow things down to a crawl? In practice, no. We’ve not seen this in the production systems that we have built. It comes down to two factors. First, inter-service communication grows logarithmically, not exponentially. Most services don’t need to talk to most other services.

Second, responding to any message has a relatively shallow dependency-based traversal of the service network. Huh? Microservices can work in parallel. Mostly they don’t need to wait for each other. Even when they do, there’s almost never a big traffic jam of services waiting in a chain.

In fact a microservice architecture has a very nice characteristic. Out-of-the-box you get capacity to handle lots and lots of users. You can scale easily by duplicating services which by design have no shared state. Yes, it does make each individual request a little slower. How bad! In contrast, traditional monolithic approaches give you the opposite: better performance for one user, terrible performance for many. Throughput is what counts if you want to have lots of users and microservices make a very cheap trade-off in terms of pure performance. Here’s the obligatory chart showing how this works. Hint: you want to be in the green box. The red one gets you fired.

perf

One more thing. It’s easy to accuse microservices of falling afoul of the Fallacies of Distributed Computing. Essentially pretending that the network is just like making a local function call. That’s precisely wrong. Microservices embrace the network. You are naturally led to deal with the effects of the network when you use microservices because you can’t avoid them. You have to assume that every time you talk to another component in the system you will incur network traffic. It enforces a productive discipline and clarity of mind. Architecture astronauts are quickly and severely punished.

The Search Engine

You won’t build a search engine in one blog post. Let’s look at the module information page. This collects data from multiple sources and displays it all together. Let’s take a bias towards the Availability aspect of the CAP Theorem. Whatever else happens, we want to display a web page to the user.

We’ll keep things simple. Each Node.js module has description data stored by the NPM registry. Let’s get a subset of that information and display it. Most Node.js modules also have a Github repository. It’s useful to know how many stars and forks a module has as that gives some measure of how good it is. Down the road, we might pull information from other sources, like the module author’s blog or a Google search.

So we have a set of information sources, mostly external, certainly unreliable, and definitely slow! Whatever else happens we’ve decided that we’ll always display the information page even if we have no information or only some of it. In microservice terms, you publish a message looking for information and you display whatever you get back within a certain time-frame (or some other cut-off condition). This scatter-gather pattern is explained in detail by Fred George in this video.

This acceptance of faulty behavior, this anticipation that hard choices will have to be made and results will be imperfect, is a key part of the microservice philosophy. We aim to reduce uncertainty and the frequency of failure but we never seek to eliminate it. Solutions which cannot tolerate component failure are inherently fragile. We aim to be anti-fragile.

The Architecture

There needs to be a microservice for the information page and then one for each data source. For the moment, that means one for NPM and one for Github. And we’ll also need a web server! Let’s use KrakenJS because it’s pretty cool and has easy server-side templates.

Let’s call the information page microservice nodezoo-info, the data source ones nodezoo-npm & nodezoo-github, and the KrakenJS process can be nodezoo-web. That gives you four Node.js processes to run at a minimum, or not – you’ll see in a later part of this series how to reduce this to one process when developing and debugging locally.

Now, the nodezoo-web service delivers the information page. Let’s use the URL path /info/MODULE. The controller that builds that page should ask the nodezoo-info service for the details of the module. The nodezoo-info service should publish a message to any interested services that care to provide information about modules. The nodezoo-npm and nodezoo-github services should listen for that message and respond if they can.

Sounds nice and neat, but there’s a complication. Isn’t there always! You need the Github repository URL to query Github. That information is in NPM. And yet, the Github service is not allowed to know about the NPM service. For the moment let’s assume we can have Schrödinger services that both do and do not know about other services. That allows the Github service to ask the NPM service for the repository URL. To be continued…

This is so simple that you don’t even need an architecture diagram to understand it. What? It would be “unprofessional” not to provide one you say? But microservice deployments are necessarily dynamic and fluid, changing minute-by-minute to meet the demands of user load and new business requirements. The whole point is that there is no Architecture (capital A). The best mental model is to suspend disbelief and assume all services can see all messages, ignoring those they don’t care about. Message transport is just an optimization problem.

Oh, all right then.

arch

The lovely diagram shows you something else. Each microservice owns its own data. There is no shared database. What about consistency you say? Well think about our use case. Node.js modules are not updated that frequently. We can publish messages when they are, and any services that need to update, can do so at their own pace. But module data will be out-of-date and inconsistent in some cases! Well, shucks. That’s by design. We are not going for 100% here. We’re going for fault tolerance.

Think about the error level you find acceptable, and put remedial actions in place to achieve that. Perhaps you publish the update message multiple times over the following 24 hours. It’s wasteful of resources, but covers a lot of cases where services might be down temporarily. Perhaps you have a cross-check service that randomly samples the services to see how consistent they are. Perhaps you have a set of data services and you outsource the problem. This is the style of thinking you need to adopt. Assume things will break.

Now there are many use-cases where this cavalier approach to data is not going to fly. But there are many others where it is perfectly fine. Microservices make it easy to avoid over-engineering when you don’t have to. They can all talk to the same ACID-compliant Enterprise Database Solution if you really care that much. But then you have to install and maintain the damn thing. Expensive.

The Code

Let’s look at the NPM service first as that’s pretty simple. The code in this article uses the Seneca microservice framework. That means it’s concise, you can focus on the business logic, and you don’t need to worry about message transportation. Some may balk at the idea of using a framework for microservices. After three years of building systems this way, it makes sense for us to put our collected experience into a reusable package. But you don’t need permission to go bare bones if that’s what you want to do. The basic principles are exactly the same.

The NPM service needs to do three things:

  • return data about a module, if it has that data;
  • query the NPM registry for data about a module;
  • extract relevant data from the large NPM JSON document returned for each module.

Seneca defines microservices as collections of one or more message patterns. If a message matches the pattern, then the microservice executes it. Messages are considered to be plain old JSON documents. The patterns are simple name and value pairs. If the JSON document has matching top-level properties, then it’s a match. More specific matches win. That is, a pattern with more name and value pairs beats a pattern with fewer. And that’s it.

The pattern matching is deliberately simple so that you can simulate it inside your own head. There is no fancy stuff like regular expressions or dot-notation for sub-properties. The Principle of Least Power is very – well – powerful. If you need to get fancy, match as much as you can and do the rest in code.

As noted above, all the code for this article is open-sourced in Github. It’s available under the MIT license, so you are free to re-use for your own projects, open-source or commercial.

Here’s the message action that extracts data from the NPM JSON document.

// excerpt from: nodezoo-npm/npm.js

 seneca.add(
   'role:npm,cmd:extract',
   {
     data: { required$:true, object$:true },
   },
   cmd_extract)

 function cmd_extract( args, done ) {
   var seneca  = this

   var data       = args.data
   var dist_tags  = data['dist-tags'] || {}
   var latest     = ((data.versions||{})[dist_tags.latest]) || {}
   var repository = latest.repository || {}

   var out = {
     name:    data._id,
     version: dist_tags.latest,
     giturl:  repository.url
   }

   done(null,out)
 }

The message action pattern is role:npm, cmd:extract. This is completely arbitrary. The role property namespaces the pattern, and the cmd property names what it does. Not all patterns are commands, some are just announcements for general interest, as you’ll see in later parts.

The implementation of the pattern, the action that it triggers, is placed in the cmd_extract function. This is just good code organization. It also means you can list all the patterns at the top of the source code file, which gives you a little bit of built-in documentation.

The seneca.add method adds a new message pattern. You supply the pattern and its implementing function. Optionally you can specify the arguments you need. These are additional properties that should be present in the message JSON document. In this case, a data property must be present, and it’s value must be an object. That is, the data property should contain the the NPM JSON result. For more details of the argument validation format, see the parambulator Node.js module.

The action implementation follows the standard Node.js callback pattern:

function some_action( args, done ) {
  // context-specific seneca instance – gives you nice logs!
  var seneca = this

  ...

  // there is no error, so the first argument is null
  done(null,out)
}

The args object is the message document. This what you act on. The done function is the callback to call when you are finished. If there was an error, you should pass an Error object as the first argument to done. Otherwise, pass a null, and then your result as the second. Returning a result is not required. Some messages are fire-and-forget, and there is no result, as such. Always call done though, even if it’s just done(); as you need to signal that you’re finished processing.

The extraction logic is nothing special, it just picks values out of an object data structure. To see the full structure, try this for example.

Data Storage

Microservices do need to store data occasionally. If you want to be puritanical, then data storage should also be a microservice! Data operations are just more messages. A load message maybe, a save message, a remove message, and so forth. Purity is overrated. Seneca does do things this way, but it provides a nice and easy ActiveRecord-style interface for data entities, so you don’t have to worry about it.

This gives you all sorts of lovely side-benefits. Choose this database! No, that one! Change database in mid-project. Who cares! Need to cache data in redis or memcached? Intercept the messages! No need for the business logic code to know anything about it. Need special case validation or data munging? Intercept and do what you want!

I’ll stop now.

Here’s what the API looks like:

  • Create a new data entity: var color = seneca.make$(“color”)
  • Save some data: color.name = “red”; color.save$( callback )
  • Load some data: color.load$( {name:”red”}, callback )

For more details, see the Seneca Data Entities documentation.

For the moment, you’ll store all data as plain text JSON files on disk. Yes, plain old JSON. On disk. No database needed. Later on, in production, we can worry about LevelDB or Mongo or Postgres (no code changes required! I promise). For now, let’s make local development and debugging super easy.

OK, let’s respond to a request for information about a Node.js module:

  seneca.add(
    'role:npm,cmd:get',
    {
      name:   { required$:true, string$:true },
    },
    cmd_get)

  function cmd_get( args, done ) {
    var seneca  = this
    var npm_ent = seneca.make$('npm')

    var npm_name = args.name

    npm_ent.load$( npm_name, function(err,npm){
      if( err ) return done(err);

      // npm is not null! there was data! return it!
      if( npm ) {
        return done(null,npm);
      }

      // query the npmjs registry as we
      // don't know about this module yet
      else {
        seneca.act(
          'role:npm,cmd:query',
          {name:npm_name},
          done)
      }
    })
  }

This implements the role:npm,cmd:get pattern. You need to provide a name property in your message. Speaking of which, here’s an example message that will work:

{
  "role": "npm",
  "cmd":  "get",
  "name": "express"
}

The code attempts to load a module description with the given name. The npm entity stores this data. If it is found, well then just return it! Job done. The caller gets back a JSON document with data fields as per the role:npm,cmd:extract action.

If the module was not found in the data store, then you need to query the NPM registry. Submit a new message with pattern role:npm,cmd:query. That’s that action that does that job. Return whatever it returns.

Finally, let’s look at the query action. This needs to make a HTTP call out the the registry.npmjs.org site to pull down the module JSON description. The best module to do this is request.

  seneca.add(
    'role:npm,cmd:query',
    {
      name: { required$:true, string$:true },
    },
    cmd_query)

  function cmd_query( args, done ) {
    var seneca  = this
    var npm_ent = seneca.make$('npm')

    var npm_name = args.name

    var url = options.registry+npm_name
    request.get( url, function(err,res,body){
      if(err) return done(err);

      var data = JSON.parse(body)

      seneca.act('role:npm,cmd:extract',{data:data},function(err,data){
        if(err) return done(err)

        npm_ent.load$(npm_name, function(err,npm){
          if( err ) return done(err);

          // already have this module
          // update it with .data$ and then save with .save$
          if( npm ) {
            return npm.data$(data).save$(done);
          }

          // a new module! manually set the id, and then
          // make a new unsaved entity with .make$,
          // set the data and save
          else {
            data.id$ = npm_name
            npm_ent.make$(data).save$(done);
          }
        })

      })
    })
  }

This action constructs the query URL and performs a HTTP GET against it. The resulting document is then parsed by the role:npm,cmd:extract action to pull out the properties of interest and merged into any existing data on that module, or else saved as a new data entity. The data.id$ = npm_name line is where you manually set the identifier you want to use for a new data entity.

And that’s all this microservice does. 112 lines of code. Not bad.


Check back next week for part two, when we’ll cover testing the microservice. Can’t wait until then? Let us know your thoughts, ideas and experiences with microservices below, or give us a shout on Twitter!

FullStack

While It’s Compiling: Skills Matter interviews Karolina Szczur

Karolina Szczur is a designer and developer at &yet and contributes to various open source projects such as NodeCopter, AmpersandJS and Otalk.

We caught up with Karolina ahead of her talk at Fullstack: The Node and JavaScript Conference where she will be looking at up-to-date front-end tooling and analysing alternative approaches to compiling, building and automation processes. Here she gives us an insight into her talk topic, the iterative nature of design and her views and thoughts on the Open Source community.

karolina-szczur


Your recent work includes roles such as the UX Lead at Nodejitsu, writer and editor at Smashing Magazine for the mobile, coding & design sections, and Front-end Developer/Design Lead at XHTMLized. However, you started out doing photography, via psychology, anthropology and the history of art among other things – what first drew you towards development and design?

Honestly, I don’t recall a specific aha moment. I’ve always been drawn to art and photography—we used to have a darkroom at home when I was younger. As soon as digital photography went somewhat mainstream (or at least it became more accessible to the layperson) I started my adventure with editing in Photoshop 5 (yes, it was the 90s). That quickly led to trying to do more than adjust the colours or exposure. It was at the time when GeoCities was at its peak, so my photography passion escalated to “What if I could have a website too?”. I’d say it was rather natural curiosity and organic growth of interest in interconnected fields.

At the end of a recent article you wrote that design “is by nature an iterative process”, and that contributions from others is essential. Considering your experience in front-end design, can you take us a little further into this iterative process?

It takes an extraordinary amount of craftsmanship to create something that’s in any way timeless. Great design and front-end don’t happen instantly. Research, prototyping, implementation and testing take time. But trying to chase perfection before launching only prevents shipping. Like many designers, I’ve struggled with attempting to achieve some arbitrary ideal, but that kind of thinking doesn’t foster creativity.

Content we design for often changes, sometimes drastically. Web standards evolve. We can’t rely on predictability of either the medium or the technologies we have at hand. Working on the web’s rapidly evolving platform means our work will always be an iterative process, whether or not we choose that. There will always be something to improve or fix and we have to be ready to embrace the absolute lack of perfection.

There has been a lot of discussion lately surrounding the importance of shipping—iteration is empowering because it removes the boundaries of judgemental and perfectionist ideals. Collaboration and contributions from others take this idea even further by allowing room for constant feedback, broader perspective, and fresh ideas.

When working on a project, how do you ensure you maintain ownership whilst still benefiting from the input of others?

Ownership is essential. It doesn’t even have to be designated (“This person is the owner”). Often I see individuals stepping up as leaders. In a way, every single person involved in a certain project is an owner—they are all responsible for its execution and success. When everyone feels like that instead of irrelevant cog in a giant machine, feedback happens naturally.

Building up fearless candor and honest feedback requires zero judgmentalism and allowing others to step up. Making everyone a leader doesn’t take ownership away—it helps us to be better when working towards a mutual goal.

From a practical standpoint: ask for specific input as often as you can and enable anyone to express their opinions if they are willing to do so. Kill bikeshedding.

In your upcoming talk at FullStack you’re looking at the dangers of introducing too many tools in the development process, making everything more complex and time-intensive. How can you tell when your work-flow is as optimised as it’s going to get?

The topic of tooling is as opinionated as debating the superiority of programming languages or promises over callbacks. Automation and optimisation are crucial, both for complex, bigger projects and the small ones—they let us focus on tasks that simply cannot be automated and bring desired levels of uniformity.

The questions I like to ask myself are: Can it be faster? Can I use less tools and more natively available utilities? And last but not least—is it understandable and simple enough to empower my team members and collaborators? I believe that all tools are ultimately created to make our lives easier, thus making sure that it doesn’t introduce cognitive overhead for others is crucial. If it’s standardised, fast, reliable and comprehensible for others—I know I’m home.

It seems that a lot of this comes from the work you’ve been doing on standardisation of front-end tooling at &yet. Can you tell us a little bit more about &yet, and the work you’re doing there?

I’ve been working at &yet for over a year now—it’s been a very humbling experience that allowed me to grow both as a professional and as a person. I’ve been mostly focusing on internal projects, be it our own products or processes for optimising design and development.

In the last months I’ve been busy with a larger endeavour I like to call front-end standardisation. With a hugely multidisciplinary team, where almost everyone can code at some level, it’s important to create an environment that not only enables learning good patterns but also fosters code consistency:

“All code in any code-base should look like a single person typed it, no matter how many people contributed.”

These efforts been mostly focused on creating a code guide and pattern library in a form of small, modular CSS framework. It’s definitely a very challenging project and requires a lot of collaboration and feedback from team members. Standards work is never one person creating arbitrary rules to follow, but an collective effort for reaching consensus of optimal approaches suitable for given teams.

Next adventure? Improvements to our WebRTC-based video chat app— Talky.

&yet describes itself as an Open Web Company, and on your own website you say that you strongly believe in Open Source. What makes Open Source so important in your opinion?

Open Source was always at the core of our interests—we’re strongly involved in the Node, XMPP, WebRTC, and JavaScript communities. We support people who write and maintain libraries that we use on Gittip and we contribute and share everything we can.

It aligns perfectly with our people first approach to everything we do. After all, writing software or designing is just a means to an end. If we can enable others to make the world a better place even slightly by open sourcing our work or paying others to do so, then we will. We are building these things and working on standards for the good of all of us. If that doesn’t make open source important then I have no idea what does.

karolina-szczur-blog-post

&yet have been involved in creating a few community events themselves, and you’ve spoken at conferences such as JSConf EU, the Future of Web Design in Prague and the Frontend Conference in Zurich, amongst many others. How important is the community to the future of Open Source?

The dictionary definition of Open Source focuses on the openness and public availability of the source code. I want to go a little bit further—without people, Open Source dies. With collaboration and contributions projects flourish.

I don’t want to undermine the quality or livelihood of one-person projects (there’s plenty of those on npm) but we’ve learned the need for a community with our recent endeavour, AmpersandJS. We got quick traction and many contributors—one of them already added to the core team. One day we were debating the accessibility of the role attribute and got almost 70 comments from people deeply involved in that matter. We wouldn’t be able to improve the project and push it forward so fast without the help of the community.

Maintaining an Open Source project is a job—it takes a tremendous amount of time and, in a semi-direct way, money. Finding and even fixing bugs is so much harder without others using the software in the wild. I can’t see the future of Open Source without the community surrounding it.

Considering the conferences you’ve been involved in have taken place all over the world, do you see yourself as part of an intrinsically global community, or has your experience been that developers’ goals and missions change depending on where they are?

The idea of global community still feels a little bit like a dream to me. There are a lot of subcommunities, which on one hand is completely natural (the design community, the JavaScript community or the Ruby community) but when those communities become too exclusive and the barriers to entry become discouraging for newcomers it’s hard to talk about a global population of people involved in tech. There’s not only division in specific fields of interest or programming languages but also cultural differences—The Bay Area is so much different from the fairly small town of Kraków where I live, so I can see how priorities could shift depending on the location.

Nonetheless what I would love to see is less judgmentalism, more openness towards beginners as well as inclusivity towards people of colour, women, and other minorities, including people for whom English is not their first language. I certainly don’t feel excluded nor do I try to categorise myself as a part of a certain group, but until we all feel fairly comfortable to say that I can’t see a healthy, global community, there’s still work to do to make everyone feel welcome.


While It’s Compiling is a continuing series of interviews with experts across a range of bleeding-edge technologies and practices, exclusive to Skills Matter. Be sure to subscribe to this blog for future interviews, or follow us on Twitter.

Find out who we’ll be interviewing next, and get a chance to put your questions forward with the hashtag #whileitscompiling.

fullstack-2014-1000x150px