+ +

+ + +Often, a developer wants to use 3rd-party code, usually available as a library +of some kind. +In the Javascript world, the concept of a module is relatively new, so there +was no standard until recently. +Many platforms (browsers) still don't support modules, which makes code +reuse harder. +The njs does not (yet) support modules, too. +This article describes ways to overcome this limitation, using the +Node.js ecosystem as an example. + + + +Examples in this article use features that appeared in +njs +0.3.8 + + + +There is a number of issues that may arise when 3rd-party code is added to njs: + + + +Multiple files that reference each other, and their +dependencies + +Platform-specific APIs + +Modern standard language constructions + + + + + + +The good news is that such problems are not something new or +specific to njs. +Javascript developers face them daily when trying to support multiple +disparate platforms with very different properties. +There are instruments designed to resolve the above-mentioned issues. + + + + + + + +Multiple files that reference each other, and their dependencies + +This can be solved by merging all the interdependent code into a single file. +Tools like +browserify or +webpack +accept an entire project and produce a single file containing +your code and all the dependencies. + + + + +Platform-specific APIs + +You can use multiple libraries that implement such APIs in a platform-agnostic +manner (at the expense of performance, though). +Particular features can also be implemented using the +polyfill approach. + + + + +Modern standard language constructions + +Such code can be transpiled: this means performing a number of transformations +that rewrite newer language features in accordance with an older standard. +For example, babel project can +be used to this purpose. + + + + + + + + + +In this guide, we will use two relatively large npm-hosted libraries: + + + + +protobufjs - +a library for creating and parsing protobuf messages used by the +gRPC protocol. + + + + +dns-packet - +a library for processing DNS protocol packets. + + + + + + +

+ +

+ + + + +This document mostly employs a generic approach and AVOIDS specific best +practice advices concerning Node.js and the rapidly evolving JavaScript +ecosystem. +Make sure to consult the corresponding package's manual BEFORE following the +steps suggested here. + + +First (assuming Node.js is installed and operational), let's create an +empty project and install some dependencies; the commands below assume we're +in the working directory: + + +$ mkdir my_project && cd my_project +$ npx license choose_your_license_here > LICENSE +$ npx gitignore node + +$ cat > package.json <<EOF +{ + "name": "foobar", + "version": "0.0.1", + "description": "", + "main": "index.js", + "keywords": [], + "author": "somename <some.email@example.com> (https://example.com)", + "license": "some_license_here", + "private": true, + "scripts": { + "test": "echo \"Error: no test specified\" && exit 1" + } +} +EOF +$ npm init -y +$ npm install browserify + + + +

+ +

+ + +The library provides a parser for the .proto interface +definitions and a code generator for message parsing and generation. + + + +In this example, we will use the +helloworld.proto +file from the gRPC examples. +Our goal is to create two messages: HelloRequest and +HelloResponse. +We will use the +static +mode of protobufjs instead of dynamically generating classes, because +njs doesn't support adding new functions dynamically due to security +considerations. + + + +Next, the library is installed and javascript code implementing +message marshalling is generated from the protocol definition: + +$ npm install protobufjs +$ npx pbjs -t static-module helloworld.proto > static.js + + + + + +Thus, the static.js file becomes our new dependency, storing +all the code we need to implement message processing. +The set_buffer() function contains code that uses the +library to create a buffer with the serialized HelloRequest +message. +The code resides in the code.js file: + + +var pb = require('./static.js'); + +// Example usage of protobuf library: prepare a buffer to send +function set_buffer(pb) +{ + // set fields of gRPC payload + var payload = { name: "TestString" }; + + // create an object + var message = pb.helloworld.HelloRequest.create(payload); + + // serialize object to buffer + var buffer = pb.helloworld.HelloRequest.encode(message).finish(); + + var n = buffer.length; + + var frame = new Uint8Array(5 + buffer.length); + + frame[0] = 0; // 'compressed' flag + frame[1] = (n & 0xFF000000) >>> 24; // length: uint32 in network order + frame[2] = (n & 0x00FF0000) >>> 16; + frame[3] = (n & 0x0000FF00) >>> 8; + frame[4] = (n & 0x000000FF) >>> 0; + + frame.set(buffer, 5); + + return frame; +} + +var frame = set_buffer(pb); + + + + +To ensure it works, we execute the code using node: + + +$ node ./code.js +Uint8Array [ + 0, 0, 0, 0, 12, 10, + 10, 84, 101, 115, 116, 83, + 116, 114, 105, 110, 103 +] + + +You can see thet this got us a properly encoded gRPC frame. +Now let's run it with njs: + + +$ njs ./code.js +Thrown: +Error: Cannot find module "./static.js" + at require (native) + at main (native) + + + + +Modules are not supported, so we've received an exception. +To overcome this issue, let's use browserify +or other similar tool. + + + +An attempt to process our existing code.js file will result +in a bunch of JS code that is supposed to run in a browser, i.e. immediately +upon loading. +This isn't something we actually want. +Instead, we want to have an exported function that +can be referenced from the nginx configuration. +This requires some wrapper code. + + +In this guide, we use njs cli in all examples for the sake of simplicity. +In real life, you will be using nginx njs module to run your code. + + + + + +The load.js file contains the library-loading code that +stores its handle in the global namespace: + +global.hello = require('./static.js'); + +This code will be replaced with merged content. +Our code will be using the "global.hello" handle to access +the library. + + + +Next, we process it with browserify to get all dependencies into a single file: + +$ npx browserify load.js -o bundle.js -d + + +The result is huge file that contains all our dependencies: + + +(function(){function...... +... +... +},{"protobufjs/minimal":9}]},{},[1]) +//# sourceMappingURL.............. + + +To get final "njs_bundle.js" file we concatenate +"bundle.js" and the following code: + + +// Example usage of protobuf library: prepare a buffer to send +function set_buffer(pb) +{ + // set fields of gRPC payload + var payload = { name: "TestString" }; + + // create an object + var message = pb.helloworld.HelloRequest.create(payload); + + // serialize object to buffer + var buffer = pb.helloworld.HelloRequest.encode(message).finish(); + + var n = buffer.length; + + var frame = new Uint8Array(5 + buffer.length); + + frame[0] = 0; // 'compressed' flag + frame[1] = (n & 0xFF000000) >>> 24; // length: uint32 in network order + frame[2] = (n & 0x00FF0000) >>> 16; + frame[3] = (n & 0x0000FF00) >>> 8; + frame[4] = (n & 0x000000FF) >>> 0; + + frame.set(buffer, 5); + + return frame; +} + +// functions to be called from outside +function setbuf() +{ + return set_buffer(global.hello); +} + +// call the code +var frame = setbuf(); +console.log(frame); + + +Let's run the file using node to make sure things still work: + +$ node ./njs_bundle.js +Uint8Array [ + 0, 0, 0, 0, 12, 10, + 10, 84, 101, 115, 116, 83, + 116, 114, 105, 110, 103 +] + + +Now let's proceed further with njs: + + +$ /njs ./njs_bundle.js +Uint8Array [0,0,0,0,12,10,10,84,101,115,116,83,116,114,105,110,103] + + +The last thing will be to use njs-specific API to convert +array into byte string, so it could be usable by nginx module. +We can add the following snippet before the last line: + +if (global.njs) { + return String.bytesFrom(frame) +} + + +Finally, we got it working: + + +$ njs ./njs_bundle.js |hexdump -C +00000000 00 00 00 00 0c 0a 0a 54 65 73 74 53 74 72 69 6e |.......TestStrin| +00000010 67 0a |g.| +00000012 + +This is the intended result. +Response parsing can be implemented similarly: + +function parse_msg(pb, msg) +{ + // convert byte string into integer array + var bytes = msg.split('').map(v=>v.charCodeAt(0)); + + if (bytes.length < 5) { + throw 'message too short'; + } + + // first 5 bytes is gRPC frame (compression + length) + var head = bytes.splice(0, 5); + + // ensure we have proper message length + var len = (head[1] << 24) + + (head[2] << 16) + + (head[3] << 8) + + head[4]; + + if (len != bytes.length) { + throw 'header length mismatch'; + } + + // invoke protobufjs to decode message + var response = pb.helloworld.HelloReply.decode(bytes); + + console.log('Reply is:' + response.message); + + + +

+ +

+ + +This example uses a library for generation and parsing of DNS packets. +This a case worth considering because the library and its dependencies +use modern language constructions not yet supported by njs. +In turn, this requires from us an extra step: transpiling the source code. + + + +Additional node packages are needed: + +$ npm install @babel/core @babel/cli @babel/preset-env babel-loader +$ npm install webpack webpack-cli +$ npm install buffer +$ npm install dns-packet + + +The configuration file, webpack.config.js: + +const path = require('path'); + +module.exports = { + entry: './load.js', + mode: 'production', + output: { + filename: 'wp_out.js', + path: path.resolve(__dirname, 'dist'), + }, + optimization: { + minimize: false + }, + node: { + global: true, + }, + module : { + rules: [{ + test: /\.m?js$$/, + exclude: /(bower_components)/, + use: { + loader: 'babel-loader', + options: { + presets: ['@babel/preset-env'] + } + } + }] + } +}; + +Note we are using "production" mode. +In this mode webpack does not use "eval" construction +not supported by njs. + +The referenced load.js file is our entry point: + +global.dns = require('dns-packet') +global.Buffer = require('buffer/').Buffer + + +We start the same way, by producing a single file for the libraries: + + +$ npx browserify load.js -o bundle.js -d + + +Next, we process the file with webpack, which itself invokes babel: + + +$ npx webpack --config webpack.config.js + + +This command produces the dist/wp_out.js file, which is a +transpiled version of bundle.js. + +We need to concatenate it with code.js +that stores our code: + + +function set_buffer(dnsPacket) +{ +// create DNS packet bytes +var buf = dnsPacket.encode({ + type: 'query', + id: 1, + flags: dnsPacket.RECURSION_DESIRED, + questions: [{ + type: 'A', + name: 'google.com' + }] +}) + +return buf; +} + + +Note that in this example generated code is not wrapped into function and we +do not need to call it explicitely. +The result is in the "dist" directoty: + + +$ cat dist/wp_out.js code.js > njs_dns_bundle.js + + +Let's call our code at the end of a file: + +var b = setbuf(1); +console.log(b); + + +And execute it using node: + + +$ node ./njs_dns_bundle_final.js +Buffer [Uint8Array] [ + 0, 1, 1, 0, 0, 1, 0, 0, + 0, 0, 0, 0, 6, 103, 111, 111, + 103, 108, 101, 3, 99, 111, 109, 0, + 0, 1, 0, 1 +] + + +Make sure this works as expected, and then run it with njs: + +$ njs ./njs_dns_bundle_final.js +Uint8Array [0,1,1,0,0,1,0,0,0,0,0,0,6,103,111,111,103,108,101,3,99,111,109,0,0,1,0,1] + + + + + +The response can be parsed as follows: + +function parse_response(buf) +{ + var bytes = buf.split('').map(v=>v.charCodeAt(0)); + + var b = global.Buffer.from(bytes); + + var packet = dnsPacket.decode(b); + + var resolved_name = packet.answers[0].name; + + // expected name is 'google.com', according to our request above +} + + + + +

+ +