changeset 2480:73d254c3376d

Added the "Using node modules with njs" article.
author Vladimir Homutov <vl@nginx.com>
date Tue, 21 Jan 2020 17:10:35 +0300
parents b5138cd3321d
children 8ed243471444
files xml/en/GNUmakefile xml/en/docs/njs/index.xml xml/en/docs/njs/node_modules.xml
diffstat 3 files changed, 575 insertions(+), 0 deletions(-) [+]
line wrap: on
line diff
--- a/xml/en/GNUmakefile	Tue Jan 21 17:00:28 2020 +0300
+++ b/xml/en/GNUmakefile	Tue Jan 21 17:10:35 2020 +0300
@@ -129,6 +129,7 @@
 		njs/examples						\
 		njs/install						\
 		njs/reference						\
+		njs/node_modules					\
 
 TOP =									\
 		download						\
--- a/xml/en/docs/njs/index.xml	Tue Jan 21 17:00:28 2020 +0300
+++ b/xml/en/docs/njs/index.xml	Tue Jan 21 17:10:35 2020 +0300
@@ -50,6 +50,10 @@
 </listitem>
 
 <listitem>
+<link doc="node_modules.xml"/>
+</listitem>
+
+<listitem>
 <link doc="compatibility.xml"/>
 </listitem>
 
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/xml/en/docs/njs/node_modules.xml	Tue Jan 21 17:10:35 2020 +0300
@@ -0,0 +1,570 @@
+<?xml version="1.0"?>
+
+<!--
+  Copyright (C) Nginx, Inc.
+  -->
+
+<!DOCTYPE article SYSTEM "../../../../dtd/article.dtd">
+
+<article name="Using node modules with njs"
+        link="/en/docs/njs/node_modules.html"
+        lang="en"
+        rev="3">
+
+<section id="intro" name="Introduction">
+
+<para>
+Often, a developer wants to use 3rd-party code, usually available as a library
+of some kind.
+In the Javascript world, the concept of a module is relatively new, so there
+was no standard until recently.
+Many platforms (browsers) still don't support modules, which makes code
+reuse harder.
+The njs does not (yet) support modules, too.
+This article describes ways to overcome this limitation, using the
+<link url="https://nodejs.org/">Node.js</link> ecosystem as an example.
+</para>
+
+<note>
+Examples in this article use features that appeared in
+<link doc="index.xml">njs</link>
+<link doc="changes.xml" id="njs0.3.8">0.3.8</link>
+</note>
+
+<para>
+There is a number of issues that may arise when 3rd-party code is added to njs:
+
+<list type="bullet">
+
+<listitem>Multiple files that reference each other, and their
+dependencies</listitem>
+
+<listitem>Platform-specific APIs</listitem>
+
+<listitem>Modern standard language constructions</listitem>
+
+</list>
+
+</para>
+
+<para>
+The good news is that such problems are not something new or
+specific to njs.
+Javascript developers face them daily when trying to support multiple
+disparate platforms with very different properties.
+There are instruments designed to resolve the above-mentioned issues.
+</para>
+
+<para>
+
+<list type="bullet">
+
+<listitem>
+Multiple files that reference each other, and their dependencies
+<para>
+This can be solved by merging all the interdependent code into a single file.
+Tools like
+<link url="http://browserify.org/">browserify</link> or
+<link url="https://webpack.js.org/">webpack</link>
+accept an entire project and produce a single file containing
+your code and all the dependencies.
+</para>
+</listitem>
+
+<listitem>
+Platform-specific APIs
+<para>
+You can use multiple libraries that implement such APIs in a platform-agnostic
+manner (at the expense of performance, though).
+Particular features can also be implemented using the
+<link url="https://polyfill.io/v3/">polyfill</link> approach.
+</para>
+</listitem>
+
+<listitem>
+Modern standard language constructions
+<para>
+Such code can be transpiled: this means performing a number of transformations
+that rewrite newer language features in accordance with an older standard.
+For example, <link url="https://babeljs.io/"> babel</link> project can
+be used to this purpose.
+</para>
+</listitem>
+
+</list>
+
+</para>
+
+
+<para>
+In this guide, we will use two relatively large npm-hosted libraries:
+
+<list type="bullet">
+
+<listitem>
+<link url="https://www.npmjs.com/package/protobufjs">protobufjs</link> -
+a library for creating and parsing protobuf messages used by the
+<link url="https://grpc.io/">gRPC</link> protocol.
+
+</listitem>
+
+<listitem>
+<link url="https://www.npmjs.com/package/dns-packet">dns-packet</link> -
+a library for processing DNS protocol packets.
+</listitem>
+
+</list>
+
+</para>
+
+</section>
+
+<section id="environment" name="Environment">
+
+<para>
+
+<note>
+This document mostly employs a generic approach and AVOIDS specific best
+practice advices concerning Node.js and the rapidly evolving JavaScript
+ecosystem.
+Make sure to consult the corresponding package's manual BEFORE following the
+steps suggested here.
+</note>
+
+First (assuming Node.js is installed and operational), let's create an
+empty project and install some dependencies; the commands below assume we're
+in the working directory:
+
+<example>
+$ mkdir my_project &amp;&amp; cd my_project
+$ npx license choose_your_license_here > LICENSE
+$ npx gitignore node
+
+$ cat &gt; package.json &lt;&lt;EOF
+{
+  "name":        "foobar",
+  "version":     "0.0.1",
+  "description": "",
+  "main":        "index.js",
+  "keywords":    [],
+  "author":      "somename &lt;some.email@example.com&gt; (https://example.com)",
+  "license":     "some_license_here",
+  "private":     true,
+  "scripts": {
+    "test": "echo \"Error: no test specified\" &amp;&amp; exit 1"
+  }
+}
+EOF
+$ npm init -y
+$ npm install browserify
+</example>
+</para>
+
+</section>
+
+<section id="protobuf" name="Protobujfs">
+
+<para>
+The library provides a parser for the <literal>.proto</literal> interface
+definitions and a code generator for message parsing and generation.
+</para>
+
+<para>
+In this example, we will use the
+<link url="https://github.com/grpc/grpc/blob/master/examples/protos/helloworld.proto">helloworld.proto</link>
+file from the gRPC examples.
+Our goal is to create two messages: <literal>HelloRequest</literal> and
+<literal>HelloResponse</literal>.
+We will use the
+<link url="https://github.com/protobufjs/protobuf.js/blob/master/README.md#reflection-vs-static-code">static</link>
+mode of protobufjs instead of dynamically generating classes, because
+njs doesn't support adding new functions dynamically due to security
+considerations.
+</para>
+
+<para>
+Next, the library is installed and javascript code implementing
+message marshalling is generated from the protocol definition:
+<example>
+$ npm install protobufjs
+$ npx pbjs -t static-module helloworld.proto > static.js
+</example>
+
+</para>
+
+<para>
+Thus, the <literal>static.js</literal> file becomes our new dependency, storing
+all the code we need to implement message processing.
+The <literal>set_buffer()</literal> function contains code that uses the
+library to create a buffer with the serialized <literal>HelloRequest</literal>
+message.
+The code resides in the <literal>code.js</literal> file:
+
+<example>
+var pb = require('./static.js');
+
+// Example usage of protobuf library: prepare a buffer to send
+function set_buffer(pb)
+{
+    // set fields of gRPC payload
+    var payload = { name: "TestString" };
+
+    // create an object
+    var message = pb.helloworld.HelloRequest.create(payload);
+
+    // serialize object to buffer
+    var buffer = pb.helloworld.HelloRequest.encode(message).finish();
+
+    var n = buffer.length;
+
+    var frame = new Uint8Array(5 + buffer.length);
+
+    frame[0] = 0;                        // 'compressed' flag
+    frame[1] = (n &amp; 0xFF000000) &gt;&gt;&gt; 24;  // length: uint32 in network order
+    frame[2] = (n &amp; 0x00FF0000) &gt;&gt;&gt; 16;
+    frame[3] = (n &amp; 0x0000FF00) &gt;&gt;&gt;  8;
+    frame[4] = (n &amp; 0x000000FF) &gt;&gt;&gt;  0;
+
+    frame.set(buffer, 5);
+
+    return frame;
+}
+
+var frame = set_buffer(pb);
+</example>
+</para>
+
+<para>
+To ensure it works, we execute the code using node:
+
+<example>
+$ node ./code.js
+Uint8Array [
+    0,   0,   0,   0,  12, 10,
+   10,  84, 101, 115, 116, 83,
+  116, 114, 105, 110, 103
+]
+</example>
+
+You can see thet this got us a properly encoded <literal>gRPC</literal> frame.
+Now let's run it with njs:
+
+<example>
+$ njs ./code.js
+Thrown:
+Error: Cannot find module "./static.js"
+    at require (native)
+    at main (native)
+</example>
+</para>
+
+<para>
+Modules are not supported, so we've received an exception.
+To overcome this issue, let's use <literal>browserify</literal>
+or other similar tool.
+</para>
+
+<para>
+An attempt to process our existing <literal>code.js</literal> file will result
+in a bunch of JS code that is supposed to run in a browser, i.e. immediately
+upon loading.
+This isn't something we actually want.
+Instead, we want to have an exported function that
+can be referenced from the nginx configuration.
+This requires some wrapper code.
+
+<note>
+In this guide, we use njs cli in all examples for the sake of simplicity.
+In real life, you will be using nginx njs module to run your code.
+</note>
+
+</para>
+
+<para>
+The <literal>load.js</literal> file contains the library-loading code that
+stores its handle in the global namespace:
+<example>
+global.hello = require('./static.js');
+</example>
+This code will be replaced with merged content.
+Our code will be using the "<literal>global.hello</literal>" handle to access
+the library.
+</para>
+
+<para>
+Next, we process it with browserify to get all dependencies into a single file:
+<example>
+$ npx browserify load.js -o bundle.js -d
+</example>
+
+The result is huge file that contains all our dependencies:
+
+<example>
+(function(){function......
+...
+...
+},{"protobufjs/minimal":9}]},{},[1])
+//# sourceMappingURL..............
+</example>
+
+To get final "<literal>njs_bundle.js</literal>" file we concatenate
+"<literal>bundle.js</literal>" and the following code:
+
+<example>
+// Example usage of protobuf library: prepare a buffer to send
+function set_buffer(pb)
+{
+    // set fields of gRPC payload
+    var payload = { name: "TestString" };
+
+    // create an object
+    var message = pb.helloworld.HelloRequest.create(payload);
+
+    // serialize object to buffer
+    var buffer = pb.helloworld.HelloRequest.encode(message).finish();
+
+    var n = buffer.length;
+
+    var frame = new Uint8Array(5 + buffer.length);
+
+    frame[0] = 0;                        // 'compressed' flag
+    frame[1] = (n &amp; 0xFF000000) &gt;&gt;&gt; 24;  // length: uint32 in network order
+    frame[2] = (n &amp; 0x00FF0000) &gt;&gt;&gt; 16;
+    frame[3] = (n &amp; 0x0000FF00) &gt;&gt;&gt;  8;
+    frame[4] = (n &amp; 0x000000FF) &gt;&gt;&gt;  0;
+
+    frame.set(buffer, 5);
+
+    return frame;
+}
+
+// functions to be called from outside
+function setbuf()
+{
+    return set_buffer(global.hello);
+}
+
+// call the code
+var frame = setbuf();
+console.log(frame);
+</example>
+
+Let's run the file using node to make sure things still work:
+<example>
+$ node ./njs_bundle.js
+Uint8Array [
+    0,   0,   0,   0,  12, 10,
+   10,  84, 101, 115, 116, 83,
+  116, 114, 105, 110, 103
+]
+</example>
+
+Now let's proceed further with njs:
+
+<example>
+$ /njs ./njs_bundle.js
+Uint8Array [0,0,0,0,12,10,10,84,101,115,116,83,116,114,105,110,103]
+</example>
+
+The last thing will be to use njs-specific API to convert
+array into byte string, so it could be usable by nginx module.
+We can add the following snippet before the last line:
+<example>
+if (global.njs) {
+    return String.bytesFrom(frame)
+}
+</example>
+
+Finally, we got it working:
+
+<example>
+$ njs ./njs_bundle.js |hexdump -C
+00000000  00 00 00 00 0c 0a 0a 54  65 73 74 53 74 72 69 6e  |.......TestStrin|
+00000010  67 0a                                             |g.|
+00000012
+</example>
+This is the intended result.
+Response parsing can be implemented similarly:
+<example>
+function parse_msg(pb, msg)
+{
+    // convert byte string into integer array
+    var bytes = msg.split('').map(v=>v.charCodeAt(0));
+
+    if (bytes.length &lt; 5) {
+        throw 'message too short';
+    }
+
+    // first 5 bytes is gRPC frame (compression + length)
+    var head = bytes.splice(0, 5);
+
+    // ensure we have proper message length
+    var len = (head[1] &lt;&lt; 24)
+              + (head[2] &lt;&lt; 16)
+              + (head[3] &lt;&lt; 8)
+              + head[4];
+
+    if (len != bytes.length) {
+        throw 'header length mismatch';
+    }
+
+    // invoke protobufjs to decode message
+    var response = pb.helloworld.HelloReply.decode(bytes);
+
+    console.log('Reply is:' + response.message);
+</example>
+</para>
+
+</section>
+
+<section id="dnspacket" name="DNS-packet">
+
+<para>
+This example uses a library for generation and parsing of DNS packets.
+This a case worth considering because the library and its dependencies
+use modern language constructions not yet supported by njs.
+In turn, this requires from us an extra step: transpiling the source code.
+</para>
+
+<para>
+Additional node packages are needed:
+<example>
+$ npm install @babel/core @babel/cli @babel/preset-env babel-loader
+$ npm install webpack webpack-cli
+$ npm install buffer
+$ npm install dns-packet
+</example>
+
+The configuration file, webpack.config.js:
+<example>
+const path = require('path');
+
+module.exports = {
+    entry: './load.js',
+    mode: 'production',
+    output: {
+        filename: 'wp_out.js',
+        path: path.resolve(__dirname, 'dist'),
+    },
+    optimization: {
+        minimize: false
+    },
+    node: {
+        global: true,
+    },
+    module : {
+        rules: [{
+            test: /\.m?js$$/,
+            exclude: /(bower_components)/,
+            use: {
+                loader: 'babel-loader',
+                options: {
+                    presets: ['@babel/preset-env']
+                }
+            }
+        }]
+    }
+};
+</example>
+Note we are using "<literal>production</literal>" mode.
+In this mode webpack does not use "<literal>eval</literal>" construction
+not supported by njs.
+
+The referenced <literal>load.js</literal> file is our entry point:
+<example>
+global.dns = require('dns-packet')
+global.Buffer = require('buffer/').Buffer
+</example>
+
+We start the same way, by producing a single file for the libraries:
+
+<example>
+$ npx browserify load.js -o bundle.js -d
+</example>
+
+Next, we process the file with webpack, which itself invokes babel:
+
+<example>
+$ npx webpack --config webpack.config.js
+</example>
+
+This command produces the <literal>dist/wp_out.js</literal> file, which is a
+transpiled version of <literal>bundle.js</literal>.
+
+We need to concatenate it with <literal>code.js</literal>
+that stores our code:
+
+<example>
+function set_buffer(dnsPacket)
+{
+// create DNS packet bytes
+var buf = dnsPacket.encode({
+    type: 'query',
+    id: 1,
+    flags: dnsPacket.RECURSION_DESIRED,
+    questions: [{
+        type: 'A',
+        name: 'google.com'
+    }]
+})
+
+return buf;
+}
+</example>
+
+Note that in this example generated code is not wrapped into function and we
+do not need to call it explicitely.
+The result is in the "<literal>dist</literal>" directoty:
+
+<example>
+$ cat dist/wp_out.js code.js > njs_dns_bundle.js
+</example>
+
+Let's call our code at the end of a file:
+<example>
+var b = setbuf(1);
+console.log(b);
+</example>
+
+And execute it using node:
+
+<example>
+$ node ./njs_dns_bundle_final.js
+Buffer [Uint8Array] [
+    0,   1,   1, 0,  0,   1,   0,   0,
+    0,   0,   0, 0,  6, 103, 111, 111,
+  103, 108, 101, 3, 99, 111, 109,   0,
+    0,   1,   0, 1
+]
+</example>
+
+Make sure this works as expected, and then run it with njs:
+<example>
+$ njs ./njs_dns_bundle_final.js
+Uint8Array [0,1,1,0,0,1,0,0,0,0,0,0,6,103,111,111,103,108,101,3,99,111,109,0,0,1,0,1]
+</example>
+
+</para>
+
+<para>
+The response can be parsed as follows:
+<example>
+function parse_response(buf)
+{
+    var bytes = buf.split('').map(v=>v.charCodeAt(0));
+
+    var b = global.Buffer.from(bytes);
+
+    var packet = dnsPacket.decode(b);
+
+    var resolved_name = packet.answers[0].name;
+
+    // expected name is 'google.com', according to our request above
+}
+</example>
+
+</para>
+
+</section>
+
+</article>