Mercurial > hg > nginx-site
view xml/en/docs/njs/node_modules.xml @ 2632:52500ccb82c1
Updated docs for the upcoming NGINX Plus release.
author | Yaroslav Zhuravlev <yar@nginx.com> |
---|---|
date | Fri, 04 Dec 2020 13:34:05 +0000 |
parents | 5cd72684e5b8 |
children |
line wrap: on
line source
<?xml version="1.0"?> <!-- Copyright (C) Nginx, Inc. --> <!DOCTYPE article SYSTEM "../../../../dtd/article.dtd"> <article name="Using node modules with njs" link="/en/docs/njs/node_modules.html" lang="en" rev="6"> <section id="intro"> <para> Often, a developer wants to use 3rd-party code, usually available as a library of some kind. In the JavaScript world, the concept of a module is relatively new, so there was no standard until recently. Many platforms (browsers) still don't support modules, which makes code reuse harder. This article describes ways to reuse <link url="https://nodejs.org/">Node.js</link> code in njs. </para> <note> Examples in this article use features that appeared in <link doc="index.xml">njs</link> <link doc="changes.xml" id="njs0.3.8">0.3.8</link> </note> <para> There is a number of issues that may arise when 3rd-party code is added to njs: <list type="bullet"> <listitem> Multiple files that reference each other and their dependencies </listitem> <listitem> Platform-specific APIs </listitem> <listitem> Modern standard language constructions </listitem> </list> </para> <para> The good news is that such problems are not something new or specific to njs. JavaScript developers face them daily when trying to support multiple disparate platforms with very different properties. There are instruments designed to resolve the above-mentioned issues. <list type="bullet"> <listitem> Multiple files that reference each other, and their dependencies <para> This can be solved by merging all the interdependent code into a single file. Tools like <link url="http://browserify.org/">browserify</link> or <link url="https://webpack.js.org/">webpack</link> accept an entire project and produce a single file containing your code and all the dependencies. </para> </listitem> <listitem> Platform-specific APIs <para> You can use multiple libraries that implement such APIs in a platform-agnostic manner (at the expense of performance, though). Particular features can also be implemented using the <link url="https://polyfill.io/v3/">polyfill</link> approach. </para> </listitem> <listitem> Modern standard language constructions <para> Such code can be transpiled: this means performing a number of transformations that rewrite newer language features in accordance with an older standard. For example, <link url="https://babeljs.io/"> babel</link> project can be used to this purpose. </para> </listitem> </list> </para> <para> In this guide, we will use two relatively large npm-hosted libraries: <list type="bullet"> <listitem> <link url="https://www.npmjs.com/package/protobufjs">protobufjs</link>— a library for creating and parsing protobuf messages used by the <link url="https://grpc.io/">gRPC</link> protocol </listitem> <listitem> <link url="https://www.npmjs.com/package/dns-packet">dns-packet</link>— a library for processing DNS protocol packets </listitem> </list> </para> </section> <section id="environment" name="Environment"> <para> <note> This document mostly employs a generic approach and avoids specific best practice advices concerning Node.js and JavaScript. Make sure to consult the corresponding package's manual before following the steps suggested here. </note> First (assuming Node.js is installed and operational), let's create an empty project and install some dependencies; the commands below assume we are in the working directory: <example> $ mkdir my_project && cd my_project $ npx license choose_your_license_here > LICENSE $ npx gitignore node $ cat > package.json <<EOF { "name": "foobar", "version": "0.0.1", "description": "", "main": "index.js", "keywords": [], "author": "somename <some.email@example.com> (https://example.com)", "license": "some_license_here", "private": true, "scripts": { "test": "echo \"Error: no test specified\" && exit 1" } } EOF $ npm init -y $ npm install browserify </example> </para> </section> <section id="protobuf" name="Protobufjs"> <para> The library provides a parser for the <literal>.proto</literal> interface definitions and a code generator for message parsing and generation. </para> <para> In this example, we will use the <link url="https://github.com/grpc/grpc/blob/master/examples/protos/helloworld.proto">helloworld.proto</link> file from the gRPC examples. Our goal is to create two messages: <literal>HelloRequest</literal> and <literal>HelloResponse</literal>. We will use the <link url="https://github.com/protobufjs/protobuf.js/blob/master/README.md#reflection-vs-static-code">static</link> mode of protobufjs instead of dynamically generating classes, because njs doesn't support adding new functions dynamically due to security considerations. </para> <para> Next, the library is installed and the JavaScript code implementing message marshalling is generated from the protocol definition: <example> $ npm install protobufjs $ npx pbjs -t static-module helloworld.proto > static.js </example> </para> <para> Thus, the <literal>static.js</literal> file becomes our new dependency, storing all the code we need to implement message processing. The <literal>set_buffer()</literal> function contains code that uses the library to create a buffer with the serialized <literal>HelloRequest</literal> message. The code resides in the <literal>code.js</literal> file: <example> var pb = require('./static.js'); // Example usage of protobuf library: prepare a buffer to send function set_buffer(pb) { // set fields of gRPC payload var payload = { name: "TestString" }; // create an object var message = pb.helloworld.HelloRequest.create(payload); // serialize object to buffer var buffer = pb.helloworld.HelloRequest.encode(message).finish(); var n = buffer.length; var frame = new Uint8Array(5 + buffer.length); frame[0] = 0; // 'compressed' flag frame[1] = (n & 0xFF000000) >>> 24; // length: uint32 in network byte order frame[2] = (n & 0x00FF0000) >>> 16; frame[3] = (n & 0x0000FF00) >>> 8; frame[4] = (n & 0x000000FF) >>> 0; frame.set(buffer, 5); return frame; } var frame = set_buffer(pb); </example> </para> <para> To ensure it works, we execute the code using node: <example> $ node ./code.js Uint8Array [ 0, 0, 0, 0, 12, 10, 10, 84, 101, 115, 116, 83, 116, 114, 105, 110, 103 ] </example> You can see that this got us a properly encoded <literal>gRPC</literal> frame. Now let's run it with njs: <example> $ njs ./code.js Thrown: Error: Cannot find module "./static.js" at require (native) at main (native) </example> </para> <para> Modules are not supported, so we've received an exception. To overcome this issue, let's use <literal>browserify</literal> or other similar tool. </para> <para> An attempt to process our existing <literal>code.js</literal> file will result in a bunch of JS code that is supposed to run in a browser, i.e. immediately upon loading. This isn't something we actually want. Instead, we want to have an exported function that can be referenced from the nginx configuration. This requires some wrapper code. <note> In this guide, we use njs <link doc="cli.xml">cli</link> in all examples for the sake of simplicity. In real life, you will be using nginx njs module to run your code. </note> </para> <para> The <literal>load.js</literal> file contains the library-loading code that stores its handle in the global namespace: <example> global.hello = require('./static.js'); </example> This code will be replaced with merged content. Our code will be using the "<literal>global.hello</literal>" handle to access the library. </para> <para> Next, we process it with <literal>browserify</literal> to get all dependencies into a single file: <example> $ npx browserify load.js -o bundle.js -d </example> The result is a huge file that contains all our dependencies: <example> (function(){function...... ... ... },{"protobufjs/minimal":9}]},{},[1]) //# sourceMappingURL.............. </example> To get final "<literal>njs_bundle.js</literal>" file we concatenate "<literal>bundle.js</literal>" and the following code: <example> // Example usage of protobuf library: prepare a buffer to send function set_buffer(pb) { // set fields of gRPC payload var payload = { name: "TestString" }; // create an object var message = pb.helloworld.HelloRequest.create(payload); // serialize object to buffer var buffer = pb.helloworld.HelloRequest.encode(message).finish(); var n = buffer.length; var frame = new Uint8Array(5 + buffer.length); frame[0] = 0; // 'compressed' flag frame[1] = (n & 0xFF000000) >>> 24; // length: uint32 in network byte order frame[2] = (n & 0x00FF0000) >>> 16; frame[3] = (n & 0x0000FF00) >>> 8; frame[4] = (n & 0x000000FF) >>> 0; frame.set(buffer, 5); return frame; } // functions to be called from outside function setbuf() { return set_buffer(global.hello); } // call the code var frame = setbuf(); console.log(frame); </example> Let's run the file using node to make sure things still work: <example> $ node ./njs_bundle.js Uint8Array [ 0, 0, 0, 0, 12, 10, 10, 84, 101, 115, 116, 83, 116, 114, 105, 110, 103 ] </example> Now let's proceed further with njs: <example> $ njs ./njs_bundle.js Uint8Array [0,0,0,0,12,10,10,84,101,115,116,83,116,114,105,110,103] </example> The last thing will be to use njs-specific API to convert array into byte string, so it could be usable by nginx module. We can add the following snippet before the line <literal>return frame; }</literal>: <example> if (global.njs) { return String.bytesFrom(frame) } </example> Finally, we got it working: <example> $ njs ./njs_bundle.js |hexdump -C 00000000 00 00 00 00 0c 0a 0a 54 65 73 74 53 74 72 69 6e |.......TestStrin| 00000010 67 0a |g.| 00000012 </example> This is the intended result. Response parsing can be implemented similarly: <example> function parse_msg(pb, msg) { // convert byte string into integer array var bytes = msg.split('').map(v=>v.charCodeAt(0)); if (bytes.length < 5) { throw 'message too short'; } // first 5 bytes is gRPC frame (compression + length) var head = bytes.splice(0, 5); // ensure we have proper message length var len = (head[1] << 24) + (head[2] << 16) + (head[3] << 8) + head[4]; if (len != bytes.length) { throw 'header length mismatch'; } // invoke protobufjs to decode message var response = pb.helloworld.HelloReply.decode(bytes); console.log('Reply is:' + response.message); } </example> </para> </section> <section id="dnspacket" name="DNS-packet"> <para> This example uses a library for generation and parsing of DNS packets. This a case worth considering because the library and its dependencies use modern language constructions not yet supported by njs. In turn, this requires from us an extra step: transpiling the source code. </para> <para> Additional node packages are needed: <example> $ npm install @babel/core @babel/cli @babel/preset-env babel-loader $ npm install webpack webpack-cli $ npm install buffer $ npm install dns-packet </example> The configuration file, webpack.config.js: <example> const path = require('path'); module.exports = { entry: './load.js', mode: 'production', output: { filename: 'wp_out.js', path: path.resolve(__dirname, 'dist'), }, optimization: { minimize: false }, node: { global: true, }, module : { rules: [{ test: /\.m?js$$/, exclude: /(bower_components)/, use: { loader: 'babel-loader', options: { presets: ['@babel/preset-env'] } } }] } }; </example> Note we are using "<literal>production</literal>" mode. In this mode webpack does not use "<literal>eval</literal>" construction not supported by njs. The referenced <literal>load.js</literal> file is our entry point: <example> global.dns = require('dns-packet') global.Buffer = require('buffer/').Buffer </example> We start the same way, by producing a single file for the libraries: <example> $ npx browserify load.js -o bundle.js -d </example> Next, we process the file with webpack, which itself invokes babel: <example> $ npx webpack --config webpack.config.js </example> This command produces the <literal>dist/wp_out.js</literal> file, which is a transpiled version of <literal>bundle.js</literal>. We need to concatenate it with <literal>code.js</literal> that stores our code: <example> function set_buffer(dnsPacket) { // create DNS packet bytes var buf = dnsPacket.encode({ type: 'query', id: 1, flags: dnsPacket.RECURSION_DESIRED, questions: [{ type: 'A', name: 'google.com' }] }) return buf; } </example> Note that in this example generated code is not wrapped into function and we do not need to call it explicitly. The result is in the "<literal>dist</literal>" directory: <example> $ cat dist/wp_out.js code.js > njs_dns_bundle.js </example> Let's call our code at the end of a file: <example> var b = set_buffer(global.dns); console.log(b); </example> And execute it using node: <example> $ node ./njs_dns_bundle_final.js Buffer [Uint8Array] [ 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 6, 103, 111, 111, 103, 108, 101, 3, 99, 111, 109, 0, 0, 1, 0, 1 ] </example> Make sure this works as expected, and then run it with njs: <example> $ njs ./njs_dns_bundle_final.js Uint8Array [0,1,1,0,0,1,0,0,0,0,0,0,6,103,111,111,103,108,101,3,99,111,109,0,0,1,0,1] </example> </para> <para> The response can be parsed as follows: <example> function parse_response(buf) { var bytes = buf.split('').map(v=>v.charCodeAt(0)); var b = global.Buffer.from(bytes); var packet = dnsPacket.decode(b); var resolved_name = packet.answers[0].name; // expected name is 'google.com', according to our request above } </example> </para> </section> </article>