Mercurial > hg > nginx-site

comparison xml/en/docs/njs/node_modules.xml @ 2480:73d254c3376d

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Added the "Using node modules with njs" article.
author Vladimir Homutov <vl@nginx.com>
date Tue, 21 Jan 2020 17:10:35 +0300
parents
children 8ed243471444
comparison
equal deleted inserted replaced
2479:b5138cd3321d 2480:73d254c3376d
1 <?xml version="1.0"?>
2
3 <!--
4 Copyright (C) Nginx, Inc.
5 -->
6
7 <!DOCTYPE article SYSTEM "../../../../dtd/article.dtd">
8
9 <article name="Using node modules with njs"
10 link="/en/docs/njs/node_modules.html"
11 lang="en"
12 rev="3">
13
14 <section id="intro" name="Introduction">
15
16 <para>
17 Often, a developer wants to use 3rd-party code, usually available as a library
18 of some kind.
19 In the Javascript world, the concept of a module is relatively new, so there
20 was no standard until recently.
21 Many platforms (browsers) still don't support modules, which makes code
22 reuse harder.
23 The njs does not (yet) support modules, too.
24 This article describes ways to overcome this limitation, using the
25 <link url="https://nodejs.org/">Node.js</link> ecosystem as an example.
26 </para>
27
28 <note>
29 Examples in this article use features that appeared in
30 <link doc="index.xml">njs</link>
31 <link doc="changes.xml" id="njs0.3.8">0.3.8</link>
32 </note>
33
34 <para>
35 There is a number of issues that may arise when 3rd-party code is added to njs:
36
37 <list type="bullet">
38
39 <listitem>Multiple files that reference each other, and their
40 dependencies</listitem>
41
42 <listitem>Platform-specific APIs</listitem>
43
44 <listitem>Modern standard language constructions</listitem>
45
46 </list>
47
48 </para>
49
50 <para>
51 The good news is that such problems are not something new or
52 specific to njs.
53 Javascript developers face them daily when trying to support multiple
54 disparate platforms with very different properties.
55 There are instruments designed to resolve the above-mentioned issues.
56 </para>
57
58 <para>
59
60 <list type="bullet">
61
62 <listitem>
63 Multiple files that reference each other, and their dependencies
64 <para>
65 This can be solved by merging all the interdependent code into a single file.
66 Tools like
67 <link url="http://browserify.org/">browserify</link> or
68 <link url="https://webpack.js.org/">webpack</link>
69 accept an entire project and produce a single file containing
70 your code and all the dependencies.
71 </para>
72 </listitem>
73
74 <listitem>
75 Platform-specific APIs
76 <para>
77 You can use multiple libraries that implement such APIs in a platform-agnostic
78 manner (at the expense of performance, though).
79 Particular features can also be implemented using the
80 <link url="https://polyfill.io/v3/">polyfill</link> approach.
81 </para>
82 </listitem>
83
84 <listitem>
85 Modern standard language constructions
86 <para>
87 Such code can be transpiled: this means performing a number of transformations
88 that rewrite newer language features in accordance with an older standard.
89 For example, <link url="https://babeljs.io/"> babel</link> project can
90 be used to this purpose.
91 </para>
92 </listitem>
93
94 </list>
95
96 </para>
97
98
99 <para>
100 In this guide, we will use two relatively large npm-hosted libraries:
101
102 <list type="bullet">
103
104 <listitem>
105 <link url="https://www.npmjs.com/package/protobufjs">protobufjs</link> -
106 a library for creating and parsing protobuf messages used by the
107 <link url="https://grpc.io/">gRPC</link> protocol.
108
109 </listitem>
110
111 <listitem>
112 <link url="https://www.npmjs.com/package/dns-packet">dns-packet</link> -
113 a library for processing DNS protocol packets.
114 </listitem>
115
116 </list>
117
118 </para>
119
120 </section>
121
122 <section id="environment" name="Environment">
123
124 <para>
125
126 <note>
127 This document mostly employs a generic approach and AVOIDS specific best
128 practice advices concerning Node.js and the rapidly evolving JavaScript
129 ecosystem.
130 Make sure to consult the corresponding package's manual BEFORE following the
131 steps suggested here.
132 </note>
133
134 First (assuming Node.js is installed and operational), let's create an
135 empty project and install some dependencies; the commands below assume we're
136 in the working directory:
137
138 <example>
139 $ mkdir my_project &amp;&amp; cd my_project
140 $ npx license choose_your_license_here > LICENSE
141 $ npx gitignore node
142
143 $ cat &gt; package.json &lt;&lt;EOF
144 {
145 "name": "foobar",
146 "version": "0.0.1",
147 "description": "",
148 "main": "index.js",
149 "keywords": [],
150 "author": "somename &lt;some.email@example.com&gt; (https://example.com)",
151 "license": "some_license_here",
152 "private": true,
153 "scripts": {
154 "test": "echo \"Error: no test specified\" &amp;&amp; exit 1"
155 }
156 }
157 EOF
158 $ npm init -y
159 $ npm install browserify
160 </example>
161 </para>
162
163 </section>
164
165 <section id="protobuf" name="Protobujfs">
166
167 <para>
168 The library provides a parser for the <literal>.proto</literal> interface
169 definitions and a code generator for message parsing and generation.
170 </para>
171
172 <para>
173 In this example, we will use the
174 <link url="https://github.com/grpc/grpc/blob/master/examples/protos/helloworld.proto">helloworld.proto</link>
175 file from the gRPC examples.
176 Our goal is to create two messages: <literal>HelloRequest</literal> and
177 <literal>HelloResponse</literal>.
178 We will use the
179 <link url="https://github.com/protobufjs/protobuf.js/blob/master/README.md#reflection-vs-static-code">static</link>
180 mode of protobufjs instead of dynamically generating classes, because
181 njs doesn't support adding new functions dynamically due to security
182 considerations.
183 </para>
184
185 <para>
186 Next, the library is installed and javascript code implementing
187 message marshalling is generated from the protocol definition:
188 <example>
189 $ npm install protobufjs
190 $ npx pbjs -t static-module helloworld.proto > static.js
191 </example>
192
193 </para>
194
195 <para>
196 Thus, the <literal>static.js</literal> file becomes our new dependency, storing
197 all the code we need to implement message processing.
198 The <literal>set_buffer()</literal> function contains code that uses the
199 library to create a buffer with the serialized <literal>HelloRequest</literal>
200 message.
201 The code resides in the <literal>code.js</literal> file:
202
203 <example>
204 var pb = require('./static.js');
205
206 // Example usage of protobuf library: prepare a buffer to send
207 function set_buffer(pb)
208 {
209 // set fields of gRPC payload
210 var payload = { name: "TestString" };
211
212 // create an object
213 var message = pb.helloworld.HelloRequest.create(payload);
214
215 // serialize object to buffer
216 var buffer = pb.helloworld.HelloRequest.encode(message).finish();
217
218 var n = buffer.length;
219
220 var frame = new Uint8Array(5 + buffer.length);
221
222 frame[0] = 0; // 'compressed' flag
223 frame[1] = (n &amp; 0xFF000000) &gt;&gt;&gt; 24; // length: uint32 in network order
224 frame[2] = (n &amp; 0x00FF0000) &gt;&gt;&gt; 16;
225 frame[3] = (n &amp; 0x0000FF00) &gt;&gt;&gt; 8;
226 frame[4] = (n &amp; 0x000000FF) &gt;&gt;&gt; 0;
227
228 frame.set(buffer, 5);
229
230 return frame;
231 }
232
233 var frame = set_buffer(pb);
234 </example>
235 </para>
236
237 <para>
238 To ensure it works, we execute the code using node:
239
240 <example>
241 $ node ./code.js
242 Uint8Array [
243 0, 0, 0, 0, 12, 10,
244 10, 84, 101, 115, 116, 83,
245 116, 114, 105, 110, 103
246 ]
247 </example>
248
249 You can see thet this got us a properly encoded <literal>gRPC</literal> frame.
250 Now let's run it with njs:
251
252 <example>
253 $ njs ./code.js
254 Thrown:
255 Error: Cannot find module "./static.js"
256 at require (native)
257 at main (native)
258 </example>
259 </para>
260
261 <para>
262 Modules are not supported, so we've received an exception.
263 To overcome this issue, let's use <literal>browserify</literal>
264 or other similar tool.
265 </para>
266
267 <para>
268 An attempt to process our existing <literal>code.js</literal> file will result
269 in a bunch of JS code that is supposed to run in a browser, i.e. immediately
270 upon loading.
271 This isn't something we actually want.
272 Instead, we want to have an exported function that
273 can be referenced from the nginx configuration.
274 This requires some wrapper code.
275
276 <note>
277 In this guide, we use njs cli in all examples for the sake of simplicity.
278 In real life, you will be using nginx njs module to run your code.
279 </note>
280
281 </para>
282
283 <para>
284 The <literal>load.js</literal> file contains the library-loading code that
285 stores its handle in the global namespace:
286 <example>
287 global.hello = require('./static.js');
288 </example>
289 This code will be replaced with merged content.
290 Our code will be using the "<literal>global.hello</literal>" handle to access
291 the library.
292 </para>
293
294 <para>
295 Next, we process it with browserify to get all dependencies into a single file:
296 <example>
297 $ npx browserify load.js -o bundle.js -d
298 </example>
299
300 The result is huge file that contains all our dependencies:
301
302 <example>
303 (function(){function......
304 ...
305 ...
306 },{"protobufjs/minimal":9}]},{},[1])
307 //# sourceMappingURL..............
308 </example>
309
310 To get final "<literal>njs_bundle.js</literal>" file we concatenate
311 "<literal>bundle.js</literal>" and the following code:
312
313 <example>
314 // Example usage of protobuf library: prepare a buffer to send
315 function set_buffer(pb)
316 {
317 // set fields of gRPC payload
318 var payload = { name: "TestString" };
319
320 // create an object
321 var message = pb.helloworld.HelloRequest.create(payload);
322
323 // serialize object to buffer
324 var buffer = pb.helloworld.HelloRequest.encode(message).finish();
325
326 var n = buffer.length;
327
328 var frame = new Uint8Array(5 + buffer.length);
329
330 frame[0] = 0; // 'compressed' flag
331 frame[1] = (n &amp; 0xFF000000) &gt;&gt;&gt; 24; // length: uint32 in network order
332 frame[2] = (n &amp; 0x00FF0000) &gt;&gt;&gt; 16;
333 frame[3] = (n &amp; 0x0000FF00) &gt;&gt;&gt; 8;
334 frame[4] = (n &amp; 0x000000FF) &gt;&gt;&gt; 0;
335
336 frame.set(buffer, 5);
337
338 return frame;
339 }
340
341 // functions to be called from outside
342 function setbuf()
343 {
344 return set_buffer(global.hello);
345 }
346
347 // call the code
348 var frame = setbuf();
349 console.log(frame);
350 </example>
351
352 Let's run the file using node to make sure things still work:
353 <example>
354 $ node ./njs_bundle.js
355 Uint8Array [
356 0, 0, 0, 0, 12, 10,
357 10, 84, 101, 115, 116, 83,
358 116, 114, 105, 110, 103
359 ]
360 </example>
361
362 Now let's proceed further with njs:
363
364 <example>
365 $ /njs ./njs_bundle.js
366 Uint8Array [0,0,0,0,12,10,10,84,101,115,116,83,116,114,105,110,103]
367 </example>
368
369 The last thing will be to use njs-specific API to convert
370 array into byte string, so it could be usable by nginx module.
371 We can add the following snippet before the last line:
372 <example>
373 if (global.njs) {
374 return String.bytesFrom(frame)
375 }
376 </example>
377
378 Finally, we got it working:
379
380 <example>
381 $ njs ./njs_bundle.js |hexdump -C
382 00000000 00 00 00 00 0c 0a 0a 54 65 73 74 53 74 72 69 6e |.......TestStrin|
383 00000010 67 0a |g.|
384 00000012
385 </example>
386 This is the intended result.
387 Response parsing can be implemented similarly:
388 <example>
389 function parse_msg(pb, msg)
390 {
391 // convert byte string into integer array
392 var bytes = msg.split('').map(v=>v.charCodeAt(0));
393
394 if (bytes.length &lt; 5) {
395 throw 'message too short';
396 }
397
398 // first 5 bytes is gRPC frame (compression + length)
399 var head = bytes.splice(0, 5);
400
401 // ensure we have proper message length
402 var len = (head[1] &lt;&lt; 24)
403 + (head[2] &lt;&lt; 16)
404 + (head[3] &lt;&lt; 8)
405 + head[4];
406
407 if (len != bytes.length) {
408 throw 'header length mismatch';
409 }
410
411 // invoke protobufjs to decode message
412 var response = pb.helloworld.HelloReply.decode(bytes);
413
414 console.log('Reply is:' + response.message);
415 </example>
416 </para>
417
418 </section>
419
420 <section id="dnspacket" name="DNS-packet">
421
422 <para>
423 This example uses a library for generation and parsing of DNS packets.
424 This a case worth considering because the library and its dependencies
425 use modern language constructions not yet supported by njs.
426 In turn, this requires from us an extra step: transpiling the source code.
427 </para>
428
429 <para>
430 Additional node packages are needed:
431 <example>
432 $ npm install @babel/core @babel/cli @babel/preset-env babel-loader
433 $ npm install webpack webpack-cli
434 $ npm install buffer
435 $ npm install dns-packet
436 </example>
437
438 The configuration file, webpack.config.js:
439 <example>
440 const path = require('path');
441
442 module.exports = {
443 entry: './load.js',
444 mode: 'production',
445 output: {
446 filename: 'wp_out.js',
447 path: path.resolve(__dirname, 'dist'),
448 },
449 optimization: {
450 minimize: false
451 },
452 node: {
453 global: true,
454 },
455 module : {
456 rules: [{
457 test: /\.m?js$$/,
458 exclude: /(bower_components)/,
459 use: {
460 loader: 'babel-loader',
461 options: {
462 presets: ['@babel/preset-env']
463 }
464 }
465 }]
466 }
467 };
468 </example>
469 Note we are using "<literal>production</literal>" mode.
470 In this mode webpack does not use "<literal>eval</literal>" construction
471 not supported by njs.
472
473 The referenced <literal>load.js</literal> file is our entry point:
474 <example>
475 global.dns = require('dns-packet')
476 global.Buffer = require('buffer/').Buffer
477 </example>
478
479 We start the same way, by producing a single file for the libraries:
480
481 <example>
482 $ npx browserify load.js -o bundle.js -d
483 </example>
484
485 Next, we process the file with webpack, which itself invokes babel:
486
487 <example>
488 $ npx webpack --config webpack.config.js
489 </example>
490
491 This command produces the <literal>dist/wp_out.js</literal> file, which is a
492 transpiled version of <literal>bundle.js</literal>.
493
494 We need to concatenate it with <literal>code.js</literal>
495 that stores our code:
496
497 <example>
498 function set_buffer(dnsPacket)
499 {
500 // create DNS packet bytes
501 var buf = dnsPacket.encode({
502 type: 'query',
503 id: 1,
504 flags: dnsPacket.RECURSION_DESIRED,
505 questions: [{
506 type: 'A',
507 name: 'google.com'
508 }]
509 })
510
511 return buf;
512 }
513 </example>
514
515 Note that in this example generated code is not wrapped into function and we
516 do not need to call it explicitely.
517 The result is in the "<literal>dist</literal>" directoty:
518
519 <example>
520 $ cat dist/wp_out.js code.js > njs_dns_bundle.js
521 </example>
522
523 Let's call our code at the end of a file:
524 <example>
525 var b = setbuf(1);
526 console.log(b);
527 </example>
528
529 And execute it using node:
530
531 <example>
532 $ node ./njs_dns_bundle_final.js
533 Buffer [Uint8Array] [
534 0, 1, 1, 0, 0, 1, 0, 0,
535 0, 0, 0, 0, 6, 103, 111, 111,
536 103, 108, 101, 3, 99, 111, 109, 0,
537 0, 1, 0, 1
538 ]
539 </example>
540
541 Make sure this works as expected, and then run it with njs:
542 <example>
543 $ njs ./njs_dns_bundle_final.js
544 Uint8Array [0,1,1,0,0,1,0,0,0,0,0,0,6,103,111,111,103,108,101,3,99,111,109,0,0,1,0,1]
545 </example>
546
547 </para>
548
549 <para>
550 The response can be parsed as follows:
551 <example>
552 function parse_response(buf)
553 {
554 var bytes = buf.split('').map(v=>v.charCodeAt(0));
555
556 var b = global.Buffer.from(bytes);
557
558 var packet = dnsPacket.decode(b);
559
560 var resolved_name = packet.answers[0].name;
561
562 // expected name is 'google.com', according to our request above
563 }
564 </example>
565
566 </para>
567
568 </section>
569
570 </article>