![\"next\"](\"//images.ctfassets.net/hspc7zpa5cvq/5gcP7k5WIw7h4mSYuZIDMq/898ad62724d65c969e60d305e1e85d2e/next.png\")
In my previous blog post, I explored the differences, advantages, and disadvantages of three of the most popular Node.js frameworks: Express, Koa, and Hapi. In this blog post, I’m going to examine the differences between three more very popular frameworks: Next, Nuxt, and Nest. These three frameworks are server-side rendering, and they are closely related to React, Vue, and Angular (the three most widely used front-end frameworks), respectively.
\nThe comparison is based on:
\nNext is the most popular framework compared to the other two. It has more npm weekly downloads, GitHub stars and number of contributors.
\nNext.js is a React framework that lets you build server-side rendering and static web applications using React.
\nInstall it:
\nnpm install --save next react react-dom\nand add a script to your package.json like this:
\n{\n \"scripts\": {\n \"dev\": \"next\",\n \"build\": \"next build\",\n \"start\": \"next start\"\n }\n}\nAfter that, the file-system is the main API. Every .js file becomes a route that gets automatically processed and rendered.
\nPopulate ./pages/index.js inside your project:
function Home() {\n return <div>Hello world!</div>;\n}\n\nexport default Home;\nThen just run npm run dev and go to http://localhost:3000. To use another port, you can run npm run dev -- -p <your port here>.
To measure the performance, I used Apache Bench for benchmarking, which highlights how many requests per second the app is capable of serving. I also used lighthouse to audit performance, accessibility, best practices, and SEO.
\n
This is a basic Hello World app in Next.js. It handles 550.87 requests per second. This value is the result of dividing the number of requests by the total time taken. The average time spent per request is 18.153 ms.
\nCompared to the other two frameworks, Next.js scored better overall than Nuxt.js but worse than Nest.js
\n![\"next1\"](\"//images.ctfassets.net/hspc7zpa5cvq/6SWp0mED1Nb9fXIBM7WwXC/4982654aa94d5d89fc6a0053b1ed90d1/next1.png\")
In the report provided by lighthouse, we can see that the performance, accessibility, best practices, and SEO scores are all above 70, which is good, but compared with the other two frameworks, it had the lowest score for Performance and has the highest score in Best Practices.
\nThe Next.js community communicates through chat, slack, issues and pull request on GitHub.
\nAlso, in the repo awesome-nextjs, there is a list of essentials, articles, boilerplates, extensions, apps, books, and videos that are useful for developers using Next.js
\nNuxt is a Vue.js Meta Framework to create complex, fast, and universal web applications quickly.
\nInstall it:
\n$ npm i nuxt\nTo create a basic app:
\n$ npx create-nuxt-app <project-name>\nYou can start directly with the CLI create-nuxt-app for the latest updates.\nOr you can start by using one of the starter templates:\nstarter: Basic Nuxt.js project template\nexpress: Nuxt.js + Express\nkoa: Nuxt.js + Koa\nadonuxt: Nuxt.js + AdonisJS\nmicro: Nuxt.js + Micro\nnuxtent: Nuxt.js + Nuxtent module for content heavy sites
\nThis is the most basic example of a “Hello World!” app on Nuxt:
\n<template>\n <div>\n <h1>Hello world!</h1>\n <NLink to=\"/about\">\n About Page\n </NLink>\n </div>\n</template>\n\n<script>\nexport default {\n head: {\n title: 'Home page'\n }\n}\n</script>\nthis.items[key]=value or adding a new data property. ![\"nuxt\"](\"//images.ctfassets.net/hspc7zpa5cvq/77aAB8VVmQi8Kif9gTCQ4U/2fd522c337487dce659994e37e8edbeb/nuxt.png\")
This is a basic Hello World app in Nuxt.js. It handles 190.05 requests per second. The average time spent per request is 52.619 ms. On this metric, Nuxt.js performs the worst compared to the other two frameworks.
\n![\"nuxt1\"](\"//images.ctfassets.net/hspc7zpa5cvq/7hS5nyqzb8UAFd9NidP5sQ/e0cf7d7936142535c0c8aedaad038d81/nuxt1.png\")
Nuxt.js has the highest score in three of the four measures; performance, accesibility and SEO.
\nThere is a GitHub organization where you can find modules and projects from the Nuxt.js community. There is also a curated list of awesome things related to Nuxt.js awesome-nuxt including Modules, tools, mention of Nuxt.js, showcase, tutorials, blogs, books, starter template, official examples, and projects using Nuxt.js.
\nThe community communicates through Gitter Chat Room, Telegram, Russian community, Discord, Twitter and YouTube Channel
\nA progressive Node.js framework for building efficient, scalable, and enterprise-grade server-side applications on top of TypeScript and JavaScript (ES6, ES7, ES8), Nest is heavily inspired by Angular.
\nNest is a framework for building efficient, scalable Node.js server-side applications. It uses modern JavaScript, is built with TypeScript (preserves compatibility with pure JavaScript) and combines elements of OOP (Object Oriented Programming), FP (Functional Programming), and FRP (Functional Reactive Programming).
\nUnder the hood, Nest makes use of Express, but also provides compatibility with a wide range of other libraries, like e.g. Fastify, allowing for easy use of the myriad third-party plugins which are available.
\nInstall it:
\n$ npm i @nestjs/cli\n$ nest new project-name\nAlternatively, to install the TypeScript starter project with Git:
\n$ git clone https://github.com/nestjs/typescript-starter.git project\n$ cd project\n$ npm install\n$ npm run start\nAfter installing Nest.js with the npm cli command, and creating a new project with nest new project-name, a src/ directory will be created and populated with several core files, including main.ts.\nThe main.ts includes an async function, which will bootstrap our application:
import { NestFactory } from '@nestjs/core';\nimport { ApplicationModule } from './app.module';\n\nasync function bootstrap() {\n const app = await NestFactory.create(ApplicationModule);\n await app.listen(3000);\n}\nbootstrap();\nAnd then to run the app that listens on port 3000, you execute:
\n$ npm run start\n![\"nest\"](\"//images.ctfassets.net/hspc7zpa5cvq/1I29Vhrd0whIOGhQsGPMUT/8092e2e37a828c4cad9f70628854f378/nest.png\")
This is a basic Hello World app in Nest.js. It handles 928.18 requests per second. The average time spent per request is 10.774 ms. On this metric, Nest.js performed the best out of the three frameworks we compared.
\n![\"nest2\"](\"//images.ctfassets.net/hspc7zpa5cvq/4bL41CK2YoIadG76nKqbgp/a4dd5d419adfee21dbfa09d8b7088c61/nest2.png\")
In the report provided by lighthouse, Nest.js has a very high performance, but scored comparatively lower on other key factors:accessibility, best practices and SEO.
\nThere is a group of developers providing handy packages on NestJS Community organization GitHub. Some of their popular packages are: nestjs-config, a config module for NestJS using dotenv. nest-access-control, Role and Attribute-based access control for NestJS and nestjs-flub, pretty error stack viewer.
Even if Nest is not the most popular framework, is the one that has the better performance and has many advantages. You should give it a try!
\nThe community has a spectrum chat and Twitter
"}},"categories":[{"name":"Node.js","slug":"node-js"},{"name":"Performance","slug":"performance"}]}},{"node":{"slug":"understanding-worker-threads-in-node-js","title":"Understanding Worker Threads in Node.js","image":{"file":{"url":"//images.ctfassets.net/xmu5vdhtphau/1v4gDfjjlafQJlKOnwWG3l/f4599196d0a510d6fa0c83893edb3905/23-min.png","fileName":"23-min.png"}},"bodyContent":{"childMarkdownRemark":{"html":"To understand Workers, first, it’s necessary to understand how Node.js is structured.
\nWhen a Node.js process is launched, it runs:
\nOne process: a process is a global object that can be accessed anywhere and has information about what’s being executed at a time.
\nOne thread: being single-threaded means that only one set of instructions is executed at a time in a given process.
\nOne event loop: this is one of the most important aspects to understand about Node. It’s what allows Node to be asynchronous and have non-blocking I/O, — despite the fact that JavaScript is single-threaded — by offloading operations to the system kernel whenever possible through callbacks, promises and async/await.
\nOne JS Engine Instance: this is a computer program that executes JavaScript code.
\nOne Node.js Instance: the computer program that executes Node.js code.
\nIn other words, Node runs on a single thread, and there is just one process happening at a time in the event loop. One code, one execution, (the code is not executed in parallel). This is very useful because it simplifies how you use JavaScript without worrying about concurrency issues.
\nThe reason it was built with that approach is that JavaScript was initially created for client-side interactions (like web page interactions, or form validation) -- nothing that required the complexity of multithreading.
\nBut, as with all things, there is a downside: if you have CPU-intensive code, like complex calculations in a large dataset taking place in-memory, it can block other processes from being executed. Similarly, If you are making a request to a server that has CPU-intensive code, that code can block the event loop and prevent other requests of being handled.
\nA function is considered “blocking” if the main event loop must wait until it has finished executing the next command. A “Non-blocking” function will allow the main event loop to continue as soon as it begins and typically alerts the main loop once it has finished by calling a “callback”.
\n\n\nThe golden rule: don’t block the event loop, try to keep it running it and pay attention and avoid anything that could block the thread like synchronous network calls or infinite loops.
\n
It’s important to differentiate between CPU operations and I/O (input/output) operations. As mentioned earlier, the code of Node.js is NOT executed in parallel. Only I/O operations are run in parallel, because they are executed asynchronously.
\nSo Worker Threads will not help much with I/O-intensive work because asynchronous I/O operations are more efficient than Workers can be. The main goal of Workers is to improve the performance on CPU-intensive operations not I/O operations.
\nFurthermore, there are already solutions for CPU intensive operations: multiple processes (like cluster API) that make sure that the CPU is optimally used.
\nThis approach is advantageous because it allows isolation of processes, so if something goes wrong in one process, it doesn’t affect the others. They also have stability and identical APIs. However, this means sacrificing shared memory, and the communication of data must be via JSON.
\nSo, people might think that adding a new module in Node.js core will allow us to create and sync threads, thus solving the problem of CPU-intensive operations.
\nWell, no, not really. If threads are added, the nature of the language itself will change. It’s not possible to add threads as a new set of available classes or functions. In languages that support multithreading (like Java), keywords such as “synchronized” help to enable multiple threads to sync.
\nAlso, some numeric types are not atomic, meaning that if you don’t synchronize them, you could end up having two threads changing the value of a variable and resulting that after both threads have accessed it, the variable has a few bytes changed by one thread and a few bytes changed by the other thread and thus, not resulting in any valid value. For example, in the simple operation of 0.1 + 0.2 has 17 decimals in JavaScript (the maximum number of decimals).
\nvar x = 0.1 + 0.2; // x will be 0.30000000000000004
But floating point arithmetic is not always 100% accurate. So if not synchronized, one decimal may get changed using Workers, resulting in non-identical numbers.
\nThe best solution for CPU performance is Worker Threads. Browsers have had the concept of Workers for a long time.
\nInstead of having:
\nWorker threads have:
\nAs we can see in the following image:
\n![\"worker-diagram@2x](\"//images.ctfassets.net/hspc7zpa5cvq/20h5efXHT4bQbuf44mdq2H/a40944191d031217a9169b17a8ef35d6/worker-diagram_2x__1_.jpg\")
The worker_threads module enables the use of threads that execute JavaScript in parallel. To access it:
const worker = require('worker_threads');\nWorker Threads have been available since Node.js 10, but are still in the experimental phase.
\nWhat is ideal, is to have multiple Node.js instances inside the same process. With Worker threads, a thread can end at some point and it’s not necessarily the end of the parent process. It’s not a good practice for resources that were allocated by a Worker to hang around when the Worker is gone-- that’s a memory leak, and we don’t want that. We want to embed Node.js into itself, give Node.js the ability to create a new thread and then create a new Node.js instance inside that thread; essentially running independent threads inside the same process.
\nWhat makes Worker Threads special:
\nArrayBuffers to transfer memory from one thread to anotherSharedArrayBuffer that will be accessible from either thread. It lets you share memory between threads (limited to binary data).Atomics available, it lets you do some processes concurrently, more efficiently and allows you to implement conditions variables in JavaScriptMessagePort, used for communicating between different threads. It can be used to transfer structured data, memory regions and other MessagePorts between different Workers.MessageChannel represents an asynchronous, two-way communications channel used for communicating between different threads.WorkerData is used to pass startup data. An arbitrary JavaScript value that contains a clone of the data passed to this thread’s Worker constructor. The data is cloned as if using postMessage()API
\nconst { worker, parentPort } = require(‘worker_threads’) => The worker class represents an independent JavaScript execution thread and the parentPort is an instance of the message portnew Worker(filename) or new Worker(code, { eval: true }) => are the two main ways of starting a worker (passing the filename or the code that you want to execute). It’s advisable to use the filename in production.worker.on(‘message’), worker/postMessage(data) => for listening to messages and sending them between the different threads. parentPort.on(‘message’), parentPort.postMessage(data) => Messages sent using parentPort.postMessage() will be available in the parent thread using worker.on('message'), and messages sent from the parent thread using worker.postMessage() will be available in this thread using parentPort.on('message').const { Worker } = require('worker_threads');\n\nconst worker = new Worker(`\nconst { parentPort } = require('worker_threads');\nparentPort.once('message',\n message => parentPort.postMessage({ pong: message })); \n`, { eval: true });\nworker.on('message', message => console.log(message)); \nworker.postMessage('ping'); \n$ node --experimental-worker test.js\n{ pong: ‘ping’ }\nWhat this essentially does is create a new thread using a new Worker, the code inside the Worker is listening for a message on parentPort and once it receives the message, it is going to post the message back to the main thread.
You have to use the --experimental-worker because Workers are still experimental.
Another example:
\n const {\n Worker, isMainThread, parentPort, workerData\n } = require('worker_threads');\n\n if (isMainThread) {\n module.exports = function parseJSAsync(script) {\n return new Promise((resolve, reject) => {\n const worker = new Worker(filename, {\n workerData: script\n });\n worker.on('message', resolve);\n worker.on('error', reject);\n worker.on('exit', (code) => {\n if (code !== 0)\n reject(new Error(`Worker stopped with exit code ${code}`));\n });\n });\n };\n } else {\n const { parse } = require('some-js-parsing-library');\n const script = workerData;\n parentPort.postMessage(parse(script));\n }\nIt requires:
\nWorker: the class that represents an independent JavaScript execution thread.isMainThread: a boolean that is true if the code is not running inside of a Worker thread.parentPort: the MessagePort allowing communication with the parent thread If this thread was spawned as a Worker.workerData: An arbitrary JavaScript value that contains a clone of the data passed to this thread’s Worker constructor.In actual practice for these kinds of tasks, use a pool of Workers instead. Otherwise, the overhead of creating Workers would likely exceed their benefit.
\nThe contributors to Workers in Node.js are looking for feedback, if you have used Workers before and want to contribute, you can leave your feedback here
\nWorkers have chrome DevTools support to inspect Workers in Node.js.
\nAnd worker_threads is a promising experimental module if you need to do CPU-intensive tasks in your Node.js application. Keep in mind that it’s still experimental, so it is advisable to wait before using it in production. For now, you can use Worker pools instead.
Special thanks to Anna Henningsen and her amazing talk of Node.js: The Road to Workers
\n\nNode.js multithreading: What are Worker Threads and why do they matter? - by Alberto Gimeno
\nIntroduction to Javascript Processes - by Nico Valencia
\n"}},"categories":[{"name":"Tutorials","slug":"tutorials"},{"name":"Node.js","slug":"node-js"}]}},{"node":{"slug":"microservices-in-node-js","title":"Explain like I'm 5: Microservices in Node.js","image":{"file":{"url":"//images.ctfassets.net/xmu5vdhtphau/6yr8kp0ySnhTFiFp61SpSc/4966628ebe58c7b42a913b387547ff29/22-min.png","fileName":"22-min.png"}},"bodyContent":{"childMarkdownRemark":{"html":"Microservices are an architectural approach based on building an application as a collection of small services.
\nLet’s think of an application as a store. Applications are traditionally “monolithic” which means they are built as a single, autonomous unit --think of your favorite big-box store, selling everything from bananas to lawn furniture.
\n![\"lego\"](\"//images.ctfassets.net/hspc7zpa5cvq/77vR95BkR7cPdBpk8GU9qN/08f2319787496a113693f4a76a8d06a9/lego.png\")
Everything is contained inside the unit. Let’s imagine that the person in this image - I’ll call him Jeff - is going inside the store to buy chewing gum and t-shirts. He can get a shopping cart to carry his purchases, look at products in different aisles, and pay at the checkstand before leaving--essentially, everything he needs is inside the store. These could also just as easily be components of an online Lego store application 👇.
\n![\"lego-store\"](\"//images.ctfassets.net/hspc7zpa5cvq/4DGmPnumf0bVfLZiOAVtEF/5c201fe9423d10b4f1d34b94c011cf8c/lego-store.png\")
All of this is run within a single process, and if our Lego store becomes very popular and we want to expand the business, we will have to add more Lego blocks in the same unit... and in the case of the online store, add more servers in order to scale it out.
\nSo, every change (even minor changes) in our Lego store (physical or online) can be slow or tedious as it affects the entire system. In the case of Jeff’s store, a modification can require the removal or addition of multiple Lego blocks, affecting the entire structure. In the monolithic online store, a modification made to a small section of code might require building and deploying an entirely new version of software. So, scaling specific functions or components of the application, also means you have to scale the entire system.
\nOther problems with a monolithic approach in an online application are:
\nThis is where microservices come to the rescue!
\n![\"microservices\"](\"//images.ctfassets.net/hspc7zpa5cvq/KCURMUDObC5hrpczE8bxQ/67cc4ad8b0cf96137fe39d5b679eb4b2/microservices.jpg\")
Instead of containing everything in a single unit, the microservices-based application is broken down into smaller, lightweight pieces based on a logical construct. The application consists of independent small (micro-) services, and when we deploy or scale the app, individual services get distributed within a set of machines which we call “a cluster” in the service fabric world.
\nSo in our Lego store example, perhaps one microservice contains a shopping cart, another one a product catalog, while another handles checkout, and so on. This approach allows developers to embrace compact and specialized tools that get each job done properly. Microservices are exactly that, scaled to enterprise level.
\n![\"Microservices](\"//images.ctfassets.net/hspc7zpa5cvq/6szXHVXaOaCxZD2ORnZ0FY/73f0df8b11694cc3274d9483a99e7f11/Microservices_Diagram.jpg\")
Each service has its own unique and well-defined role, runs in its own process, and communicates via HTTP APIs or messaging. Each microservice can be deployed, upgraded, scaled, and restarted independently of all the sibling services in the application. They are typically managed by an automated system, making it possible to deploy frequent updates to live applications without affecting the end-users.
\nFollowing this pattern, Jeff’s store will be very different: now he won’t have one big store where he can find everything he needs, but there would have multiple stores and each store will be independent and have specific functions. The first store may contain only Lego castles, another one bridges, and another one, Lego rockets 🚀.
\n![\"Screen](\"//images.ctfassets.net/hspc7zpa5cvq/1LV7Wy8f7xhgltfsFgKtG7/4c437b8b0bb1384bcd7836bd439e0f5e/Screen_Shot_2019-08-08_at_4.36.33_PM.png\")
All of the Lego stores will be part of a “Lego shopping mall” or “cluster,” and if I want to expand, scale, upgrade, or modify only the store selling rockets, the castle store (and the rest) won’t be affected.
\nIn other words, developers identify the separate service “pieces” that are logically closely related and necessary parts of a project. Then, they choose from the options available that meet their particular needs, from open source to enterprise solutions, and knit everything together into a functional application.
\nIn this example, we’re going to create a microservice using Node.js which connects to an external API.
\nThe requirement for this service is to accept two Zip Codes of two Lego stores and return the distance between them in miles.
\nnpm init in the root folder for the project. This will create a package.json file that will prompt some questions about the package, if you are not sure how to answer you can use the default.$ npm install express request --save\nLet’s look at the structure of the folders. There are two files and a folder created by the npm init command. These are package.json, package-lock.json, and node_modules. When we installed the express and request packages, their dependencies were downloaded and saved in node_modules.
![\"Screen](\"//images.ctfassets.net/hspc7zpa5cvq/669RoiBZTvj67W7QNHccrl/2d7fd15349bd9e7f0056b27e3f5a0e68/Screen_Shot_2019-08-01_at_3.54.03_PM.png\")
The primary file in our project is named server.js. And your package.json should look similar to this ☝️.
\nThen we create two folders, api for files that will support the API, and service for the logic to connect to a third-party API.
![\"Screen](\"//images.ctfassets.net/hspc7zpa5cvq/5aE2egdP7TrDWu1Az3stj9/eaaa5b311f14d2f28339ade15a3aee22/Screen_Shot_2019-08-01_at_3.58.17_PM.png\")
Let’s build our service!
\nCreate a file in the root folder for your project called server.js which will be our primary file. This file contains the code below.
\nconst express = require('express')\nconst app = express();\nconst port = process.env.PORT || 3000;\n\nconst routes = require('./api/routes');\nroutes(app);\napp.listen(port, function() {\n console.log('Server started on port: ' + port);\n});\nThis file is creating our server and assigns routes to process all requests.
\nWe first require express into the file, and use it to create a new app object const app = express(); then we specify the port, in this case, we use the environment variable called PORT, and if the variable isn’t defined, it will use the default port: 3000.
Then we bring the routes object from the routes.js file in the api folder. We’ll pass the app to the routes object, and that sets the routes for our application. Finally, we’ll tell the app to start listening on the port we defined and to display a message to the console when this process is complete.
\nThe next step is to define the routes for the microservices and then assign each to a target in the controller object (that will control the flow of data in the application). We’ll build the controller in the next step. We’ll have two endpoints. One endpoint called “about” that returns information about the application. And a “distance” endpoint that includes two path parameters, both Zip Codes of the Lego store. This endpoint returns the distance, in miles, between these two Zip Codes.
\n'use strict';\n\nconst controller = require('./controller');\n\nmodule.exports = function(app) {\n app.route('/about')\n .get(controller.about);\n app.route('/distance/:zipcode1/:zipcode2')\n .get(controller.getDistance);\n};\nThe ‘use strict’ directive at the top of the file is used by new versions of Javascript to enforce secure coding practices. The first thing we’ll do is to create a controller object which we’ll define in the next step.\nModule.exports allows us to declare functions inside a module, and have them available for use in another file. This file constitutes the routes module, which we imported into our primary server.js file and used it to define the routes for our express app.\nThis function adds two routes to the app. The first route listens for GET requests on the /about endpoint. These requests are handled by the about function in the controller. The second route listens for GET requests on the /distance endpoint. The getDistance function in the controller handles these requests. Two parameters are also specified. These are labeled zipcode1 and zipcode2 respectively.\nLet’s build the controller to handle those requests.
Within the controller file, we’re going to create a controller object with two properties. Those properties are the functions to handle the requests we defined in the routes module.
\n'use strict';\n\nvar properties = require('../package.json')\nvar distance = require('../service/distance');\n\nvar controllers = {\n about: function(req, res) {\n var aboutInfo = {\n name: properties.name,\n version: properties.version\n }\n res.json(aboutInfo);\n },\n getDistance: function(req, res) {\n distance.find(req, res, function(err, dist) {\n if (err)\n res.send(err);\n res.json(dist);\n });\n },\n};\n\nmodule.exports = controllers;\nWe have two distinct parts to our controller. We’ll go through the code for the about functionality first. At the top, we create an object called properties which references the package.json file which npm created when it bootstrapped the project. This file is in JavaScript Object Notation or JSON for short. This format affords us the ability to import and use the information it contains.
\nWithin the controllers object, we define a property called about. This property is a function which accepts request and response objects. We’ll only use the response object for this function. Using the name and version information from the package.json file, we’ll build a new object and return it as the response.
\nFor the getDistance functionality, we’ll start by bringing in the distance module. We’ll pass the request and response objects to the find function within this module. This function also includes a callback function. This function accepts an error object (err) and a distance object (dist). If there is an error in the response, we return that with our response; otherwise, we send back the results of the find function.
\nWe’re ready for the final piece of the puzzle. This file handles the call to a third-party API. We’ll use the distance API provided by ZipCodeAPI.com. (You need an API key to use this, and it is free if you register. You can also use the key from the example if you want to test your service, but this key frequently expires during the day).
\nI set my key as an environment variable on my system and named it ZIPCODEAPIKEY. The default key in the code is an expired test key from the ZipCodeAPI website.
\nvar request = require('request');\n\nconst apiKey = process.env.ZIPCODE_API_KEY || \"hkCt1nW1wF1rppaEmoor7T9G4ta7R5wFSu8l1dokNz8y53gGZHDneWWVosbEYirC\";\nconst zipCodeURL = 'https://www.zipcodeapi.com/rest/';\n\nvar distance = {\n find: function(req, res, next) {\n request(zipCodeURL + apiKey \n + '/distance.json/' + req.params.zipcode1 + '/' \n + req.params.zipcode2 + '/mile',\n function (error, response, body) {\n if (!error && response.statusCode == 200) {\n response = JSON.parse(body);\n res.send(response);\n } else {\n console.log(response.statusCode + response.body);\n res.send({distance: -1});\n }\n });\n\n }\n};\n\nmodule.exports = distance;\nWe’re using the request package to execute the external HTTP request, and we already discussed the api Key above. Ensure that you update it unless you want to start by testing the error conditions.
\nThe find function accepts request, response and next objects as parameters. The request object accepts the URL of the service we’re calling and then defines a callback function to handle the response.
\nIf there are no errors, and the status of the response is HTTP Status code 200, then the function parses out the body of the response into an object called response and returns it on the resp object. Since the ZipCodeAPI returns with a JSON response, we could forward this directly. Parsing it out allows us the option of doing more with the response if we choose to.
We log failures to the console, and then a result of -1 is sent on the response object. You may opt to create an error object to return as well.
\nFinally, we export the distance object, which allows the controller to instantiate it and call its functions as needed.
\nAssuming there aren’t any typos, your application should be ready to execute. Open a console window and run the following command:
\nnpm start\nAssuming it starts correctly, and the port you define is 3000, you can now open your browser and navigate to:
\nhttp://localhost:3000/about when you will see the name of the app and the version.
\n![\"Screen](\"//images.ctfassets.net/hspc7zpa5cvq/58lstZiyoh5Pyow5UeF4RP/cb5a84facefb43e0024f2111c313cbaf/Screen_Shot_2019-08-01_at_7.17.37_PM.png\")
Now if you add two parameters, the two zip codes, you will see something like this:
\nhttp://localhost:3000/distance/84010/97229
\n![\"Screen](\"//images.ctfassets.net/hspc7zpa5cvq/1VMI1kMNPmWNd4ip3UDolz/9c0237d15d4b1ac01375bb05a5ca5411/Screen_Shot_2019-08-01_at_7.18.55_PM.png\")
And that’s it! Using microservices to know the distance between two zip codes!
\nIn microservices, every single service is independently deployable, scalable and updatable, this is what makes microservices such an appealing architectural approach to the industry.
\nA microservice is loosely coupled and interacts with other microservices for well-defined interfaces using protocols like http, they remain consistent and available in the presence of failure, meaning even if the machine goes down that host a microservice, the functionality provided by the service should still be offered by the application.
\nWhile microservices are great, there is quite some work involved to build a scalable microservice application on a platform as you need to consider things like cluster management, service orchestration, inter-service communication and so on, and you also need to put a lot of effort into following DevOpsbest practices.
\nNot to mention that microservices can also require increased testing complexity and possibly increased memory/computing resources. Thus, despite the abundant potential benefits, those knowledgeable in the field caution that microservices are not automatically the right solution for every project.
\nStreams in Node.js have a reputation for being hard to work with, and even harder to understand.
\nIn the words of Dominic Tarr: “Streams are Node’s best and most misunderstood idea.” Even Dan Abramov, creator of Redux and core team member of React.js is afraid of Node streams.
\nThis article will help you understand streams and how to work with them. So, don’t be afraid. We can figure this out!
\nStreams are one of the fundamental concepts that power Node.js applications. They are data-handling method and are used to read or write input into output sequentially.
\nStreams are a way to handle reading/writing files, network communications, or any kind of end-to-end information exchange in an efficient way.
\nWhat makes streams unique, is that instead of a program reading a file into memory all at once like in the traditional way, streams read chunks of data piece by piece, processing its content without keeping it all in memory.
\nThis makes streams really powerful when working with large amounts of data, for example, a file size can be larger than your free memory space, making it impossible to read the whole file into the memory in order to process it. That’s where streams come to the rescue!
\nUsing streams to process smaller chunks of data, makes it possible to read larger files.
\nLet’s take a “streaming” services such as YouTube or Netflix for example: these services don’t make you download the video and audio feed all at once. Instead, your browser receives the video as a continuous flow of chunks, allowing the recipients to start watching and/or listening almost immediately.
\nHowever, streams are not only about working with media or big data. They also give us the power of ‘composability’ in our code. Designing with composability in mind means several components can be combined in a certain way to produce the same type of result. In Node.js it’s possible to compose powerful pieces of code by piping data to and from other smaller pieces of code, using streams.
\nStreams basically provide two major advantages compared to other data handling methods:
\nfs.createWriteStream() lets us write data to a file using streams.fs.createReadStream() lets us read the contents of a file.net.SocketIf you have already worked with Node.js, you may have come across streams. For example, in a Node.js based HTTP server, request is a readable stream and response is a writable stream. You might have used the fs module, which lets you work with both readable and writable file streams. Whenever you’re using Express you are using streams to interact with the client, also, streams are being used in every database connection driver that you can work with, because of TCP sockets, TLS stack and other connections are all based on Node.js streams.
We first require the Readable stream, and we initialize it.
\nconst Stream = require('stream')\nconst readableStream = new Stream.Readable()\nNow that the stream is initialized, we can send data to it:
\nreadableStream.push('ping!')\nreadableStream.push('pong!')\nIt’s highly recommended to use async iterator when working with streams. According to Dr. Axel Rauschmayer, Asynchronous iteration is a protocol for retrieving the contents of a data container asynchronously (meaning the current “task” may be paused before retrieving an item). Also, it’s important to mention that the stream async iterator implementation use the ‘readable’ event inside.
\nYou can use async iterator when reading from readable streams:
\nimport * as fs from 'fs';\n\nasync function logChunks(readable) {\n for await (const chunk of readable) {\n console.log(chunk);\n }\n}\n\nconst readable = fs.createReadStream(\n 'tmp/test.txt', {encoding: 'utf8'});\nlogChunks(readable);\n\n// Output:\n// 'This is a test!\\n'\nIt’s also possible to collect the contents of a readable stream in a string:
\nimport {Readable} from 'stream';\n\nasync function readableToString2(readable) {\n let result = '';\n for await (const chunk of readable) {\n result += chunk;\n }\n return result;\n}\n\nconst readable = Readable.from('Good morning!', {encoding: 'utf8'});\nassert.equal(await readableToString2(readable), 'Good morning!');\nNote that, in this case, we had to use an async function because we wanted to return a Promise.
\nIt’s important to keep in mind to not mix async functions with EventEmitter because currently, there is no way to catch a rejection when it is emitted within an event handler, causing hard to track bugs and memory leaks. The best current practice is to always wrap the content of an async function in a try/catch block and handle errors, but this is error prone. This pull request aims to solve this issue once it lands on Node core.
To learn more about Node.js streams via async iteration, check out this great article.
\nstream.Readable.from(iterable, [options]) it’s a utility method for creating Readable Streams out of iterators, which holds the data contained in iterable. Iterable can be a synchronous iterable or an asynchronous iterable. The parameter options is optional and can, among other things, be used to specify a text encoding.
const { Readable } = require('stream');\n\nasync function * generate() {\n yield 'hello';\n yield 'streams';\n}\n\nconst readable = Readable.from(generate());\n\nreadable.on('data', (chunk) => {\n console.log(chunk);\n});\nAccording to Streams API, readable streams effectively operate in one of two modes: flowing and paused. A Readable stream can be in object mode or not, regardless of whether it is in flowing mode or paused mode.
\nstream.read() method must be called explicitly to read chunks of data from the stream.In a flowing mode, to read data from a stream, it’s possible to listen to data event and attach a callback. When a chunk of data is available, the readable stream emits a data event and your callback executes. Take a look at the following snippet:
\nvar fs = require(\"fs\");\nvar data = '';\n\nvar readerStream = fs.createReadStream('file.txt'); //Create a readable stream\n\nreaderStream.setEncoding('UTF8'); // Set the encoding to be utf8. \n\n// Handle stream events --> data, end, and error\nreaderStream.on('data', function(chunk) {\n data += chunk;\n});\n\nreaderStream.on('end',function() {\n console.log(data);\n});\n\nreaderStream.on('error', function(err) {\n console.log(err.stack);\n});\n\nconsole.log(\"Program Ended\");\nThe function call fs.createReadStream() gives you a readable stream. Initially, the stream is in a static state. As soon as you listen to data event and attach a callback it starts flowing. After that, chunks of data are read and passed to your callback. The stream implementor decides how often a data event is emitted. For example, an HTTP request may emit a data event once every few KBs of data are read. When you are reading data from a file you may decide you emit a data event once a line is read.
When there is no more data to read (end is reached), the stream emits an end event. In the above snippet, we listen to this event to get notified when the end is reached.
\nAlso, if there is an error, the stream will emit and notify the error.
\nIn paused mode, you just need to call read() on the stream instance repeatedly until every chunk of data has been read, like in the following example:
\nvar fs = require('fs');\nvar readableStream = fs.createReadStream('file.txt');\nvar data = '';\nvar chunk;\n\nreadableStream.on('readable', function() {\n while ((chunk=readableStream.read()) != null) {\n data += chunk;\n }\n});\n\nreadableStream.on('end', function() {\n console.log(data)\n});\nThe read() function reads some data from the internal buffer and returns it. When there is nothing to read, it returns null. So, in the while loop, we check for null and terminate the loop. Note that the readable event is emitted when a chunk of data can be read from the stream.
\nAll Readable streams begin in paused mode but can be switched to flowing mode in one of the following ways:
stream.resume() method.stream.pipe() method to send the data to a Writable.The Readable can switch back to paused mode using one of the following:
stream.pause() method.stream.unpipe() method.The important concept to remember is that a Readable will not generate data until a mechanism for either consuming or ignoring that data is provided. If the consuming mechanism is disabled or taken away, the Readable will attempt to stop generating the data.\nAdding a readable event handler automatically make the stream to stop flowing, and the data to be consumed via readable.read(). If the 'readable' event handler is removed, then the stream will start flowing again if there is a 'data' event handler.
To write data to a writable stream you need to call write() on the stream instance. Like in the following example:
var fs = require('fs');\nvar readableStream = fs.createReadStream('file1.txt');\nvar writableStream = fs.createWriteStream('file2.txt');\n\nreadableStream.setEncoding('utf8');\n\nreadableStream.on('data', function(chunk) {\n writableStream.write(chunk);\n});\nThe above code is straightforward. It simply reads chunks of data from an input stream and writes to the destination using write(). This function returns a boolean value indicating if the operation was successful. If true, then the write was successful and you can keep writing more data. If false is returned, it means something went wrong and you can’t write anything at the moment. The writable stream will let you know when you can start writing more data by emitting a drain event.
Calling the writable.end() method signals that no more data will be written to the Writable. If provided, the optional callback function is attached as a listener for the 'finish' event.
// Write 'hello, ' and then end with 'world!'.\nconst fs = require('fs');\nconst file = fs.createWriteStream('example.txt');\nfile.write('hello, ');\nfile.end('world!');\n// Writing more now is not allowed!\nUsing a writable stream you can read data from a readable stream:
\nconst Stream = require('stream')\n\nconst readableStream = new Stream.Readable()\nconst writableStream = new Stream.Writable()\n\nwritableStream._write = (chunk, encoding, next) => {\n console.log(chunk.toString())\n next()\n}\n\nreadableStream.pipe(writableStream)\n\nreadableStream.push('ping!')\nreadableStream.push('pong!')\n\nwritableStream.end()\nYou can also use async iterators to write to a writable stream, which is recommended
\nimport * as util from 'util';\nimport * as stream from 'stream';\nimport * as fs from 'fs';\nimport {once} from 'events';\n\nconst finished = util.promisify(stream.finished); // (A)\n\nasync function writeIterableToFile(iterable, filePath) {\n const writable = fs.createWriteStream(filePath, {encoding: 'utf8'});\n for await (const chunk of iterable) {\n if (!writable.write(chunk)) { // (B)\n // Handle backpressure\n await once(writable, 'drain');\n }\n }\n writable.end(); // (C)\n // Wait until done. Throws if there are errors.\n await finished(writable);\n}\n\nawait writeIterableToFile(\n ['One', ' line of text.\\n'], 'tmp/log.txt');\nassert.equal(\n fs.readFileSync('tmp/log.txt', {encoding: 'utf8'}),\n 'One line of text.\\n');\nThe default version of stream.finished() is callback-based but can be turned into a Promise-based version via util.promisify() (line A).
\nIn this example, it is used the following two patterns:
\nWriting to a writable stream while handling backpressure (line B):
\nif (!writable.write(chunk)) {\n await once(writable, 'drain');\n}\nClosing a writable stream and waiting until writing is done (line C):
\nwritable.end();\nawait finished(writable);\nPiping is a mechanism where we provide the output of one stream as the input to another stream. It is normally used to get data from one stream and to pass the output of that stream to another stream. There is no limit on piping operations. In other words, piping is used to process streamed data in multiple steps.
\nIn Node 10.x was introduced stream.pipeline(). This is a module method to pipe between streams forwarding errors and properly cleaning up and provide a callback when the pipeline is complete.
Here is an example of using pipeline:
\nconst { pipeline } = require('stream');\nconst fs = require('fs');\nconst zlib = require('zlib');\n\n// Use the pipeline API to easily pipe a series of streams\n// together and get notified when the pipeline is fully done.\n// A pipeline to gzip a potentially huge video file efficiently:\n\npipeline(\n fs.createReadStream('The.Matrix.1080p.mkv'),\n zlib.createGzip(),\n fs.createWriteStream('The.Matrix.1080p.mkv.gz'),\n (err) => {\n if (err) {\n console.error('Pipeline failed', err);\n } else {\n console.log('Pipeline succeeded');\n }\n }\n);\npipeline should be used instead of pipe, as pipe is unsafe.
The Node.js stream module provides the foundation upon which all streaming APIs are build.
\nThe Stream module is a native module that shipped by default in Node.js. The Stream is an instance of the EventEmitter class which handles events asynchronously in Node. Because of this, streams are inherently event-based.
\nTo access the stream module:
\nconst stream = require('stream');\nThe stream module is useful for creating new types of stream instances. It is usually not necessary to use the stream module to consume streams.
Due to their advantages, many Node.js core modules provide native stream handling capabilities, most notably:
\nnet.Socket is the main node api that is stream are based on, which underlies most of the following APIsprocess.stdin returns a stream connected to stdinprocess.stdout returns a stream connected to stdoutprocess.stderr returns a stream connected to stderrfs.createReadStream() creates a readable stream to a filefs.createWriteStream() creates a writable stream to a filenet.connect() initiates a stream-based connectionhttp.request() returns an instance of the http.ClientRequest class, which is a writable streamzlib.createGzip() compress data using gzip (a compression algorithm) into a streamzlib.createGunzip() decompress a gzip stream.zlib.createDeflate() compress data using deflate (a compression algorithm) into a streamzlib.createInflate() decompress a deflate stream![\"cheat1\"](\"//images.ctfassets.net/hspc7zpa5cvq/3s8p9qQe6zDEfhwLyq5fQR/7ce28dd384a590009aaaa16886bb83eb/cheat1.png\")
![\"cheat2\"](\"//images.ctfassets.net/hspc7zpa5cvq/1vZrS7rb7U7DRb5TtT8hs8/bb6285794d4141c3a572770651fcc753/cheat2.png\")
\n![\"cheat3\"](\"//images.ctfassets.net/hspc7zpa5cvq/2rPAh1mmsLg02i0hLBG2DG/9829c1fb9cf396d952f1d155ce799f13/cheat3.png\")
\n![\"cheat3\"](\"//images.ctfassets.net/hspc7zpa5cvq/3HncUkiyjekdWjN8MwQbfi/29b3ad1806e8ce161a889c76667e780f/cheat3.png\")
Here are some important events related to writable streams:
\nerror – Emitted to indicate that an error has occurred while writing/piping.pipeline – When a readable stream is piped into a writable stream, this event is emitted by the writable stream.unpipe – Emitted when you call unpipe on the readable stream and stop it from piping into the destination stream.This was all about the basics of streams. Streams, pipes, and chaining are the core and most powerful features in Node.js. Streams can indeed help you write neat and performant code to perform I/O.
\nAlso, there is a Node.js strategic initiative worth looking to, called BOB, aiming to improve Node.js streaming data interfaces, both within Node.js core internally, and hopefully also as future public APIs.
\nSpecial thanks to Matteo Colina and Jeremiah Senkpiel for your feedback!
\nTracking down memory leaks in Node.js has been a recurring topic, people are always interested in learning more about due to the complexity and the range of causes.
\nNot all memory leaks are immediately obvious - quite the opposite; however once we identify a pattern, we must look for a correlation between memory usage, objects held in memory and response time. When examining objects, look into how many of them are collected, and whether any of them are usual, depending on the framework or technique used to serve the content (ex. Server Side Rendering). Hopefully, after you finish this article, you'll be able to understand, and look for a strategy to debug the memory consumption of a Node.js application.
\nJavaScript is a garbage collected language and Google’s V8 is a JavaScript engine, initially created for Google Chrome, that can be used as a standalone runtime in many instances. Two important operations of the Garbage Collector in Node.js are:
\nSomething important to keep in mind: When the Garbage Collector runs, it pauses your application entirely until it finishes its work. As such you will need to minimize its work by taking care of your objects’ references.
\nAll memory used by a Node.js process is being automatically allocated and de-allocated by the V8 JavaScript engine. Let’s see how this looks in practice.
\nIf you think of memory as a graph, then imagine V8 keeping a graph of all variables in the program, starting from the ‘Root node’. This could be your window or the global object in a Node.js module, usually known as the dominator. Something important to keep in mind is you don’t control how this Root node is de-allocated.
\nNext, you’ll find an Object node, usually known as leaves (there are no child references). Finally, there are 4 types of data types in JavaScript: Boolean, String, Number, and Object.
\nV8 will walk through the graph and try to identify groups of data that can no longer be reached from the Root node. If it’s not reachable from the Root node, V8 assumes that the data is no longer used and releases the memory. Remember: to determine whether an object is live, it is necessary to check if is reachable through some chain of pointers from an object which is live by definition; everything else, such as an object being unreachable from a root node or not referenceable by a root node or another live object is considered garbage.
\nIn a nutshell, the garbage collector has two main tasks;
\nIt can get tricky when you need to track remote references from another process, but in Node.js applications, we use a single process which makes our life a bit easier.
\nV8 uses a scheme similar to the Java Virtual Machine and divides the memory into segments. The thing that wraps the scheme concept is known as Resident Set, which refers to the portion of memory occupied by a process that is held in the RAM.
\nInside the Resident Set you will find:
\nTwo more important things to keep in mind:
\nNode.js has an object describing the memory usage of the Node.js process measured in bytes. Inside the object you’ll find:
\nChrome DevTools is a great tool that can be used to diagnose memory leaks in Node.js applications via remote debugging. Other tools exist and they will give you the similar. This blog post relies on one of those different tools in order to give you a clear clear understanding of what is happening. However, you need to keep in mind that profiling is an intensive CPU task, which can impact your application negatively. Be aware!
\nThe Node.js application we are going to profile is a simple HTTP API Server that has multiple endpoints, returning different information to whoever is consuming the service. You can clone the repository of the Node.js application used here.
\nconst http = require('http')\n\nconst leak = []\n\nfunction requestListener(req, res) {\n\n if (req.url === '/now') {\n let resp = JSON.stringify({ now: new Date() })\n leak.push(JSON.parse(resp))\n res.writeHead(200, { 'Content-Type': 'application/json' })\n res.write(resp)\n res.end()\n } else if (req.url === '/getSushi') {\n function importantMath() {\n let endTime = Date.now() + (5 * 1000);\n while (Date.now() < endTime) {\n Math.random();\n }\n }\n\n function theSushiTable() {\n return new Promise(resolve => {\n resolve('🍣');\n });\n }\n\n async function getSushi() {\n let sushi = await theSushiTable();\n res.writeHead(200, { 'Content-Type': 'text/html; charset=utf-8' })\n res.write(`Enjoy! ${sushi}`);\n res.end()\n }\n\n getSushi()\n importantMath()\n } else {\n res.end('Invalid request')\n }\n}\n\nconst server = http.createServer(requestListener)\nserver.listen(process.env.PORT || 3000)\nStart the Node.js application:
\nWe have been using a 3S (3 Snapshot) approach to diagnostics and identify possible memory issues. Interesting enough, we found this was an approach that has been used by Loreena Lee at the Gmail team for a long time to solve memory issues. A walkthrough for this approach:
\nchrome://inspect.inspect button from one of your applications in the Remote Target section located at the bottom.Note: Make sure you have the Inspector attached to the Node.js application you want to profile. You can also connect to Chrome DevTools using ndb .
You are going to see a Debugger Connected message in the output of your console when the app is running.
In this case, we took the first snapshot without any load or processing being done by the service. This a tip for certain use-cases: it’s fine if we are completely sure the application doesn’t require any warm up before accepting request or do some processing. Sometimes it makes sense to do a warm-up action before taking the first heap snapshot as there are cases where you might be doing lazy initialization for global variables on the first invocation.
\nIn this case we are going to run npm run load-mem. This will start ab to simulate traffic/load in your Node.js application.
The Comparison view can help you identify those Objects too:
\nIn the view you’ll see a list of leaked objects that are still hanging around, top level entries (a row per constructor), columns for distance of the object to the GC root, number of object instances, shallow size and retained size. You can select one to see what is being retained in its retaining tree. A good rule of thumb is to first ignore the items wrapped in parenthesis as those are built-in structures. The @ character is objects’ unique ID, allowing you to compare heap snapshots on per-object basis.
A typical memory leak might retain a reference to an object that’s expected to only last during one request cycle by accidentally storing a reference to it in a global object that cannot be garbage collected.
\nThis example generates a random object with the date timestamp when the request was made to imitate an application object that might be returned from an API query and purposefully leak it by storing it in a global array. Looking at a couple of the retained Object’s you can see some examples of the data that has been leaked, which you can use to track down the leak in your application.
\nNSolid is great for this type of use-case, because it gives you a really good insight of how memory is increasing on every task or load-test you perform. You can also see in real time how every profiling action impacts CPU if you were curious.
\nIn real world situations, memory leaks happen when you are not looking at the tool you use to monitor your application, something great about NSolid is the ability to set thresholds and limits for different metrics of your application. For example, you can set NSolid to take a heap snapshot if more than X amount of memory is being used or during X time memory hasn’t recovered from a high consumption spike. Sounds great, right?
\nV8’s garbage collector is mainly based on the Mark-Sweep collection algorithm which consists of tracing garbage collection that operates by marking reachable objects, then sweeping over memory and recycling objects that are unmarked (which must be unreachable), putting them on a free list. This is also known as a generational garbage collector where objects may move within the young generation, from the young to the old generation, and within the old generation.
\nMoving objects is expensive since the underlying memory of objects needs to be copied to new locations and the pointers to those objects are also subject to updating.
\nFor mere mortals, this could be translated to:
\nV8 looks recursively for all objects’ reference paths to the Root node. For example: In JavaScript, the \"window\" object is an example of a global variable that can act as a Root. The window object is always present, so the garbage collector can consider it and all of its children to be always present (i.e. not garbage). If any reference has no path to the Root node. especially when it looks for unreferenced objects recursively, it will be marked as garbage and will be swept later to free that memory and return it to the OS.
\nHowever, modern garbage collectors improve on this algorithm in different ways, but the essence is the same: reachable pieces of memory are marked as such and the rest is considered garbage.
\nRemember, everything that can be reached from a Root is not considered garbage. Unwanted references are variables kept somewhere in the code that will not be used anymore and point to a piece of memory that could otherwise be freed, so to understand the most common leaks in JavaScript, we need to know the ways references are commonly forgotten.
\nOrinoco is the codename of the latest GC project to make use of the latest and greatest parallel, incremental and concurrent technique for garbage collection, featuring the ability to free the main thread. One of the significant metrics describing Orinoco’s performance is how often and how long the main thread pauses while the garbage collector performs its functions. For classic ‘stop-the-world’ collectors, these time-intervals impact the application’s user experience due to delays, poor-quality rendering, and an increase in response time.
\nV8 distributes the work of garbage collection between auxiliary streams in young memory (scavenging). Each stream receives a set of pointers, followed by moving all living objects into “to-space”.
\nWhen moving objects into ‘to-space’, threads need to synchronize through atomic read / write / compare and swap operations to avoid a situation where, for example, another thread found the same object, but followed a different path, and tries to move it.
\nQuote from V8 page:
\n\n\nAdding parallel, incremental and concurrent techniques to the existing GC was a multi-year effort, but has paid off, moving a lot of work to background tasks. It has drastically improved pause times, latency, and page load, making animation, scrolling, and user interaction much smoother. The parallel Scavenger has reduced the main thread young generation garbage collection total time by about 20%–50%, depending on the workload. Idle-time GC can reduce Gmail’s JavaScript heap memory by 45% when it is idle. Concurrent marking and sweeping has reduced pause times in heavy WebGL games by up to 50%.
\n
The Mark-Evacuate collector consists of three phases: marking, copying, and updating pointers. To avoid sweeping pages in the young generation to maintain free lists, the young generation is still maintained using a semi-space that is always kept compact by copying live objects into “to-space” during garbage collection. It's advantage being parallel is that ‘exact liveness’ information is available. This information can be used to avoid copying by just moving and relinking pages that contain mostly live objects, which is also performed by the full Mark-Sweep-Compact collector. It works by marking live objects in the heap in the same fashion as the mark-sweep algorithm, meaning the heap will often be fragmented. V8 currently ships with the parallel Scavenger which reduces the main thread young generation garbage collection total time by about 20%–50% across a large set of benchmarks.
\nEverything related to pausing of the main thread, response time and page load has significantly improved, which allows animations, scrolling and user interaction on the page to be much smoother. The parallel collector made it possible to reduce the total duration of processing of young memory by 20–50%, depending on the load. However, the work is not over: Reducing pauses remains an important task to simplify the lives of web users, and we continue to look for the possibility of using more advanced techniques to achieve the goal.
\nMost developers don’t need to think about GC when developing JavaScript programs, but understanding some of the internals can help you think about memory usage and helpful programming patterns. For example, given the structure of the heap in V8, based on generations, low-living objects are actually quite cheap in terms of GC, since we pay mainly for the surviving objects. This kind of pattern is not only particular to JavaScript but also to many languages with garbage collection support.
\nThis article was first published in NodeSource blog on March 2017.
\nThe package.json file is core to the Node.js ecosystem and is a basic part of understanding and working with Node.js, npm, and even modern JavaScript. The package.json is used as what equates to a manifest about applications, modules, packages, and more - it's a tool to that's used to make modern development streamlined, modular, and efficient.
As a developer in the Node.js ecosystem, understanding the basics of package.json is one of the first steps to really kicking off your development experience with Node.js.
Because of how essential understanding the basics of package.json is to development with Node.js, I've gone through and outlined some of the most common and important properties of a package.json file that you'll need to use package.json effectively.
package.jsonname propertyThe name property in a package.json file is one of the fundamental components of the package.json structure. At its core, name is a string that is exactly what you would expect - the name of the module that the package.json is describing.
Inside your package.json, the name property as a string would look something like this:
\"name\": \"metaverse\"\nDespite having only a few material restrictions (a max length of 214 characters, can't begin with . or _, no uppercase letters, and no characters that aren't URL-friendly), one interesting aspect of the name property is that, there have been software ecosystems that have developed standard naming conventions that enable discoverability simply by using the name property.
A few examples of this kind of namespacing are babel-plugin- for Babel and the the webpack -loader tooling.
version propertyThe version property is a key part of a package.json, as it denotes the current version of the module that the package.json file is describing.
While the version property isn't required to follow semver, semver is the standard used by the vast majority of modules and projects in the Node.js ecosystem - and the module version, according to semver, is what you'll typically find in the version property of a package.json file.
Inside your package.json, the version property as a string using semver could look like this:
\"version\": \"5.12.4\"\nlicense propertyThe license property of a package.json file is used to note what license the module that the package.json file is describing. While there are some complex ways you can use the license property of a package.json file (to do things like dual-licensing or defining your own license), the most typical usage of it is to use a SPDX License identifier - some examples that you may recognize are MIT, ISC, and GPL-3.0.
Inside your package.json, the license property with an MIT license look like this:
\"license\": \"MIT\"\ndescription propertyThe description property of a package.json file is a string that contains a human-readable description about the module - basically, it's the module developer's chance to quickly let users know what exactly a module does. The description property is frequently indexed by search tools like npm search and the npm CLI search tool to help find relevant packages based on a search query.
Inside your package.json, the description property would look like this:
\"description\": \"The Metaverse virtual reality. The final outcome of all virtual worlds, augmented reality, and the Internet.\"\nkeywords propertyThe keywords property inside a package.json file is, as you may have guessed, a collection of keywords about a module. Keywords can help identify a package, related modules and software, and concepts.
The keywords property is always going to be an array, with one or more strings as the array's values - each one of these strings will, in turn, be one of the project's keywords.
Inside your package.json, the keywords array would look something like this:
\"keywords\": [\n \"metaverse\",\n \"virtual reality\",\n \"augmented reality\",\n \"snow crash\"\n ]\npackage.jsonmain propertyThe main property of a package.json is a direction to the entry point to the module that the package.json is describing. In a Node.js application, when the module is called via a require statement, the module's exports from the file named in the main property will be what's returned to the Node.js application.
Inside your package.json, the main property, with an entry point of app.js, would look like this:
\"main\": \"app.js\",\nrepository propertyThe repository property of a package.json is an array that defines where the source code for the module lives. Typically, for open source projects, this would be a public GitHub repo, with the repository array noting that the type of version control is git, and the URL of the repo itself. One thing to note about this is that it's not just a URL the repo can be accessed from, but the full URL that the version control can be accessed from.
Inside your package.json, the repository property would look like this:
\"repository\": {\n \"type\": \"git\",\n \"url\": \"https://github.com/bnb/metaverse.git\"\n }\nscripts propertyThe scripts property of a package.json file is simple conceptually, but is complex functionally to the point that it's used as a build tool by many.
At its simplest, the scripts property takes an object with as many key/value pairs as desired. Each one of the keys in these key/value pairs is the name of a command that can be run. The corresponding value of each key is the actual command that is run. Scripts are frequently used for testing, building, and streamlining of the needed commands to work with a module.
Inside your package.json, the scripts property with a build command to execute node app.js (presumably to build your application) and a test command using Standard would look like this:
\"scripts\": {\n \"build\": \"node app.js\",\n \"test\": \"standard\"\n }\ndependencies propertyThe dependencies property of a module's package.json is where dependencies - the other modules that this module uses - are defined. The dependencies property takes an object that has the name and version at which each dependency should be used. Tying things back to the version property defined earlier, the version that a module needs is defined. Do note that you'll frequently find carets (^) and tildes (~) included with package versions. These are the notation for version range - taking a deep-dive into these is outside the scope of this article, but you can learn more in our primer on semver.
Inside your package.json, the dependencies property of your module may look something like this:
\"dependencies\": {\n \"async\": \"^0.2.10\",\n \"npm2es\": \"~0.4.2\",\n \"optimist\": \"~0.6.0\",\n \"request\": \"~2.30.0\",\n \"skateboard\": \"^1.5.1\",\n \"split\": \"^0.3.0\",\n \"weld\": \"^0.2.2\"\n },\ndevDependencies propertyThe devDependencies property of a package.json is almost identical to the dependencies property in terms of structure, with a key difference. The dependencies property is used to define the dependencies that a module needs to run in production. The devDependencies property is usually used to define the dependencies the module needs to run in development.
Inside your package.json, the devDependencies property would look something like this:
\"devDependencies\": {\n \"escape-html\": \"^1.0.3\",\n \"lucene-query-parser\": \"^1.0.1\"\n }\nIf you want to keep learning about Node.js, npm, package.json, and development with the Node.js stack, I've got some awesome articles for you.
\nWe also have a guide on some great utilities for Node.js developers - if you want to kick your developer experience to 11, be sure to check it out to find some tools to help you get there.
\nThe goal with this guide was to help kickstart you with package.json for development with Node.js and npm. If you want to take the leap and ensure that you're always on solid footing with Node.js and npm modules, you should check out NodeSource Certified Modules - an awesome tool to help ensure that you spend more time building applications and less time worrying about modules.
This article was first published in NodeSource blog on February 2017.
\nUsing npm effectively is a cornerstone of modern web development, no matter if it's exclusively with Node.js, as a package manager or build tool for the front-end, or even as a piece of workflows in other languages and on other platforms.
\nReally understanding npm as a tool, understanding the core concepts, can be something that's difficult for a beginner - I spent many hours just trying to figure out small details that would seem minor or be taken for granted by others.
\nAs such, I've written up a basic and detailed guide for understanding npm, for those who are entirely new to Node.js, npm, and the surrounding ecosystem.
\npackage.jsonAs a general rule, any project that's using Node.js will need to have a package.json file. What is a package.json file?
At its simplest, a package.json file can be described as a manifest of your project that includes the packages and applications it depends on, information about its unique source control, and specific metadata like the project's name, description, and author.
Let's break down the core parts of a typical package.json file:
Inside a package.json, you'll almost always find metadata specific to the project - no matter if it's a web application, Node.js module, or even just a plain JavaScirpt library. This metadata helps identify the project and acts as a baseline for users and contributors to get information about the project.
\nHere's an example of how these fields would look in a package.json file:
\n{\n \"name\": \"metaverse\", // The name of your project\n \"version\": \"0.92.12\", // The version of your project\n \"description\": \"The Metaverse virtual reality. The final outcome of all virtual worlds, augmented reality, and the Internet.\", // The description of your project\n \"main\": \"index.js\"\n \"license\": \"MIT\" // The license of your project\n}\nA package.json file is always structured in the JSON format, which allows it to be easily read as metadata and parsed by machines.
If needing to format a package.json file manually to get your project up and running seems a bit daunting, there's a handy command that will automatically generate a base package.json file for you - if you'd like to learn how to use it, take a peek at the npm init instructions below!
dependencies and devDepenendcies in your package.jsonThe other majorly important aspect of a package.json is that it contains a collection of any given project's dependencies. These dependencies are the modules that the project relies on to function properly.
Having dependencies in your project's package.json allows the project to install the versions of the modules it depends on. By running an install command (see the instructions for npm install below) inside of a project, you can install all of the dependencies that are listed in the project's package.json - meaning they don't have to be (and almost never should be) bundled with the project itself.
Second, it allows the separation of dependencies that are needed for production and dependencies that are needed for development. In production, you're likely not going to need a tool to watch your CSS files for changes and refresh the app when they change. But in both production and development, you'll want to have the modules that enable what you're trying to accomplish with your project - things like your web framework, API tools, and code utilities.
\nWhat would a project's package.json look like with dependencies and devDependencies? Let's expand on the previous example of a package.json to include some.
{\n \"name\": \"metaverse\",\n \"version\": \"0.92.12\",\n \"description\": \"The Metaverse virtual reality. The final outcome of all virtual worlds, augmented reality, and the Internet.\",\n \"main\": \"index.js\"\n \"license\": \"MIT\",\n \"devDependencies\": {\n \"mocha\": \"~3.1\",\n \"native-hello-world\": \"^1.0.0\",\n \"should\": \"~3.3\",\n \"sinon\": \"~1.9\"\n },\n \"dependencies\": {\n \"fill-keys\": \"^1.0.2\",\n \"module-not-found-error\": \"^1.0.0\",\n \"resolve\": \"~1.1.7\"\n }\n}\nOne key difference between the dependencies and the other common parts of a package.json is that they're both objects, with multiple key/value pairs. Every key in both dependencies and devDependencies is a name of a package, and every value is the version range that's acceptable to install (according to Semantic Versioning - to learn more about Semantic Versioning, also known as semver, check out our primer on semver).
When using npm, you're most likely going to be using the command line tool for the majority of your interactions. As such, here's a detailed rundown of the commands that you'll encounter and need to use most frequently.
\nnpm init to Initialize a ProjectThe npm init command is a step-by-step tool to scaffold out your project. It will prompt you for input for a few aspects of the project in the following order:
It's worth noting that if you're content with the suggestion that the npm init command provides next to the prompt, you can simply hit Return or Enter to accept the suggestion and move on to the next prompt.
Once you run through the npm init steps above, a package.json file will be generated and placed in the current directory. If you run it in a directory that's not exclusively for your project, don't worry! Generating a package.json doesn't really do anything, other than create a package.json file. You can either move the package.json file to a directory that's dedicated to your project, or you can create an entirely new one in such a directory.
npm init:npm init # This will trigger the initialization\nnpm init --yes to Instantly Initialize a ProjectIf you want to get on to building your project, and don't want to spend the (albeit brief) time answering the prompts that come from npm init, you can use the --yes flag on the npm init command to automatically populate all options with the default npm init values.
\n\nNote: You can configure what these default values are with the npm configuration - that's a more advanced topic, and outside the scope of this beginner's guide to npm.
\nThat said, if you're interested in setting that up, you can learn how to set these defaults in the eleventh tip of our npm tricks article.
\n
npm init --yes # This will trigger automatically populated initialization.\nnpm installInstalling modules from npm is one of the most basic things you should learn to do when getting started with npm. As you dive deeper, you'll begin to learn some variations on installing modules, but here's the very core of what you need to know to install a standalone module into the current directory:
\nnpm install <module>\nIn the above command, you'd replace <module> with the name of the module you want to install. For example, if you want to install Express (the most used and most well known Node.js web framework), you could run the following command:
npm install express\nThe above command will install the express module into /node_modules in the current directory. Whenever you install a module from npm, it will be installed into the node_modules folder.
In addition to triggering an install of a single module, you can actually trigger the installation of all modules that are listed as dependencies and devDependencies in the package.json in the current directory. To do so, you'll simply need to run the command itself:
npm install\nOnce you run this, npm will begin the installation process of all of the current project's dependencies.
\nAs an aside, one thing to note is that there's an alias for npm install that you may see in the wild when working with modules from the ecosystem. The alias is npm i, where i takes the place of install.
This seemingly minor alias is a small gotcha for beginners - including myself, several times when I was learning - to the Node.js and npm ecosystems, as there's not a standardized, single way that module creators and maintainers will instruct on how to install their module.
\nnpm install <module> # Where <module> is the name of the module you want to install\nnpm i <module> # Where <module> is the name of the module you want to install - using the i alias for installation\npackage.json as a dependencyAs with npm init, the npm install command has a flag or two that you'll find useful in your workflow - it'll save you time and effort with regard to your project's package.json file.
When you're running npm install to install a module, you can add the optional flag --save to the command. This flag will add the module as a dependency of your project to the project's package.json as an entry in dependencies.
npm install <module> --save # Where <module> is the name of the module you want to install\npackage.json as a Developer dependencyThere's a flag that is nearly an exact duplicate, in terms of functionality, of the --save flag when installing a module: --save-dev. There are a few a key differences between the two - instead of saving the module being installed and added to package.json as an entry in dependencies, it will save it as an entry in the devDependencies.
The semantic difference here is that dependencies are for use in production - whatever that would entail for your project. On the other hand, devDependencies are a collection of the dependencies that are used in development of your application - the modules that you use to build it, but don't need to use when it's running. This could include things like testing tools, a local server to speed up your development, and more.
npm install <module> --save-dev # Where <module> is the name of the module you want to install\nThe final, and most common, flag for npm install that you should are the flags to install a module globally on your system.
Global modules can be extremely useful - there are tons tools, utilities, and more for both development and general usage that you can install globally to use.
\nTo install a module from npm globally, you'll simply need to use the --global flag when running the install command to have the module install globally, rather than locally (to the current directory).
\n\nNote: One caveat with global modules is that, by default, npm will install them to a system directory, not a local one. With this as the default, you'll need to authenticate as a privileged user on your system to install global modules.
\nAs a best practice, you should change the default installation location from a system directory to a user directory. If you'd like to learn to do this, take a peek at the seventh tip in our npm tricks article!
\n
npm install <module> --global # Where <module> is the name of the module you want to install globally\nnpm install <module> -g # Where <module> is the name of the module you want to install globally, using the -g alias\nIf you want to keep learning about npm and all its facets, I've got a few awesome things for you. A bit ago, we shared a few npm tricks to knock your wombat socks off. Even better, we wrote a follow-up with even more npm tricks! This beginner's guide is a great springboard to get off the ground, and both of those will help you start optimizing your work with npm! If you'd like to go even further with npm and start deploying Node.js apps and npm modules into production, you should definitely take a look at NodeSource Certified Modules - it's an awesome tool that'll compliment your newly acquired npm skills!
"}},"categories":[{"name":"How To","slug":"how-to"},{"name":"Tutorials","slug":"tutorials"},{"name":"Node.js","slug":"node-js"}]}}]},"contentfulHero":{"headLine":"List of useful articles and blog post on Node.js","coverImage":{"file":{"url":"//images.ctfassets.net/xmu5vdhtphau/3tCG630sR3116gwqT52u9C/dadf240e71f8ab06e2492da2e46a561a/blog-min.jpg","fileName":"blog-min.jpg"}}}},"pageContext":{"limit":7,"skip":21,"numPages":5,"currentPage":4,"type":"blog","hero":"hero-blogs","prevPath":"/blogs/page/3","nextPath":"/blogs/page/5"}}}