Solve Hard Problems Faster with Node.js

Data Brokering with Node.js: Process data at the speed of technology

Free instant access to our full course.

Start learning now

What you'll learn

• How to install Node.js and manage a local development environment
• How to read, and write data via both the file system and Node.js streams
• The inner workings of Node's event loop, promises, and asynchronous APIs

• Using npm to manage 3rd party libraries, and keeping custom code organized with modules
• Setting up an API proxy server using Express
• Common design patterns for applications that process large amounts of data with Node.js

What you'll build

Two different Node applications that use an Extract, Transform, Load (ETL) pipeline to process data from NASA's exoplanet API: one using the `fs` module, and the other using streams.

An Express-based API proxy server that allows us to customize data from the Exoplanet API and return a subset of the data customized for our needs, plus performance optimizations like caching and data compression.

Tutorials

This course includes ~12 hours of written and video content:

• Processing Large Amounts of Data with Node.js
• Node's Module System
• Write Asynchronous Node.js Code With Async, Await, and Promises
• Node.js for Reading and Writing Files
• What Are Streams in Node.js?
• Install and Upgrade Node.js
• Manage Application Configuration with Environment Variables in Node.js
• Set up API proxy middleware with Node.js Express
• Explore Node's Event Loop
• User Authentication in Node.js Applications

Processing Large Amounts of Data with Node.js

Learn to process a large amount of data with Node.js, (called data brokering), using the ETL (Extract, Transform, Load) pattern. With Node, you can share data between services without tightly coupling applications together or rewriting them. We often have data locked away in a legacy API, third party system, or somewhere else. When we need to access that data from a new frontend application, a GraphQL server, (or anywhere in our stack where we weren't accessing data before), Node.js is an excellent tool.

In this Node tutorial series, learn about using the Node ETL pattern to retrieve, move, and process data anywhere.

Tutorials:

• Overview of Data Brokering With Node.js
• Node.js ETL Pipeline: What Are We Building?
• What Is the Node.js ETL Pipeline?
• ETL: Extract Data with Node.js
• ETL: Transform Data with Node.js
• ETL: Load Data to Destination with Node.js

Node's Module System

A module is any file or directory in the node_modules directory that can be loaded by the Node.js require() function. As building blocks of code structure, Node.js modules allow developers to better structure, reuse, and share code. Node.js applications use Node package manager (npm) or yarn and a package.json file to keep track of the modules they depend on. Learn what npm is, how to create a package.json file, and how to install or update specific packages using npm. These sections are full of best practices for using npm for both beginners and experienced Node.js developers.

Tutorials:

• How the Node Module System Works
• Introduction to the Node Package Manager (NPM)
• What Is package.json?
• Create a package.json File
• What Is package-lock.json?
• How to Install NPM Packages
• Uninstall Packages from a Project
• How to Use Semantic Versioning in NPM
• How to Update a Node Dependency with NPM
• What Are NPM Scripts?
• Restart a Node.js Application upon Changing a File
• Organize Your Node.js Code into Modules

Write Asynchronous Node.js Code With Async, Await, and Promises

In this tutorial series, learn how to write clean, non-blocking Node.js code using async, await, and promises. Understanding asynchronous code is an important part of understanding how Node.js works.

Tutorials:

• Understanding Promises in Node.js
• Use Promise.all to Wait for Multiple Promises
• Use JavaScript's Async/Await with Promises
• How to Make API Requests with Request-Promise in Node.js

Node.js for Reading/Writing Files

Learn about the Node.js core fs module's fs.readFile and fs.writeFile. Use Node.js to parse JSON data from files, and save a Node.js object to JSON.

Tutorials:

• What Is the fs (File System) Module?
• Read/Write JSON Files with Node.js

What Are Streams in Node.js?

Like Unix pipes, Node.js streams allow you to transfer large amounts of data in real on mobile devices or websites by connecting the data source and consumer. Node.js streams can transfer large amounts of data quickly and efficiently by breaking the data into chunks instead of reading it all into memory at once. In this Node.js tutorial, learn to use fs.createReadStream, fs.createWriteStream, and pipe to process large CSV and JSON files.

Tutorials:

• What Is a Node.js Stream?
• Use Streams to Extract, Transform, and Load CSV Data
• Stream to an HTTP Response with Node.js

Install and Upgrade Node.js

Learn how to download and install Node.js, update to the latest version of Node, and manage multiple versions of Node.js on the same environment using nvm (the Node Version Manager). These tutorials also explain the Node.js version release cycle, security updates, and LTS releases.

Tutorials:

• Overview: How to Manage Node.js Locally
• Install Node.js Locally with Node Version Manager (nvm)

Manage Application Configuration with Environment Variables in Node.js

Learn what environment variables are and they can be used to configure a Node.js application for both development and production. Then install the popular dotenv package with npm and create a .env file to define and document your variables.

Tutorials:

• How to Use Environment Variables in Node.js
• Set up and Test a Dot Env (.env) File in Node

Set up API proxy middleware with Node.js Express

Learn what a proxy server is, and why you might need a proxy for your API. Then build one using the popular Node.js Express server.

Tutorials:

• What Is an API Proxy?
• What Is the Express Node.js Framework?
• How to Add a Route to an Express Server in Node.js
• Express Middleware in Node.js
• Use Express to Create an API Proxy Server in Node.js
• How to Set up an Express.js Server in Node.js
• Set up Routes for Your API in Node.js
• Optimize an Express Server in Node.js
• Add Compression to Express in Node.js
• Add Response Caching to a Node.js Express Server

User Authentication in Node.js Applications

Authentication, and authorization, allow us to know more about the individuals who are interacting with our applications. Who are they, why are they here, and perhaps most importantly what are they allowed to do? This set of tutorials explores the fundamentals of building secure Express applications, handling sensitive user input, and communicating authentication status via sessions, cookies, and JSON Web Tokens between a Node.js back-end and a front-end application of your choice.

Tutorials:

• What Is the Difference Between Authorization and Authentication?
• What Are Form Validation and Sanitization?
• Process User Login Form with ExpressJS
• How to Validate and Sanitize an ExpressJS Form
• What Is the Difference Between Sessions and JSON Web Tokens (JWT) Authentication?
• Set Up ExpressJS Session Authentication for Node Applications
• Authenticate Users with Node ExpressJS and Passport.js

Explore Node's Event Loop

The event loop is a fundamental concept of Node.js. It opens the door to understanding Node's asynchronous processes and non-blocking I/O. It outlines the mechanisms that make Node a successful, powerful, and popular modern framework. These tutorials are useful for Node.js developers who want a deeper understanding of what's happening under the hood of every application, and those who want to take full control of every step of its running cycle.

Tutorials:

• How the Event Loop Works in Node.js
• Explore the Timers Phase of Node's Event Loop
• Explore the I/O Callbacks Phase of the Node.js Event Loop
• Explore the Immediate Callbacks Phase of Node's Event Loop

Jon says "Hi!"

Meet Jon Church, your Node.js teacher. He writes, researches, codes, and records courses that teach people the infinite possibilities of Node.js. He's also an enthusiastic supporter and contributor to the Node.js project.

Meet the rest of our team

"The Osio Labs team has done an incredible job listening to the needs of the Node.js community, working closely with a diverse contingent of members from across the globe, to craft a training program that fits the needs of today's Node.js developer."

Dan Shaw, Global Head of Developer Relations, PayPal

"Your videos have been awesome! Been watching the free ones off and on for the past few years. Bought a subscription recently and am getting a lot out of it!"

Andrew Rivera, Drupal Developer

"Great Job @drupalizeme. U are doing great guides 4 #Drupal. I personally used their site to prepare for @acquia certified program and I won."

Ildephonse Bikino, Deputy Director for ICT in Education Projects

Who is this for?

Data Brokering with Node.js aims to help JavaScripters who need a better understanding of back-end or application-layer concepts, those with a specific Node problem to solve, and those levelling up their tech skills.

Who is this by?

Hey Node is a labor of love from Osio Labs. We're Open Source Inside and Out, and we support and lead projects that foster and grow open source communities and the technology they create.