p2p Databases - Hyperbee - workshop

🎓 Workshop: What you will learn

* Some background on peer to peer and decentralized web
* How to get started using hyperspace cli
* How to create a p2p database using the hyp cli
* How to perform basic database functions using p2p db
* How to connect to a p2p database from a remote system
* How to query a remote p2p database and show results

Hyperbee is a p2p database built using b-tree structures and distributed using append-only log structures called hypercore. The folks at hyper protocol have created some tooling to work with p2p databases called hyperspace. This workshop will learn how to create a p2p database, interact with a p2p database, and connect to the database from a remote server.

Prerequisites

What do I need to know for this workshop?

• Javascript - https://developer.mozilla.org/en-US/docs/Web/Javascript
• NodeJS v14+ - https://nodejs.org
• Some understanding of NodeJS Streams - https://nodejs.dev/learn/nodejs-streams
• ExpressJS  Basics - https://expressjs.com/

What is p2p?

p2p or peer to peer is a technology where there is no dependency on centralized servers to share information. Some call p2p the decentralized web, but I think the decentralized web may be more broad. p2p enables computers to talk without having to depend on a centralized system coordinating the discussion. Some popular peer to peer technologies are bittorrent and webrtc.  BitTorrent lets you download large files in pieces from different servers, which results in a much faster download time. WebRTC enables browsers to share video, audio, and data between themselves without having to go through a central service.

What is a p2p database?

A p2p database is a database that is accessible via a p2p network so that applications can connect to the database and access the data. You may be a bit confused, why should I be interested in a p2p database? With a traditional database, you have a gateway and that gateway controls the access of creating the data and querying the data. So for everyone to use that database to create content they must connect to the database and store their information, to read the content they must connect to the database and query information. This works well in a lot of cases.

Hypothetical use case

A p2p social app, where every user created their own local database and provided the identifier of that database to a feed aggregator application, that would consume all of those user databases and merge them into one large database showing everyone's posts. The benefit of such a system is that the creator of the content would still have ownership of their content, and the feed aggregator only has control of the whole feed, but could never delete the content of a client.

Getting started

Let's use the terminal to install the hyperspace cli.

npm install --global @hyperspace/cli

Let's use the terminal to create a new folder and initialize a new NodeJS application:

mkdir p2pDb
cd p2pDb
npm init -y

Install some dependencies from npm.

npm install express hyperspace hyperbee

Copy the server.js file example server code:

Creating a hyperbee database using hyp

To create a hyperbee database we need to start the hyperspace daemon then call the create command with the cli. This will give us a hyper:// URL, this URL will be our access identifier to our hyperbee database.

hyp daemon start
hyp create bee

Setting up the DB in our server.js

There are two node modules we need to connect to our hyperbee database, hyperspace and hyperbee, let's create a new file in our project called db.js in this file, we will keep all of our DB commands.

db.js

const { Client } = require('hyperspace')
const Hyperbee = require('hyperbee')
const cuid = require('cuid')

const client = new Client()
const store = client.corestore()
const core = store.get('ee26ef1982c9f3bb3ce49adc46cbc55467ecb88779a629234af702da9965758e')
const db = new Hyperbee(core, {
  keyEncoding: 'utf-8',
  valueEncoding: 'json'
})

// add movie to db
exports.add = async function (movie) {
  return db.put(cuid(), movie)
}

// list movies
exports.list = async function () {
  var movies = []
  await new Promise(r => {
    db.createReadStream()
      .on('data', entry => movies.push(entry.value))
      .on('end', r)
  })
  return movies
}

In this db.js file, we are creating two functions, add and list the add function will add data to the hyperbee database and the list function will stream all of the documents from the hyperbee database. To connect to the database, we need to instantiate a new client to hyperspace and then connect to the hyperspace store, then get a hypercore from the store, using that core, we will instantiate a hyperbee database.

Adding a movie from our web app

To add a movie from our web app, we need to bring our db.js module into the server js module:

const { add, list } = require('./db') // at the top of serverjs

In our app.post('/') handler, we can convert to an async function and call add(req.body).

app.post('/', express.json(), async (req, res) => {
  await add(req.body)
  res.setHeader('content-type', 'text/html')
  res.send(`<li>${req.body.title}</li>`)
})

list movies from hypebee

On our get /movies endpoint, let's call the list method from the DB module and map over the results to create some <li> s with each movie title.

app.get('/movies', async (req, res) => {
  const s = await list()
  res.setHeader('content-type', 'text/html')
  res.send(s.map(m => `<li>${m.title}</li>`).join(''))
})

refresh movies when we add a new movie

After we add the movie, let's request the movie list and convert the list to <li>s to show the list of titles.

app.post('/', express.json(), async (req, res) => {
  await add(req.body)
  const s = await list()
  res.setHeader('content-type', 'text/html')
  res.send(s.map(m => `<li>${m.title}</li>`).join(''))
})

Part 2: remote app

So far this process has not been much different from a traditional database application, now we will use another virtual machine to spin up a movie list application that takes the key of the hyper database, connects, and gets the list of movies and displays in the web browser.

In this new machine, we need to set up the hyperspace environment

npm install --global @hyperspace/cli

mkdir p2pClient
cd p2pClient

npm init -y
npm install express hyperspace hyperbee

Let's create a server2.js and db2.js file

Set our key in our db2.js file and run the server.js

npx nodemon server2.js

When we open the browser, we don't get out a list of movies, because we have not told hyperspace to replicate the data.

Let's go over to our first application and in the db.js file add the following:

db.ready()
  .then(() => client.replicate(core))

This will tell hyperspace to replicate our database, now when we go to our second app and refresh the browser, we should get the list of movies!

Success!

Take some time, try to add some more movies, then go to your other app and refresh the screen and see if you see the newly added movies show up. If you did, you are remotely replicating data from one peer to another peer.

A couple of notes/features:
* hyperbees are discovered via DHT
* hyperbees are sparse - which means you do not have to download the whole dataset to query,  hyperspace will just download what you need.

There is a lot more to checkout:

Summary

p2p is fun, databases are fun, combining the two is definitely mad science and it is very easy to get frustrated and discouraged. The tools are getting better every day and I do think there is some valid use case out there for this technology. I will talk about some examples in the video recording. NodeJS is a powerful server runtime that can do a lot of things, even distribute databases via peer-to-peer networks.

If you enjoyed this workshop and would like to know more, go to https://hypercore-protocol.org/ and read the documentation.