TensorFlow.js: Μachine learning in your browser

When you talk about Machine Learning and Google’s TensorFlow, most people think of Python and specialized hardware rather than JavaScript and any browser. What TensorFlow.js can do and why it makes sense to run machine learning in the browser, explains this article.

TensorFlow.js is a JavaScript library that runs in the browser as well as with Node.js on the server. However, we are interested in this article only for the application in the browser. The interface of TensorFlow.js is strongly based on TensorFlow’s high-level API Keras . Keras code is often only distinguishable from TensorFlow.js code at second glance. Most differences are due to the different language constructs of Python and JavaScript for configuration parameters.

Machine learning with every GPU

TensorFlow.js lets you build machine learning projects from zero. If the necessary data is available, models can be trained and executed directly in the browser. TensorFlow.js uses the graphics card (GPU) of the computer via the browser API WebGL. It does lose some performance because WebGL can only be tricked into executing the matrix multiplication required by TensorFlow.js. But these are necessary because TensorFlow.js, as a machine learning strategy, mainly supports neural networks. These can be mapped very well by matrix multiplications during training as well as during prediction. Here is the first advantage of TensorFlow.js over TensorFlow: While TensorFlow currently only supports NVIDIA GPUs via CUDA, TensorFlow.js works with any graphics card. Listing 1 contains the code to use the high-level API to create a sequential neural network in the browser. Who knows TensorFlows Keras API, comes here very quickly. Tutorials can be found at tensorflow.org.

// create a sequential model
const model = tf.sequential();

// add a fully connected layer with 10 units (neurons)
model.add(tf.layers.dense({units: 10}));

// add a convolutional layer to work on a monochrome 28x28 pixel image with 8
// filter units
model.add(tf.layers.conv2d({
inputShape: [28, 28, 1],
filters: 8
}));

// compile the model like you would do in Keras
// the API speaks for itself
model.compile({
optimizer: 'adam',
loss: 'categoricalCrossentropy',
metrics: ['accuracy']
});

Tensorflow.js – Interact with all browser APIs

Addressing interfaces on different operating systems and devices can still be a painful experience. Not so when developing a browser-based application. Even access to such complex hardware as a camera or microphone is anchored in the HTML standard and supported by all current browsers. The nature of the browser, which is naturally designed for interaction, also suits you here. Interactive applications with a share of machine learning are now easier than ever.

As an example, we have a simple game, Scavenger Hunt , which of course also runs in the browser of a mobile phone and thus brings the most fun. As shown in Figure 1 , in the real world you have to quickly find an object that matches the displayed emoticon. For this purpose, the built-in camera and a trained neural network is used, which can detect the matching objects. Such a model can be used by any JavaScript developer even without machine learning skills.

detect the matching objects with tensorflowimage from entwickler.de

Machine learning without installation on any computer

TensorFlow.js allows you to deploy a model created in advance with TensorFlow. This model may already have been fully or partly trained on strong hardware. In the browser it comes then only to the application or is further trained. Figure 2 shows a Pac-Man variant that can be controlled by different poses. It is based on a pre-trained network, which is trained in the browser on their own poses. This is called transfer-learning.

Pac-Man game controlled by your cameraimage from entwickler.de

The model is converted by a supplied program and can be loaded asynchronously after loading by a line similar to the following:

const model = await tf.loadModel('model.json');

After that, the model is no longer distinguishable from a model created directly in the browser. It can, for. For example, it is very easy to use for prediction, which in turn is executed asynchronously on the GPU:

const example = tf.tensor([[150, 45, 10]]);
const prediction = model.predict(example);
const value = await prediction.data();

In addition to entertainment through games even more useful applications are conceivable here. Navigation or interaction through gestures can be helpful for people with disabilities as well as for people in special situations. And as I said: without any installation, by simply loading a website.

Another example of such position detection is the PoseNet in Figure 3 . It is already so pre-trained that it can recognize the position of face, arms and legs even with several people in the picture. Here, too, the potential of games to controllers of serious applications extends from a certain distance. The use of the PoseNet is again trivial and does not even require basic knowledge in the field of machine learning. Listing 2 briefly outlines how easy it is.

import * as posenet from '@tensorflow-models/posenet';
import * as tf from '@tensorflow/tfjs';

// load the posenet model
const model = await posenet.load();

// get the poses from a video element linked to the camera
const poses = await model.estimateMultiplePoses(video);

// poses contain
// - confidence score
// - x, y positions

position detection with tensorflowimage from entwickler.de

User data does not have to leave the browser

Especially now, where data protection by the DSGVO gets more and more important, some think twice, whether he wants to have a particular cookie on his computer or whether you want to send a statistic of his user data for the improvement of the user experience to the manufacturer of a software. But what about the reversal? The manufacturer provides a general model of how to use software, and similar to the Pac-Man game described above, it is adapted to the individual user using a transfer-learning model. Not much has happened here yet, but the potential is great. Let’s wait for the developments.

Conclusion

Machine learning in the browser does not seem to make much sense to many developers at first. But if you take a closer look, there are applications that no other platform can offer:

  1. Education: You can interact directly with machine-learning concepts and learn by experimenting
  2. Development: If you already have or want to build a JS application, you can use machine learning models or practice it directly
  3. Games: Real-Time Position Estimation only via the camera (ie how the people in front of the camera are moving) or Image Recognition can be coupled directly with games. There are very cool examples of what you can do with it. However, the possibilities go well beyond games
  4. Deployment: You already have a machine learning model and wonder how to get it into production. The browser is a solution. Even finished models can be integrated into the own application without a deep knowledge of machine learning
  5. Interactive visualizations: For interactive clustering or artistic projects

LEAVE A REPLY

Please enter your comment!
Please enter your name here