Machine learning with every GPU
TensorFlow.js lets you build machine learning projects from zero. If the necessary data is available, models can be trained and executed directly in the browser. TensorFlow.js uses the graphics card (GPU) of the computer via the browser API WebGL. It does lose some performance because WebGL can only be tricked into executing the matrix multiplication required by TensorFlow.js. But these are necessary because TensorFlow.js, as a machine learning strategy, mainly supports neural networks. These can be mapped very well by matrix multiplications during training as well as during prediction. Here is the first advantage of TensorFlow.js over TensorFlow: While TensorFlow currently only supports NVIDIA GPUs via CUDA, TensorFlow.js works with any graphics card. Listing 1 contains the code to use the high-level API to create a sequential neural network in the browser. Who knows TensorFlows Keras API, comes here very quickly. Tutorials can be found at tensorflow.org.
const model = tf.sequential();
// add a fully connected layer with 10 units (neurons)
// add a convolutional layer to work on a monochrome 28x28 pixel image with 8
// filter units
inputShape: [28, 28, 1],
// compile the model like you would do in Keras
// the API speaks for itself
Tensorflow.js – Interact with all browser APIs
Addressing interfaces on different operating systems and devices can still be a painful experience. Not so when developing a browser-based application. Even access to such complex hardware as a camera or microphone is anchored in the HTML standard and supported by all current browsers. The nature of the browser, which is naturally designed for interaction, also suits you here. Interactive applications with a share of machine learning are now easier than ever.
image from entwickler.de
Machine learning without installation on any computer
TensorFlow.js allows you to deploy a model created in advance with TensorFlow. This model may already have been fully or partly trained on strong hardware. In the browser it comes then only to the application or is further trained. Figure 2 shows a Pac-Man variant that can be controlled by different poses. It is based on a pre-trained network, which is trained in the browser on their own poses. This is called transfer-learning.
image from entwickler.de
The model is converted by a supplied program and can be loaded asynchronously after loading by a line similar to the following:
After that, the model is no longer distinguishable from a model created directly in the browser. It can, for. For example, it is very easy to use for prediction, which in turn is executed asynchronously on the GPU:
const prediction = model.predict(example);
const value = await prediction.data();
In addition to entertainment through games even more useful applications are conceivable here. Navigation or interaction through gestures can be helpful for people with disabilities as well as for people in special situations. And as I said: without any installation, by simply loading a website.
Another example of such position detection is the PoseNet in Figure 3 . It is already so pre-trained that it can recognize the position of face, arms and legs even with several people in the picture. Here, too, the potential of games to controllers of serious applications extends from a certain distance. The use of the PoseNet is again trivial and does not even require basic knowledge in the field of machine learning. Listing 2 briefly outlines how easy it is.
import * as tf from '@tensorflow/tfjs';
// load the posenet model
const model = await posenet.load();
// get the poses from a video element linked to the camera
const poses = await model.estimateMultiplePoses(video);
// poses contain
// - confidence score
// - x, y positions
image from entwickler.de
User data does not have to leave the browser
Especially now, where data protection by the DSGVO gets more and more important, some think twice, whether he wants to have a particular cookie on his computer or whether you want to send a statistic of his user data for the improvement of the user experience to the manufacturer of a software. But what about the reversal? The manufacturer provides a general model of how to use software, and similar to the Pac-Man game described above, it is adapted to the individual user using a transfer-learning model. Not much has happened here yet, but the potential is great. Let’s wait for the developments.
Machine learning in the browser does not seem to make much sense to many developers at first. But if you take a closer look, there are applications that no other platform can offer:
- Education: You can interact directly with machine-learning concepts and learn by experimenting
- Development: If you already have or want to build a JS application, you can use machine learning models or practice it directly
- Games: Real-Time Position Estimation only via the camera (ie how the people in front of the camera are moving) or Image Recognition can be coupled directly with games. There are very cool examples of what you can do with it. However, the possibilities go well beyond games
- Deployment: You already have a machine learning model and wonder how to get it into production. The browser is a solution. Even finished models can be integrated into the own application without a deep knowledge of machine learning
- Interactive visualizations: For interactive clustering or artistic projects