In my opinion, if your AI is not your primary business then AI on edge makes a lot of sense. You’re trading freedom of your model for security as the model maintainer.
Example: Your AI does style transfer for makeup. Then it’s amazing to run this on the browser on the client and get more people to click purchase for their makeup or even hair dye packages.
However, if your model is proprietary and your work is to apply the makeup/hairdye it’s harder to protect your model, as it must be downloaded to the client.

I believe more and more people will lean towards scalability and even depend on cool things like federated learning, and to do that we’ll all need to have a JavaScript arm of our MLOps

Got it, nice example Gant. So depending on the use case it can certainly make sense to have client- and server-based deployments in parallel, at least in the short- to medium-term.

absolutely. And you can easily convert TF to TFJS models. Most of them convert just fine.

That would have been my follow-up question 🙂

Thanks a lot for your response

If I understand correctly, you’re interested in my preferred backend solution for a web-hosted model.
I think microservices/cloud functions are perfect for models in the backend, because they are very “pay as you go” with infinite scalability.
So I would prefer setting up an AWS lambda that handles a result and reports back the results.

These are not limited by GPUs either. The whole problem of dynamically allocating resources for your server model is handled by the service, and faster and more scalable than either of us could hope.

The largest benefit, is that when the model is not in use, the costs are quite minimal. I’ve got several free AWS files/services I have released in the wild from past conference talks. They’ve never gone over $10 USD a month.

Okey, I got it. Thankss :heart:

In Python, there are many nice libraries for data augmentation and different image manipulations. I’m wondering what happens when we move to the browser. Do we need to re-implement all this functionality in JS?

So if you’re needing the power of python libs, I generally use Python to train for my advanced networks and then convert it down to JS.
In the book, I show you object localization and detection. Both of those I decided to do in Python for ease and then convert over.
But you can definitely do it in JS, and I show how.
In the book the Tic Tac Toe model was trained in JS, and in chapters 8, 9, 10, 11, 12 all models are trained in JS.

I think Python wins in having a history of lots of libs. But the speed of training in Node is no joke!

What are some good image manipulation libraries in node?

Similar to imgaug is this: https://github.com/piercus/image-augment

I think you’ll see more and more JS level tools coming in the next 2 years

Oh nice, thank you!

using more: web developers are playing with all kinds of amazing tech and will continue to grow. I think in quantity the current winner is web devs.
However, in quality, I think the best projects are coming from seasoned AI veterans who are leveraging AI in web. The most profitable and impressive projects are people who know how to make this new avenue of AI sing.

I expect a basic level of JavaScript skills. But all the code is available. If you’ve been programming for a while, you can probably learn a lot about JavaScript using the book. You might need to take a moment to go look up a function, but if you know Python you’ll probably be just fine.

Is there a good learn-JS-in-15-minutes tutorial for those who already know how to program?

I think if you just read/tinker with some js you can pick it up in 15mins (just remember those curly brackets ;)) (and also async stuff can trip you up if you’re not careful)

Glancing through this, here’s a 1hr jumpstart that seems decent
https://www.youtube.com/watch?v=IEf1KAcK6A8

Yes. TFJS will always lag behind. However, in a few ways it is ahead. TFJS 1.0 was using TF 2 syntax before TF 2 was released.
I think you’ll continue to see TFJS parity be a priority, but yes, lagging behind.

Thanks for your answer. I hope this project will get more attention thus voluntary contribution from the open source community.

That will be awesome for sure

Yes!

I had to do my own graphics. I did OK

Some of my examples were just plain silly

The editors were AMAZING though, they loved everything I had

I’m quite proud of this diagram… but I know what they make will be better for print

Also, I used coworkers in examples. This will let me know if they actually read the book

I was worried if some of my graphics would work in B&W, but they gave them the thumbs up!!!!

Wow, looks interesting already

TensorFlow.js can run on client or server.
So if you’re running your training on a machine using Node.js you have direct access to the filesystem via Node.js. Then you’re just managing stream and RAM.
If you’re running on the browser, you are sandboxed. This means you cannot access the filesystem and you’ll need to load your data via compatible URIs. Browsers have DBs in local storage, RAM, cookies, and all kinds of fun places you can gather data.
Because of the sandbox, I would recommend only minor amounts of training to happen in the browser if you have a significant data source. The browser is much better at utilizing trained models rather than training models.
This does break down when you want customization for your client, however. So if you’re going to do transfer learning or federated learning, you can get real benefits from training in the browser, but yes, it’s constrained.
My suggestion, do base training in Node.js, then ship the model. Do transfer/federated learning in the browser as needed.

Super, looks clearer now

That’s a tough one!!!
I guess the capstone project was pretty hard for me. Bc I had to come up with something that tied all the lessons from the book together into a cohesive unit. It was a fantastic end for the book though! But tough!
Second hardest chapter is chapter 1… I rewrote that damn thing 10xs
My favorite chapter was probably Chapter 10, which was simple image training. It felt fun and rewarding.
Chapter 6 was a secondmost favorite, because we hook a model up to a webcam!

Sounds like a lot of fun!

I have to say the pandemic helped. I was able to get into a writing frenzy. My editor told me, “Ride that wave!” and so I did. The more I wrote the easier it became and I hit a traction point where I realized I was running out of chapters. At that point, I had to assess what makes it in the book and what didn’t.
After that strange hiccup, the finish line was in sight and I used the rest of my frenzy to finish the book.
Since I finished the book, I’ve done a whole lot of nothing! I think I’m recovering. I don’t even read cool projects right now, I binge watched a whole season of a Netflix show.

How did get into this flow mode? Or it just happened itself?

To be honest, I didn’t get into it at first. I had so much trouble writing chapter 1. However, like gradient decent, each time I took a step that felt right, I leaned into it. Next thing I knew I was running.

One thing that reallllly helped, was having people review chapters. That helped me see large leaps of logic I made, and when I went back to fill in the missing gaps, my mindset carried me to how they should see the next thing and the next. Sort of like reading the book from an outside perspective, I had a conversation with a hypothetical audience.

I feel like web frameworks that use Python are generally treated as second-class in the web world. Node for cloud etc is easy.
Node has even outperformed Python in quite a few benchmarks.
IMO, if you’re using backend, it needs to match the team maintaining it. Focus on dev preference.

The big advantage is that the code is portable to client, and IoT

Similar to portability, you can provide the same computations in the client, on customer hardware, and at scale on the server.

I’m just trying to understand why I don’t see more discussion of TFJS on the backend? I don’t have stats, but I see much more traffic about TF+Python than TFJS in Node. Like you said, I’d expect the web backend work to be more JS/Java, but I just don’t see much about that in the wild.

Wendy Mak’s comment about async/await makes me think that Data Scientists might just be more comfortable with Python’s less asynchronous nature?

In my opinion it’s a momentum thing. Same reason JavaScript is everywhere.
Python was the spoken language for primary data culture, so it will take time for new solutions to root.

FYI, tfjs-node-gpu works just fine in node and uses GPUs when available using the main TF Cuda backend. Interesting question about async JS. Perhaps many folks feel more comfortable with the more traditional Python TF binding?

I’m find with async await. It’s common in JS, so it feels natural.
As for the second part, tfjs-node-gpu works great for accesing the GPU on node.
The browser uses WebGL, and they have an experiment with WebGPU which is promising.
Lastly, they are doing some amazing stuff in WASM with threads! I know this is a bit off-topic, but it’s cool.
CUDA now has JavaScript API, so I imagine tfjs-node-gpu will just keep getting better and better. It worked great for me.

> I’m find with async await. It’s common in JS, so it feels natural.
ah, cool, although, I think I like synchronus better lol – the async things gave me no end of headaches when I was a js dev 😂 (well, I was also trying to munge together something that takes about 10 steps each of which has a callback…)

Thanks for the GPU.js tip!

Oh, GPU.js is a different library. Brain.js uses GPU.js
WebGL and WebGPU are being supported as backends for TFJS

we need more acronyms ^ 😂

I know! I hadn’t heard there was a (limited) JS Cuda binding until you mentioned it.

Federated learning is pretty complicated, and there’s the idea and implementation details.
The idea (at least in my head).

1. Create a model
   
2. Send copies to people
   
3. Use transfer learning to personalize and improve the model
   
4. Send back the improved models (NOT the user’s data)
   
5. Improve the core model that is distributed.
   I’m unaware of the FL(oC) specifics, but I significantly believe Google has a lot of benefit in getting federated learning going. I think you can expect TFJS to be a core contributor to this.
   I know Google is seeking lots of adjustments to W3C standards based on TFJS, but it’s hard to know which pieces are prioritized and why.

Thanks for your valuable answer. When I googled FL with TF.js quickly, I could find only one TF.js project and it is not maintained. But I found it quite strange, because JavaScript is the default option if lots of different browsers are/must be involved. This was the motivation of my question. I am wishing that a good FL(oC) Framework on TF.js will appear soon.

I have no doubt that this will emerge. I think there’s a lot of green opportunities in TFJS for sure.