In addition to being more straightforward, applying the same total amount of compute to a single model doing end to end learning is often better for performance than splitting up compute between multiple models. As far as I'm aware, there aren't any systematic ways to tell when which method will be preferable, this is just a rule of thumb opinion.

Really cool!!!!!

To me, all that means is that the lay people will always be a generation behind from what the rich can afford to run

How do you become a labeller for surge ? I have some background in maths, my native tongue is French, but I speak English and I would be interested in working for surge.

Nah foundational models will be replaced with distributed ones

In vision/robotics this is already the case, low hardware/liw cost requirements is an incredible seller for automotive industry, so large disgusting models are out.

But we still use deep, if anything it's pretty surprising how much is possible with "shallow" models, for specialized domains, but thats still very far from explainable models

If it’s just text have you tried using an LLM like GPT3 with human verification?

>If you train a NN to generate a representative knowledge model that solves a "simple" problem that could have been solved with an explicit solution, you're still doing ML.

I guess my question would be when do you know that what you have is a representative knowledge model rather than just a simple function? Another question that might help clear it up for me is - what would have to change in order for the strandbeest program to be considered machine learning?

The reason Google doesn't bother is that they are aggressive about acquisitions. They're outsourcing the difficult risky work.

[removed]

>ChatGPT is a dialog AI,

OpenAI doesn't even call chatGPT AI or dialog AI.

​

>Non task-specific problem solving at the level of a human. The boundary between advanced AI, especially multi-task learning models, and AGI will get smaller and fuzzy in the coming decades, but there is a difference.

That is closer to the original older definition of AI, I don't think I've seen one credible definition that doesn't include problem solving. AGI just started getting used because AI had been used to much to refer to stuff that isn't AI.

See, it's not bitter lesson 1.0 when you replace "leverage computation" with "leverage large models that require hundreds of GPUs and entire internet". Sutton definitely did not write in his original essay that every bitter cycle ends with:

>breakthrough progress eventually arrives by an approach based on scaling computation

[removed]

Naieve Bayes Classifiers, support vector machines, decision trees, k-means clustering... these are part of machine learning but not AI. They may be sometimes used in AI, but so are processors and memory chips but those aren't AI either.

"Modern AI" is almost entirely made of neural networks, which is merely one of the topics in machine learning. Older AI was based on things like A*, formal logic systems, fuzzy logic, etc., little of which is "machine learning".

[removed]

When I have similar doubt, I ask myself; "forget ML, will statistics help you? Will just defining some rules will help you?" People in that industry already have some kind of idea about how to predict, learn their rules. By the way, I am not suggesting that you should not use ML. I am just asking you to look from a different angle.

Why are you both condescending and factually wrong?

(a) They ARE learning.
(b) The learning definition I used is the one that defines ML according to Tom Mitchell. You thinking it was some kind of useless definition tells more about you.

Pretty useful blog. Thank you Edwin.

seems a bit obvious since foundation models have already been around for 3-5 years

That sounds like a pretty sick machine! I'll check out GPU Direct Storage and see if I can get it working. :-)

[deleted]

[removed]

That was precisely the point.

Amazon started as a sales service and then moved to become platform. Once it was platform, everyone assumed that sales business was too small for them.

And then they started to cannibalize businesses using their platform.

Try to figure out systems that can generalize from smaller amounts of data? It's the big problem we all need to solve anyway.

There's a bunch of promising ideas that need more research:

• Neurosymbolic computing
• Expert systems built out of neural networks
• Memory augmented neural networks
• Differentiable neural computers

Compute is going to get cheaper over time though. My phone today has the FLOPs of a supercomputer from 1999.

Also if LLMs become the next big thing you can expect GPU manufacturers to include more VRAM and more hardware acceleration directed at them.