Chatgpt.

Most commonly used (and successful models) in financial world are typically variations of regression and decision trees (xgboost is being the leader). If you try to do anything else, the powers in place will hang you, cut your head, and urinate in your grave, not necessarily in that order.

Shallow learning?

How do people assume causality when every measured system contains hidden variables?

Part I. Introduction

00:00:00 : Biography

00:02:36 : Harvard hiatus 1: Becoming a DJ

00:07:40 : I really want to make AGI happen (back in 2012)

00:09:00 : Harvard math applicants and culture

00:17:33 : Harvard hiatus 2: Math autodidact

00:21:51 : Friendship with Shing-Tung Yau

00:24:06 : Landing a job at Microsoft Research: Two Fields Medalists are all you need

00:26:13 : Technical intro: The Big Picture

00:28:12 : Whiteboard outline

Part II. Classical Probability Theory

00:37:03 : Law of Large Numbers

00:45:23 : Tensor Programs Preview

00:47:25 : Central Limit Theorem

00:56:55 : Proof of CLT: Moment method

01:02:00 : Moment method explicit computations

Part III. Random Matrix Theory

01:12:45 : Setup

01:16:55 : Moment method for RMT

1:21:21 : Wigner semicircle law

Part IV. Tensor Programs

1:31:04 : Segue using RMT

1:44:22 : TP punchline for RMT

1:46:22 : The Master Theorem (the key result of TP)

1:55:02 : Corollary: Reproof of RMT results

1:56:52 : General definition of a tensor program

Part V. Neural Networks and Machine Learning

2:09:09 : Feed forward neural network (3 layers) example

2:19:16 : Neural network Gaussian Process

2:23:59 : Many large N limits for neural networks

2:27:24 : abc parametrizations (Note: "a" is absorbed into "c" here): variance and learning rate scalings

2:36:54 : Geometry of space of abc parametrizations

2:39:50 : Kernel regime

2:41:35 : Neural tangent kernel

2:43:40 : (No) feature learning

2:48:42 : Maximal feature learning

2:52:33 : Current problems with deep learning

2:55:01 : Hyperparameter transfer (muP)

3:00:31 : Wrap up

Found relevant code at https://github.com/thegregyang/GP4A + all code implementations here

--

Found relevant code at https://github.com/thegregyang/NTK4A + all code implementations here

--

Found relevant code at github.com/microsoft/mup + all code implementations here

--

To opt out from receiving code links, DM me

Shallow learning

You refere to regressions?

I have to accelerate the labelling process of a collection of sounds. To do so, I would like to build a representation from audio data and compute distances/find clusters. Do you know the most frequent representation used and/or the possible embedding techniques ?

That's an innovative training algorithm on usual architectures. We could think of innovative training algorithms on innovative architectures.

Unfortunately this is basically a different type of layer by layer training which doesn't perform better than end to end training in any case that I'm aware of. It also seems very similar to stacking which can be done with any type of model.

Great insights my G. Surely will look into your work. Might contact you in future if I want to contribute something to your idea.

Like the forward forward algorithm Jeff Hinton proposed in nips 2022 ?

that's what random forests are...

random forests and SVMs can also provide significant improvements over simple models (eg. logistic regression). however, for new problems/datasets, it's prudent to try many models.

[removed]

Doesn't even have to be alien. Just the familiar seen from a wholly different perspective.

Like whatever these models are doing here: https://scitechdaily.com/artificial-intelligence-discovers-alternative-physics/

My own work focuses on an alternative to deep learning, called Sparse Predictive Hierarchies (SPH). It is implemented in a library called AOgmaNeo (Python bindings also exist). It does not use backpropagation, runs fully online/incremental/continually (non-i.i.d.). Its main advantages are the online learning but also that it runs super fast. Recently, I was able to play Atari Pong (with learning enabled!) on a Teensy 4.1 microcontroller, and still get 60hz.

If you would like to know more about it, here is a link to a presentation I gave a while back (Google Drive).

Other than my own work, I find the Tsetlin Machine interesting as well.

Any alternative which would be able to solve the same problems would probably require a similar architecture: lot of parameters, deep connections.

There are many alternatives to deep learning on some specific tasks. But I'm not sure that if something is able to outpeform the current way we're doing deep learning on usual DL tasks, it will be something totally different (non-deep, few parameters etc.)

The future of ML regarding tasks we do with deep learning is probably just another kind of deep learning. Perhaps without backpropagation, perhaps with a totally different way to do computations, but still deep and highly parametric.

For some problems like tabular data or time series there are plenty of alternatives, some of them better than deep learning (for a specific application). But those are very different from the cutting edge vision/language models.

😂😅

Hmm interesting 🤔. I didn’t know deepmind has no outside customers. You are 💯% right though. Apparently the only “sells” to Google and its subsidiaries. Thank you for pointing this out. 🙏

An alternative would be a very large set of of 'if' statements

Mostly, giving a source of trained models, and denying the possibility of making the new ones. I see unrestricted use of the big scale general-purpose models as a biggest threat, as they are effectivel "encyclopedias of everything", and can be used for very diverse and unpredictable things.

Who decides is also a very interesting question. Ideally, public consensus, but realisitcally, those who have the capabilities to enforce those limitations.

Have you tried Named Entity Recognition (NER)? You could use NER that involves identifying and classifying named entities (such as people, organizations, and locations) in text. You could use this tool to extract the named entities from each paragraph and then filter them based on specific criteria, such as location or birth place. If you're feeling a bit ambitious, why not try semantic search?