They are certainly still used for RL (and other cases where you don’t have a gradient), but even in those contexts there have been modern advancements that cause the preferred algorithms to diverge from old-school genetic algorithms. For instance, things like Particle Swarm Optimization and the Cross Entropy Method have their conceptual origins in the similar sampling regimes as MCMC approaches, but they’ve become their own entities at this point, outperforming genetic algorithms, and really being unique and broad enough to get their own categories.

I'm new to this-- "whatever" topic
please explain to me this topic of op as if I am a child in TL:DR format.

Ps I did read, but what encryption is this "hero"!?

No, good on you for being that person. I just looked at the prefix of the URL. Thx!

Following!

Sorry to be that person, but that's not Nature, that's Scientific Reports, which is a tier 3 journal in the Nature portfolio. If Nature would be compared to a top conference, Scientific reports would be less than a workshop paper.

Exactly due to the computational demands I don't think genetic algorithms have ever really been "alive", but with compute getting cheaper I could see it seeing success similar to the rise of Deep Learning.

Evolution Strategies as stabilizers without the genetic component are already being deployed quite well e.g. AlphaStar.

Jeff Clune was quite active in that area of research and he recently joined DeepMind.

https://twitter.com/jeffclune/status/1629132544255070209

Would you even want to? Sounds like overkill to me, but maybe I am missing some use case of the embeddings.

I might be in the minority but I strongly believe in unfiltered AI (or a minimal filter, only blocking thing like directions to cool drugs or make weapons). I know they filter it for liability reasons but I wish they didn't.

Yeah it's used for automatic hyperparameter tuning.

Better than a grid or just intuition.

I don't think they are dead. Their popularity for NNs is much lower for sure.

In general, GAs can theoretically solve any problem (if you can formulate a fitness function), given long enough time. Because of that, I think they will always have some use cases.

You can say they have gone extinct in the ecosystem of completion from superior approaches 😂

In one of our recent work https://arxiv.org/abs/2012.14956, we used Genetic Algorithm to attack NLP models in a hard label black box setting, where we do not have access to the confidence scores of the model.

it cannot, the compute still scales quadratically although the memory bottleneck is now gone. however, i see everyone training at 8k or even 16k within two years, which is more than plenty for previously inaccessible problems. for context lengths at the next order of magnitude (say genomics at million basepairs), we will have to see if linear attention (rwkv) pans out, or if recurrent + memory architectures make a comeback.

Is that for everyone or just API/Enterprise users?

[deleted]

[removed]

The Gradient Podcast

Gradient Dissent Podcast

Lex Fridman's podcast has also had all the biggest names in AI as guests.

https://twimlai.com/podcast/twimlai/

Where was this introduced?

We recently published an evolutionary method to sample from the latent space of a variational autoencoder. It is still alive and well. Just a bit niche.

> We also extend FlashAttention to block-sparse attention, yielding an approximate attention algorithm that is faster than any existing approximate attention method.

...

> FlashAttention and block-sparse FlashAttention enable longer context in Transformers, yielding higher quality models (0.7 better perplexity on GPT-2 and 6.4 points of lift on long-document classification) and entirely new capabilities: the first Transformers to achieve better-than-chance performance on the Path-X challenge (seq. length 16K, 61.4% accuracy) and Path-256 (seq. length 64K, 63.1% accuracy).

In the paper bold is done using the block-sparse version. The Path-X (16K length) is done using regular FlashAttention.

OpenAI updated their page to promise they will stop doing that.

It turned out that other models have superior genes.

Can you give a few examples of how genetic algorithms and stochastic gradient estimation are related?

The TWIML AI podcast with Sam Charrington