How could we even tell if it was? You can't even prove to me that you're sentient.

We don't have tools to study consciousness, or an understanding of the principles it operates on.

Who's talking about intelligence? Of course artificial intelligence is intelligence. It's in the name. I'm saying it's not sentient.

I hope the API doesn't have the same restrictions as https://chat.openai.com

Check out robot brains by Pieter Abbeel

AFAIK, flash attention is just a very efficient implementation of attention, so still quadratic in the sequence length. Can this be a sustainable solution for when context windows go to 100s of thousands?

Who says intelligence has to work exactly like our brain?

A Boeing 747 is very different from a bird, even though they fly on the same principles.

I recently used a genetic algorithm in a supply chain application. Not the fastest algo but we made it work for our purpose

[deleted]

Criteria for tokens ? Complex, longer the prompt more tokens it'll use ?

It's not the best at Q&A or chat (yet), but kudos for all the work behind this super interesting approach. Maybe with time it will continue to improve, and I like seeing non-transformer methods showing some potential.

I don't think he actually "worked on Google's AI", as in being involved in the research and development part.

Genetic algorithms require you to create a population where the genetic operators are applied (mutation, crossover and selection).

Creating a population of neural networks implies having multiple slightly different copies of the neural network to be optimized (i.e.: the population).

This can be more computationally expensive than other techniques which will do all the learning "in-place".

Sorry i meant the really large scale models. Nobody has gotten a gpt-3/chinchilla etc scale model to actually distill properly.

DistillBERT worked though?

Wasn't this settled once and for all when they had the same exact model he worked on claim (very convincingly) that it was a frog and doing frog things, or something like this?

>You can only run what fits on your available hardware.

Precisely.

Distillation doesn't work for token predicting language models for some reason.

Certainly that is also a possibility. Or they might have done teacher student distillation.

Yep, or a mix between the two.

GLM-130B quantized to int4, OPT and BLOOM int8,

https://arxiv.org/pdf/2210.02414.pdf

Often you'll want to keep the first and last layer as int8 and can do everything else int4. You can quantize based on the layers sensitivity, etc. I also (vaguely) recall a mix of 8bit for weights, and 4bits for biases (or vice versa?),

Here is a survey on quantization methods, for mixed int8/int4 see the section IV. ADVANCED CONCEPTS: QUANTIZATION BELOW 8 BITS

https://arxiv.org/pdf/2103.13630.pdf

Here is a talk on auto48 (automatic mixed int4/int8 quantization)

https://www.nvidia.com/en-us/on-demand/session/gtcspring22-s41611/

Evolved antenna

>In radio communications, an evolved antenna is an antenna designed fully or substantially by an automatic computer design program that uses an evolutionary algorithm that mimics Darwinian evolution. This procedure has been used in recent years to design a few antennas for mission-critical applications involving stringent, conflicting, or unusual design requirements, such as unusual radiation patterns, for which none of the many existing antenna types are adequate.

^([ )^(F.A.Q)^( | )^(Opt Out)^( | )^(Opt Out Of Subreddit)^( | )^(GitHub)^( ] Downvote to remove | v1.5)

My fave has always been the radio antenna designed by a GA

thanks a lot for this. will definitely take a look

LLMs can quantize to 8 bit or 4 bit?

GAs are not great for expensive-to-evaluate functions. And those have become kind of relevant lately.

Source about Google using ViT?