Yes, we used DistilBERT (and even logistic regression) heavily in my previous startup where data volume was web scale.

Depending on the exact problem, large transformer models can be an overkill. For some straightforward text classification even logistic regression with some feature engineering can hit within 3% point of a transformer, and costs a negligible fraction of them.

Sorry for the ignorant question, but are there any practical applications of this theory?

[removed]

Use byteandbit to load it with 8 bit and then you can run it on a local machine. Basically I think running the model is not a problem, but am not very optimistic about the quality of the generated story. I am using gpt3 to generating ad copies and to be honest it still takes a lot of efforts to tune the prompts to get some decent results, let along using the open-sourced model.

In the beginning God created the heaven and the earth. And the earth was without form, and void; and darkness was upon the face of the deep. And the Spirit of God moved upon the face of the waters. And God said, Let there be light: and there was light....

And God said, Let us make man in our image, after our likeness: and let them have dominion over the fish of the sea, and over the fowl of the air, and over the cattle, and over all the earth, and over every creeping thing that creepeth upon the earth. So God created man in his own image, in the image of God created he him; male and female created he them.

And Jürgen Schmidhuber chastised God for failing to cite his papers since his creation of man and woman are special cases of Artificial Curiosity and Predictability Minimzation.

...it didn't end like we thought it would in the movies. There were no killer machines....there were paper clips, trillions of them....

[removed]

Thanks for the reference I’ll look into it

Thanks! The newest Threadrippers are still based on Zen 3. So, they don't support AVX512. Would definitely like to go with A100s, but we don't have the budget for that.

[deleted]

I don’t think this looks bad at all. I would probably push for A100 80gb gpu’s instead and the latest gen 64-core threadripper.

This is cool. How do we give feedback to the training engine so that it improves over time?

Paperclip stan

Are either of these open source and easily usable as a PyTorch optimizer ?

I'm not sure if we know this for certain, but it appears to be davinci instruct 3 with a custom prompt prefix.

Correct: https://i.imgur.com/civSg94.png

Are you using the "en_core_web_trf" model in Spacy which is based on the roberta-base transformer model?

If that model is still not accurate enough, you may need to look into using the Hugging Face transformers library and try some more recent transformer models, e.g. deberta

It depends. It really varies depending on what parameters you set for the generation. The choice of decoding and output text length can dramatically change the speed and quality of the outcome.

GPT-J-6B model I would say that it is possible to generate 10000 requests in few hours. Using only CPU will take much longer, but you could maybe generate 2000 requests in 24 hours. But again, it is strongly dependent on input and output text length and decoding type.

Are you talking about when inserting an arXiv link to find similar papers? In that case, it is important that the paper being referenced is already stored in the database. If it's a very recent paper (as in less than a week or two old), it won't work. This should be easy to fix, though, by simply scraping the abstract from arxiv.org and using it as the query.

If you're talking about searching for specific papers, I'd be interested to know the queries and the desired result. Feel free to post it here or in a DM 🙂

Here's a fairly accessible free e-book by the principal researcher on the topic, Karl Friston...

https://mitpress.mit.edu/9780262045353/active-inference/

He's got tons of papers. He's one of the most cited scientists alive.

Also, there are lectures and such on YouTube. Just search terms "free energy principle", "active inference", "predictive processing".

Some other good books are "Surfing Uncertainty" by Andy Clark and "The Predictive Mind" by Howhy.

Microsoft has a good checklist to consider if you haven't seen it.

There are many publications on fairness nowadays so I'd also suggest reading some survey papers. Here are a few that have a good number of citations:

• Fairness in Machine Learning: A Survey (2020)
• Fairness in Machine Learning (2020)

I'm pretty sure there are many workshops and conferences on fairness in AI nowadays too that would be good for ideas. There are even ML toolkits to help detect or reduce bias these days, so those would be good to search for.

Hope this helps! Fairness has become a pretty big area over the last several years

Can you reference some works along these lines? "online unsupervised generative models implementing something akin to the free energy principle and active inference"

Exactly. It's not exactly meant to be a zero shot model, but rather a conversational agent. If it were a zero shot model, here is what it could say:

• gebutzeripanim is a new phrase for me
• based on its start it sounds german
• based on its middle it sounds vaguely korean
• based on its end it sounds vaguely turkish
• it might be made up slang

But in conversation you wouldn't just start decomposing stuff you never heard about. Here are what it says if you try to fish out these answers:

• I'm sorry, but I'm not familiar with the term "gebutzeripanim." Could you provide more context or clarify what you are referring to?
• Without more context or information about the word, it is difficult for me to determine whether it has a connection to the German language. "Gebutzeripanim" is not a word that I am familiar with, and a search of online resources did not yield any results. It is possible that this is a made-up or invented word, or that it is used in a specific context that is not familiar to me.
• Without more context or information about the word, it is difficult for me to determine whether it has a connection to the Korean language. "Gebutzeripanim" is not a word that I am familiar with, and a search of online resources did not yield any results. It is possible that this is a made-up or invented word, or that it is used in a specific context that is not familiar to me. It is also possible that the word includes elements from multiple languages or that it is a hybrid of several languages. Without more information, it is difficult to say for sure.
• Without more context or information about the word, it is difficult for me to determine whether it has a connection to the Turkish language. "Gebutzeripanim" is not a word that I am familiar with, and a search of online resources did not yield any results. It is possible that this is a made-up or invented word, or that it is used in a specific context that is not familiar to me. It is also possible that the word includes elements from multiple languages or that it is a hybrid of several languages. Without more information, it is difficult to say for sure.
• It is not possible for me to determine whether "gebutzeripanim" is a made-up or invented word, as I do not have sufficient information about the word. It is possible that "gebutzeripanim" is a slang term that is not widely known or used, or that it is a word that has been created recently and has not yet gained widespread recognition. Without more context or information, it is difficult to say for sure what the word means or how it is used.

It mostly bases it answers on the fact that it doesn't know. And given that it uses past answers as part of the prompts you can see that the second time I ask it for a language it assumes it can be made out of more languages, but not because it understands it, only because I mentioned it.

If you ask it in a new session whether it is made out of words or phrases from several languages, it answers with

> I'm sorry, but I am unable to find any information about a word spelled "gebutzeripanim." It is possible that this is a made-up word or a word from a language that I am not familiar with. Can you provide any context or additional information about the word that might help me to better understand it?

Since it basically needs to explicitly see things in training, it's not really a zero-shot, but rather a few-shot model. There are instances where it seems like it can connect the dots but you can't really say it happens in the general case...

I am not sure what kind of stories you plan to generate, but I would recommend to look at huggingface model repositories. There are many models that have been already finetuned for the specific topic - for example https://huggingface.co/KoboldAI. It can save you lots of money and time.

That just means we'd have more paperclips. I see no downside here.