Very familiar with their (excellent) work. Their code is freely available as well: https://pubsonline.informs.org/doi/10.1287/mksc.2017.1050

Can send paper directly if you can’t download or find it in Google scholar.

https://youtu.be/Hwj_9wMXDVo will explain why linear regression can map non-linear functions. Just have some features squared or cubed, etc. That allows linear regression to map the most complicated non-linear functions possible.

Thanks

Any thoughts on this package?

https://www.sourcedefender.co.uk/

I'll look into pyarmor. I just found this as well https://www.sourcedefender.co.uk/ Any idea if that would work? Thanks.

> Read up on boosting.

What could a good reading list look like? I read the original papers which introduced functional gradient descent (the theoretical underpinning of boosting), but I can't say they shed much light on these techniques for me.

Is there more recommended reading to study boosting? Anything more recent, maybe? Any textbook treatments?

True but practically training a gpt model is not computationally cheap. I think instead of making such generalized language models we need to focus more one subject specific language models.

Thanks! Yes, there are many similar things. But the ChatGPT seems to have the most amazing performance.

This was generated by chatgpt, 100%.

I’m only marginally familiar with Greg’s work (skimmed some papers and listened to his talks) but i believe that both criticisms are addressed.

1. Tensor programs consider discrete time (stochastic) learning algorithms stopped at T steps in place of continuous time gradient flow until convergence (the latter is used in standard neural tangent kernel literature), hence I think the infinite width limit varies depending on the algorithm and also the order of minibatches.

2. They identify infinite width limits where representation learning happens and where it doesn’t. The behaviour changes by varying how to scale with width parameters of the weights distribution of the input, output, and middle layers and the learning rate. In particular they propose to use a limit where representation (they call them features) is maximally learned. In contrast in neural tangent kernel the representation stays fixed.

It's a sad story as I put a lot of time on generation during these years. Any possible suggestions that our research can focus on?

https://github.com/lucidrains/PaLM-rlhf-pytorch

Similar to chat get architecture you can play with this

[deleted]

That's really interesting, thanks for the share. Though I wonder if most decision trees still don't converge upon the same solutions as a neural network, even if they're capable of representing the same solutions. If trees don't converge on the same solutions, and NNs outperform trees, that would mean NNs are still needed for training, then the models can be optimized for run-time by basing a tree off the NN.

It depends on what you want to do:

• If you just want to apply NER, I'd recommend Spacy because it's fast and they have pretrained models for many languages.
• If you're looking to fine-tune or train your own NER, either Spacy or Huggingface to use BERT.
• If you're looking to build your own neural network architecture for NER, PyTorch is most popular.

Haha, that would look somewhat like a tree!

You're right that it's just a matrix multiply of a one-hot encoding. Though representing it as an embedding layer is just faster.

I wouldn't call it a fully-connected layer though. In a fully-connected layer, the input to the matrix multiply is the output of everything in the previous layer, not just the output of a single unit. The weights that multiply the output(s) of the first unit are not the same weights multiplying the output of any other unit.

It's more like a length 1 convolution that projects the one-hot vocab down to the embedding space.

What do you want to do with it?

For tabular data of a few million rows or less, you're often much better off using XGBoost, or one of the other boosting libraries. Read up on boosting. It is an alternative approach to deep learning. Technically it can also be used with neural networks, including deep learning, but in practice it is not often used with it because boosting relies on multiple weak learners whereas deep learning has long training times for the creation of one strong learner.

XGBoost and CatBoost have won many many Kaggle competitions. My former employer trained all their production models using XGBoost, and they were modeling people's credit scores. There are many reasons to use XGBoost including speed of training (much faster to train than deep neural networks) and interpretability (easier to interpret the model's decision-making process with XGBoost, because under the hood it's just decision trees).

I mostly use XGBoost, and sometimes fairly simple LSTMs. I use them primarily for financial modeling. XGBoost works well and the fast training times let me do optimization across a wide range of model parameters, without spending a bunch of money on GPUs.

If you want to do image analysis though, you do need deep learning for state-of-the-art. Ditto reinforcement learning. Ditto several other types of problems.

So, it depends.

Yeah that's pretty common. If you'd like to do more machine learning, as your team and company grows you might try asking your boss to hire more SDEs so that you can spend more time with machine learning. Or alternatively, ask for more training so that the backend engineering goes more quickly.

As for "keeping up with the field", I don't recommend worrying about it. It's challenging, maybe impossible, to actually stay up to date on everything even if it's only ML. I find it's better to make a habit of learning something every day, however small, and focus on the growth aspect rather than some sense of "falling behind".

Yeah I was using neural poly which is equivalent to wavenet. What I discovered is it will always say the same sentence, and usually the same word used in the same way, the same way, regardless of context clues. “My gosh.” Would always render exactly the same way. Really needs paragraph or dialogue driven context, as well as a bit of randomization. In a book where an author has a repetitive goto word or phrase it’s killer.

Most likely you are using the original Polly method, which is based on gluing together sounds of different phonemes. That produces monotone speech.

Try Google wavenet. It's available through google cloud api just like Polly.

There's a neural version of Polly, but I never tried it.

[removed]

I agree with what I think you’re asserting, that the number of variables that might contribute to triggering any sort of event is incalculable, but as a matter of practicality, I assume we have to view causality as a gradient of probability. In other words, we account for as many variables as we can, and make a “best guess” based on what we know. We can make “predictions” based on that until we know more, and then we redefine our overall system over time.

Edit: a word

Influence between variables is usually a measure of probability within sequences of information events, or system states. Correlation would be a measure of relatedness between individuated states of the system. Causation is inherently time/delta based, whereas correlation is asynchronous

you can use recurrent block transformers, and consider one gpt request as a block

How about Microsoft's 2019 1B investment in OpenAI then?