Recent comments in /f/MachineLearning

Decadz OP t1_j41wx24 wrote

Reply to comment by Jumbofive in [D] Are there any papers on optimization-based approaches which combine learned parameter initializations with learned optimisers? by Decadz

Yes, as you said that's what im trying to find out! Will be interesting to know whether you can combine the two approaches into one technique, or have two seperate approaches being used in one system.

Was just enquiring to make sure i'm not going to spend time reinventing the wheel haha. Will also be interesting to have some insight into how the two approach interact, and whether the benefits can stack or if they overlap.

2

Decadz OP t1_j41w1zt wrote

Reply to comment by thchang-opt in [D] Are there any papers on optimization-based approaches which combine learned parameter initializations with learned optimisers? by Decadz

Thanks for the suggestion! Brandon Amos has many great pieces of research. The linked paper is quite long, so I will need to have a more complete reading at a later date to be sure. At a glance though, this tutorial is about meta-optimization theory as opposed to what I was originally asking for which is about application of meta-optimization techniques to learning parameter initialisations + optimizers.

3

lavaboosted OP t1_j41w1fi wrote

Reply to comment by TheGreatHomer in [D] Would you consider the computer program Theo Jansen used to design the Strandbeest (beach walking mechanisms) to be Machine Learning? by lavaboosted

The data is the curve generated by each leg orientation. Each curve in the batch is then scored based on some criteria. If that isn't machine learning then neither is using a neural network and evolutionary algorithm and I think most people would say that it is.

2

Decadz OP t1_j41udg4 wrote

Reply to comment by Optimal-Asshole in [D] Are there any papers on optimization-based approaches which combine learned parameter initializations with learned optimisers? by Decadz

Thanks for the recommendation! I was unaware of this follow up work, which naturally extends Baydin et al. original work [1]. Categorically, I would consider this paper to be more about meta-optimization (theory), similar to [2, 3]. I was looking for more applied meta-optimization work.

[1] Baydin, A. G., et al. (2017). Online learning rate adaptation with hypergradient descent.

[2] Maclaurin, D., et al. (2015). Gradient-based hyperparameter optimization through reversible learning. ICML.

[3] Lorraine, J., et al (2020). Optimizing millions of hyperparameters by implicit differentiation. AISTATS

2

TheGreatHomer t1_j41rida wrote

Reply to comment by pucklermuskau in [D] Would you consider the computer program Theo Jansen used to design the Strandbeest (beach walking mechanisms) to be Machine Learning? by lavaboosted

There is no dataset from which you learn patterns. You usually evaluate objects which are then again used for mutation based on their performance.

Of course it's not happening in a vacuum, but that's not what "data" usually means.

1

Optimal-Asshole t1_j41nlj5 wrote

Reply to [D] Are there any papers on optimization-based approaches which combine learned parameter initializations with learned optimisers? by Decadz

Here’s this paper which uses gradient descent to train the meta layer, and gradient descent to train the hyper parameters of that gradient descent, and so forth. The hyperparameters of the top most meta layer matters less and less as you add meta-depth, I.e add more meta-“layers”.

https://arxiv.org/abs/1909.13371

3