Recent comments in /f/MachineLearning

UnusualClimberBear t1_j7pdue6 wrote

Reply to comment by EmbarrassedFuel in Model/paper ideas: reinforcement learning with a deterministic environment [D] by EmbarrassedFuel

This is because the information is in the books.

(free online) http://www.cds.caltech.edu/~murray/amwiki/index.php/Main_Page

https://www.amazon.com/Modern-Control-Systems-12th-Edition/dp/0136024580

Yet nonlinear breaks everything there. The usual approach is to linearize at well-chosen positions and compute the control using the closest linearization.

2

FHIR_HL7_Integrator t1_j7pcenv wrote

Reply to comment by NoSleep19 in [D] Should I focus on python or C++? by NoSleep19

I wouldn't worry so much about how much you need to know to be good. Sometimes you need to know something enough to get a job done which is how I think of Python. Reality is python is a general purpose language, it's useful in about a million different ways and it's really not a difficult language. Everyone should have python in their toolbox. So just start using python. Use it for fun little projects that aren't necessarily school work. C++ is probably going to be the more difficult of the two. C++ is what you'll want to do when you are writing anything that needs to be really fast.

I'd take the classes in C++ and teach myself python through fun projects

Someone who is a skilled with C++ is an asset and often a useful and desirable part of a research or implementation team.

1

EmbarrassedFuel OP t1_j7p519o wrote

Reply to comment by jimmymvp in Model/paper ideas: reinforcement learning with a deterministic environment [D] by EmbarrassedFuel

Basically given some predicted environment state, going forward for say 100 time steps, we need to find an optimal cost course of action. Although the environment state has been predicted, for the purposes of this task the agent can consider it deterministic. The agent has one variable of internal state and can take actions to increase or decrease this value based on interactions with the environment. We can then calculate the new cost over the given time horizon by simulating the actions chosen at each step, but this simulation is fundamentally sequential and wouldn't allow backpropagation of gradients.

>you can go with sampling approaches

What exactly do you mean by this? something like REINFORCE?

> I guess it is if you're using a MILP approach.

Not sure I follow here, but I'm not using a MILP (as in mixed integer linear program). At the moment I'm using a linear programming approximation and heuristics, which doesn't generalize well.

> some combination of MCTS with value function learning

I think this could work, however without looking into it I'm not sure that it would work at inference time in my resource-constrained setting

1

EmbarrassedFuel OP t1_j7p3xc1 wrote

Reply to comment by UnusualClimberBear in Model/paper ideas: reinforcement learning with a deterministic environment [D] by EmbarrassedFuel

I haven't been able to find anything about optimal control with all of:

  • non-linear dynamics/model
  • non-linear constraints
  • both discrete and continuously parameterized actions in the output space

but in general, discovery of papers/techniques in control theory seems to be much harder for some reason

1

jimmymvp t1_j7oybk9 wrote

Reply to Model/paper ideas: reinforcement learning with a deterministic environment [D] by EmbarrassedFuel

Ok, first off, I'm very curious what's the actual problem that you're solving. Can you describe it a bit more in detail or give a link?

If you have a perfect model that's cheap to compute, you can go with sampling approaches, I don't know how your constraints look like though. If your state/action space is too big, you might want to reduce it somehow by learning an embedding.

Is the model differentiable? I guess it is if you're using a MILP approach.

I guess some combination of MCTS with value function learning is plausible if your search space is big, such as it's done with alpha zero etc. I find the hybrid aspect of it very interesting though. It sounds like if you want to do amortized search, you need to combine MCTS and search in continuous space (sampling). Should be simple enough with a perfect model. Probably some ideas from mu zero would come in handy.

1