what did the person link? Lol why is everything getting deleted in this thread?

Yeah, the skip connections allow higher layers to have access to information from lower layers. Same thing goes for U-Nets; they're basically an autoencoder with skip connections.

This is my preferred interpretation of RESNETs too.

I'm a first year student going into computer science majoring in AI, I'm just wondering do machine learning need to know about the techniques that you learn from leetcode or not?
Is machine learning mostly about the architecture of the neural network, how to arrange it to optimise the outcome.
In the future if I wanted to be a machine learning engineer to work at for example openai (ai company) would the interview include leetcode/hackerrank questions?

Bonus for the bug report? Think of all the effort and adrenaline that went into this.

That’s really interesting. I was hoping for something self hosted, but this may give better quality which is important. I’ll check this out sometime this week. Thanks!

Question on transformer architecture:

If the task is simply to generate data given a context of data generated so far (such as in the case GPT-3), then can the architecture be simplified?

(The separation of the encoder and decoder layers seems arbitrary when they are processing the exact same data)

Look at things like replika.ai that give you a "friend" to chat with. Now imagine someone evil using that to run a romance scam.

Sure the success rate is low, but it can search for millions of potential victims at once. The cost of operation is almost zero compared to human-run scams.

On the other hand, it also gives us better tools to protect against it. We can use LLMs to examine messages and spot scams. People who are lonely enough to fall for a romance scam may compensate for their loneliness by chatting with friendly or sexy chatbots.

In the modern economy the best way to make a lot of money is to make a product that a lot of people are willing to pay money for. You can make some money scamming people, but nothing close to the money you'd make by creating the next iphone-level invention.

Also, that's not a problem of AI alignment, that's a problem of human alignment. The same problem applies to the current world or the world a thousand years ago.

But in a sense I do agree; the biggest threat from AI is not that it will go Ultron, but that humans will use it to fight our own petty struggles. Future armies will be run by AI, and weapons of war will be even more terrifying than now.

How would the AI know it’s profiling you and not the other AI you’ve set up to do all of those things for you?

It seems to me that the default behavior is going to be to make as much money as possible for whoever trained the model with only the most superficial moral constraints. Are you sure that isn't evil?

It depends on whether it's exploiting my psychology to sell me something I don't need, or if it's gathering information to find something that may actually be useful for me. I suspect the latter is a more useful strategy in the long run because people tend to adjust to counter psychological exploits.

If I'm shown an advertisement for something I actually want... that doesn't sound bad? I certainly don't like ads for irrelevant things like penis enlargement.

Lots of thanks! I didn't receive training from software engineering perspective, which seems to be an important aspect in machine learning.

Availability in GPU is terrible. Colab is better because you don't have to wait for a GPU that is usually snagged in seconds.

Availability in GPU is terrible in paper space. I would rather get colab for that and a VM for heavy loads. I got a refund when it took me a day to find a GPU. I don't have time to watch 24/7 for a GPU that is snagged in seconds. This was in the payed option.

The lucky thing is that neural networks aren't evil by default; they're useless and random by default. If you don't give them a goal they just sit there and emit random garbage.

Lack of controllability is a major obstacle to the usability of language models or image generators, so there's lots of people working on it. In the process, they will learn techniques that we can use to control future superintelligent AI.

Yeah I guess I'm pretty pessimistic about the possibility of aligned AI. Even if we dedicated more resources to it, it's a very hard problem. We don't know which model is going to end up being the first AGI and if that model isn't aligned then we won't get a second chance. We're not good at getting things right on the first try. We have to iterate. Look how many of Elon Musk's rockets blew up before they started working reliably.

Right now I see more of an AI arms race between the big tech companies than an alignment focused research program. Sure Microsoft wants aligned AI but it's important that they build it before Google, so if it's aligned enough to produce PC text most of the time that might be good enough.

We will control what the robots want, because we designed them.

That's the core of AI alignment; controlling the AI's goals.

Why are the robots going to want to keep you around if you don't do anything useful?

>I wouldn’t say it’s common to design networks with information flow in mind

I disagree. The entire point of the attention mechanism in transformers is to have a second neural network to control the flow of information.

Similarly, the autoencoder structure is ubiquitous these days, and it's based around the idea of forcing information to flow through a bottleneck. Some information must be thrown away, so the neural network learns which parts of the data are most important to keep, and you get a good understanding of the structure of the data.

I'd say many of the recent great ideas in the field have come from manipulating information flow in interesting ways.

[deleted]

Good point — what's shown in the paper (just skimmed through it) seems quite promising, wonder why this approach isn't seen more in literature

iirc Google cloud actually offers this? https://cloud.google.com/text-to-speech/custom-voice/docs#:~:text=The%20Cloud%20Text%2Dto%2DSpeech,Text%2Dto%2DSpeech%20API.

[deleted]

100% agree. Even some tutorials from frameworks like Tensorflow