Recent comments in /f/MachineLearning

Disastrous_Elk_6375 t1_jb8y5r2 wrote

Reply to [D] Neat project that would "fit" onto a 4090? by lifesthateasy

GptNeoX should fit with 8bit and low prompt sizes. GptJ-7B should fit as well with 16bit inference. On smaller models you might even be able to do some finetuning for fun.

There's a couple of coding models from salesforce that you could fit comfortably. Check out FauxPilot for a copilot "clone".

8

lynnharry t1_jb8n8p0 wrote

Reply to comment by florinandrei in [D] The MMSegmentation library from OpenMMLab appears to return the wrong results when computing basic image segmentation metrics such as the Jaccard index (IoU - intersection-over-union). It appears to compute recall (sensitivity) instead of IoU, which artificially inflates the performance metrics. by florinandrei

Pixels with ignore_index does not mean the model's output should also be ignore_index. It means the groundtruth label is not determined on those pixels and whatever your model's output is, its correctness is undetermined.

For those undetermined pixels, we simply ignore those outputs completely.

ignore_index is not used to ignore a specific category during the metric calculation, which is what you're proposing. ignore_index is simply notifying intersect_and_union some areas of the image have undetermined labels and should be ignored, and those areas are marked by the value of ignore_index.

2

deephugs t1_jb7vqn2 wrote

Reply to comment by murrdpirate in [D] I’m a Machine Learning Engineer for FAANG companies. What are some places looking for freelance / contract work for ML? by doctorjuice

Having done ML consulting work through Upwork, my experience is the rate on Upwork is really low compared to what you can get through networks, especially remote Bay Area work. Most Upwork seems to be short timelines, small payouts, and competing against low cost international talent. Any tips for Upwork you can suggest?

9

enn_nafnlaus t1_jb7sxxi wrote

Reply to What is the future of AI in medicine? [D] by adityyya13

I can say this: my mother has struggled for many, many years trying to figure out what's wrong with her and causing her weird, debilitating symptoms. She finally, at long last got a diagnosis that her doctors are pretty confident in: advanced Sjögren's.

Out of curiosity, I punched her symptoms into ChatGPT, and - without access to any test results - Sjögrens was its #2 guess, and it suggested diagnostic tests that she had done and had shown it was Sjögrens. Sjögrens actually isn't super-rare (about a percent or so of the population has it), but usually much milder, and very underdiagnosed.

I think AI tools are seriously underappreciated with respect to proposing new lines of investigation on hard-to-crack cases.

4

florinandrei OP t1_jb7pefb wrote

Reply to comment by Mediocre-Bullfrog686 in [D] The MMSegmentation library from OpenMMLab appears to return the wrong results when computing basic image segmentation metrics such as the Jaccard index (IoU - intersection-over-union). It appears to compute recall (sensitivity) instead of IoU, which artificially inflates the performance metrics. by florinandrei

> Isn't this what the ignore_index is doing?

No, it is not.

Let me repeat: ignore_index cuts holes in both the ground truth label frames, and in the prediction frames coming out of the model. Any pixels in those holes are ignored.

This includes pixels in the predictions from the model. You are ignoring chunks of the model's output.

> How else should we exclude them from the average metric?

By not computing metrics for that pixel value.

average_metric = sum(metric_index1 + metric_index2 + ... + metric_indexN) / N

Simply do not include it in the sum, and then just divide by N-1 instead.

What you are doing is not equivalent to that. What you are doing is: you discard pixels from both label frame and prediction frame based on the shape of some regions in the label frame alone. That makes no sense. Whatever the model's predictions happen to be in those holes, they are ignored even if they have pixel values different from ignore_index.

You are ignoring all the model's predictions in those holes, regardless of their pixel values.

You are discarding pixels from the model's output even if they have values different from ignore_index.

2

Mediocre-Bullfrog686 t1_jb7n704 wrote

Reply to comment by florinandrei in [D] The MMSegmentation library from OpenMMLab appears to return the wrong results when computing basic image segmentation metrics such as the Jaccard index (IoU - intersection-over-union). It appears to compute recall (sensitivity) instead of IoU, which artificially inflates the performance metrics. by florinandrei

>If there is some index you want to ignore altogether, because you are not sure about the quality of the labels, it is best to just exclude it from the calculation of the average metric.

Isn't this what the ignore_index is doing? How else should we exclude them from the average metric? By applying ignore_index we effectively ignore those pixels.

>If some users set ignore_index to the value of the background pixels, that will cut very large holes in everything, therefore discarding a lot of pixels from performance evaluation, and will severely skew the results.

Well the users definitely should not do that. This is then a matter of documentation. We cannot just get rid of ignore_index because (I think) it is used in some existing segmentation datasets.

4

z_fi t1_jb7ihpk wrote

Reply to [D] I’m a Machine Learning Engineer for FAANG companies. What are some places looking for freelance / contract work for ML? by doctorjuice

I’m on a career break, but I was as of December running the AI division of a consulting company.

I will say that finding part time or short term work is very hard. Longer term contract work is relatively easy.

most companies are struggling with the basics - data engineering, data analytics… maybe data science, but with data science you have to be able to talk to the c-suite well and without an mba the lingo is a little hard.

Machine learning projects often require a lot more time to deliver (beyond a proof of concept, and pocs don’t make money) and generally a team rather than an individual, and wayy more stakeholder support than you can muster

Usually ML projects require a lot of data which often puts you into a larger sized business which makes it very difficult to navigate as a freelancer…. You probably need to be in their system when it comes to invoicing and such and so you need to have your ducks in a row where most freelancers don’t. Freelancers, in general, succeed with smaller businesses.

Ignore anyone suggesting upwork.

One avenue I’d recommend is having an honest conversation with consulting company recruiters about what you’re looking for. Stay 1099 or do corp 2 corp. they’ll want you to come on as w2 but be a firm no. Generally these recruiters are looking for easy money and so are you. It’s definitely possible to make a meaningful business relationships here though at your level of seniority you might now know how to play the game at first

28

florinandrei OP t1_jb7cujz wrote

Reply to comment by Mediocre-Bullfrog686 in [D] The MMSegmentation library from OpenMMLab appears to return the wrong results when computing basic image segmentation metrics such as the Jaccard index (IoU - intersection-over-union). It appears to compute recall (sensitivity) instead of IoU, which artificially inflates the performance metrics. by florinandrei

The problem is: the current algorithm cuts holes in the prediction frames, based on ignore_index in the label frames.

Any pixels in the label frames equal to ignore_index will cause pixels in both label frames and prediction frames to be completely ignored from calculations. If some predicted mask pixels fall into those areas, they will be excluded from all calculations. This is the issue that needs to be addressed.

You cannot exclude pixels from the predicted frames based on pixel values in the label frames.

If there is some index you want to ignore altogether, because you are not sure about the quality of the labels, it is best to just exclude it from the calculation of the average metric.

If some users set ignore_index to the value of the background pixels, that will cut very large holes in everything, therefore discarding a lot of pixels from performance evaluation, and will severely skew the results.

1