1

TL;DR: I am expected to work on a dataset with missing/wrong information about measurement units. Is this a common practice?


Longer version:

I am studying a Masters's program composed mostly of machine learning. This semester I need to complete a 12 credit Project Module. Project Module means that there are no lectures, just a problem formulation, consultations with a teacher, and then a final report, source code and presentation are submitted.

The problem is to identify rocks by melting them with a laser and measuring the spectrum of the resulting plasma.

The training data I have is a bunch of measured spectra (example visualization). Each file comprises of comma separated tuples (wevelength, intensity).

My teacher didn't have the time to figure out what is being measured when we say "intensity" and sent me the "example" above to figure for myself. Well, I figured it's radiance, based on information of the same source as the example I was sent(here)

Now, radiance is measured in watt divided by solid angle and surface area. This is strictly non-negative. The example graph also cuts the ordinate at 0.

Yet when I look at the provided dataset, I can identify regions such as this one:

 188.5999999999995,15.175680381419365
 188.6999999999995,0.872615069518224
 188.7999999999995,-8.345751621890303
 188.8999999999995,-10.279252960079765
 188.9999999999995,-4.22895372957017
 189.09999999999948,3.978827692507486
 189.19999999999948,16.603400130260713
 189.29999999999947,-12.791825667786197
 189.39999999999947,-4.859145351410255
 189.49999999999946,4.452390240582065
 189.59999999999945,-14.106010857508583
 189.69999999999945,7.039306431256487
 189.79999999999944,-7.3039003395809505

Negative line intensities with values far from 0.

And here is my question. Both my teacher and a couple of more knowledgeable colleagues agree that there is nothing wrong with this:

The data you have is real world data collected with a hand held. Your model should be able to deal with the data as it is. You can apply preprocessing steps if you want to avoid negative data values.

To me it is absurd to even consider a dataset without first sanity checking it. Just as unacceptable I would call working with a preprocessed dataset without knowing how was it preprocessed (maybe I am seeing centered values, but no one told me the mean; maybe I am seeing a logarithmic scale).

Is it customary for a task to be presented this way?

  • 3
    Data is never perfect, and often the hard part in machine learning is cleaning the data. But, one must decide carefully whether these negative values indicate that there was some fatal flaw / error in the way the data was collected, and if there is some fundamental misunderstanding in how the measurements are being interpreted. If it is just the case that a smallish subset of the data are invalid, then that is normal and you can deal with it by cleaning the data or using a method that is robust against outliers. If there was a systematic error when collecting data, that might be much worse. – littleO May 19 at 11:07
  • 4
    I would say that it would be very helpful to understand why these negative values are occurring. It suggests that perhaps what is being measured was not radiance after all. It is important to understand the data as well as possible; we can't just shrug off things that might be hinting at a deeper problem. – littleO May 19 at 11:11
  • 2
    this is a computer science, no academia question – user48953094 May 19 at 12:45
  • 1
    @user847982 I was wondering on which site to post. I did it here, because the actual core question is should I drop this shady course and not how do I successfully work with crappy data. By all means, do flag the question for migration and the moderators will decide better than me. – Vorac May 19 at 13:37
  • 2
    Data you don't understand -> a method you don't understand -> report a result you don't understand, A+. Do an especially good job and maybe you can turn it into a publication? ;) I am only a little kidding, but its a pretty common formula in the field, for good or ill. – BrianH May 19 at 17:02
3

From the point of pure exercise in machine learning, you do not care whether the data is reliable and whether it was collected or taken from the ceiling. Your task is just to design a recognition algorithm that classifies the objects into groups based on the numbers associated with them. If you can do it, you pass, if not, you fail.

From the perspective of practical recognition of rocks, your algorithm (if it works) will be useful only if you trained it on accurate data or data with not too much noise and if you know what the numbers mean, so if you later are given the information in different units or on an altogether different scale (say, logarithmic instead of linear), you'll know how to convert before feeding them into the machine. If your algorithm is intended for real use, the people who requested the work should be most interested in getting things right, so you should be able to ask them as many relevant questions as you want. If it is just a practice problem, just forget about "rocks", "radiance", etc. View all that as an abstract classification problem about objects and numbers without any meaning whatsoever.

From what you are saying, it looks like you are in the first case scenario. I agree that the exercise is about as meaningful as the exercise in computing the area of the triangle with sides 6,7,8 and the altitude of length 5 drawn to the side of length 7. My daughter really had that as a problem on one of her exams and her teacher replied to my objections that at that stage of learning he just wanted to make sure that kids knew which side to multiply by the height. He wasn't a bad math. teacher overall, by the way, just a bit reckless about the details that weren't directly related to the current topic. Apparently, your professors have a similar attitude.

  • I learned some new information in the meantime. My instructor is also working on the project himself. Other students were given the same project with the same amount of explanation and without being informed there is further effort on the project. Conclusions are the reader's to draw. – Vorac May 23 at 14:21
0

This seems perfectly reasonable for a project at the Master's level. At that level you should be getting away from canned problems with perfect data, and actually tackling the complications that occur in the real world. Having to sort out bad data from good data and figure out normalizations and scaling transforms is a typical part of every real life application of machine learning.

I would also expect your advisor to be available to consult with on a regular (say weekly) basis. Your advisor probably won't be telling you exactly what to do next, rather you should be explaining the problems you are running into, and how you are planning to solve them. Your advisor should then be able tell you whether your approach is reasonable, and suggest other approaches if it is not.

  • Why are they even telling me about 'rocks', 'hand-held devices' or 'intensities'? Why not just give me a bunch of numbers and stage the task as we want you to find a bunch of numbers which look like the bunches of number we have provided you, but you aren't required to state in what way do they look alike? – Vorac May 19 at 18:47
  • @Vorac That would be appropriate for an undergraduate course in machine learning, but this is a Master's level project, and it is much closer to what you can expect to meet up with in the real world, except without your advisor hanging around to provide free consulting. The real world is messy and the people handing you the numbers may not even understand them themselves. – Charles E. Grant May 19 at 19:00
  • @Vorac Why are they even telling me about 'rocks', 'hand-held devices' or 'intensities'? That is an unfortunate custom: some "smart" educationist decided at some point that "real life setting" helps students to understand the material better, while IMHO it mainly just confuses them and prevents them from reaching the required level of abstraction. So we have calculus textbooks filled with pearls like "patient's temperature is a linear function of time", which attains the boiling point after 10 days or goes below absolute zero after 20, etc. – fedja May 20 at 2:29
  • @Vorac On the other hand, when you;ll be doing a real project commissioned by some industry, you'll find out that, as Charles said, a big part of work will be to squeeze from the people who give you the numbers and tasks what they mean and what the actual assignment is. So, you'll have opportunities to practice your common sense sooner than you expect. However, I don't think that your Master's project was given with this in mind. It is not a PhD. Usually it is just "show us that you possess the skills we have taught you and have some fun in the process" – fedja May 20 at 2:34
  • Dear @fedja I have worked as a developer full-time for 6 years before attempting this Master's. As you said in your answer If your algorithm is intended for real use, the people who requested the work should be most interested in getting things right, so you should be able to ask them as many relevant questions as you want. This has been my experience in every business setting. I am not getting that here and that's why I am upset and confused. If I work with this data, my results are going to be sketchy at best. At best - as in if I do perfect job. – Vorac May 20 at 4:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.