Data Scientist's Toolbox

Video Lectures

Week 3

Types of Questions

Types of analysis

• descriptive: describe the data, like census
• exploratory: find relationships (not causation)
• inferential: use a small sample to say something about a bigger population
• predictive: use data on some object to predict values for another object (still not causation)
• causal: find out what happens t one variable when you change another variable (very difficult)
• mechanistic: understand the exact changes in variables that lead to changes in other variables (very difficult, next to impossible unless system is nearly deterministic)

What is data

Quantative/qualitative.

Data is the second most important thing. The most important thing is the question.

What about Big Data

Doesn't really address this, also, I disagree with his approach. I argue that "Big Data" isn't actually determined by size, but structure. If your data-processing architecture is a automated, dynamic network, you're dealing with big data. Otherwise, you're not.

Experimental Design

Confounding: hidden variable that effects two observed variables. Look at A→B, what is actually happening is C→B and C. Causes spurious correlation.

• Sensitivity: Pr(positive result | positive)
• Specificity: Pr(negative result | negative)
• Positive Predictive Value: Pr(positive | positive result)
• Negative Predictive Value: Pr(negative | negative result)
• Accuracy: Pr(correct outcome)