Data Science Notes

https://class.coursera.org/datasci-001/class/index

Introduction

Context

Bill Howe venn diagram. "Danger Zone" is hacking + expertise without statistics.

Mike Driscolls 3 DS skills

• Statistics
• Data Munging
• Visualization

Dimensions

DS is not (or not quite)

• Business Intelligence
• Statistics
• Data(base) Management
• Visualization
• Machine Learning

If you're a → You need to learn

• DBA → unstructured data
• statistician → data that exceeds memory
• software engineer → statistical modeling
• business analyst → algorithms and tradeoffs at scale

The Continuums

• Tools vs Abstractions
• Desktop vs cloud
• hackers vs analysts
• structures vs statistics

The Course : 1

http://static.usenix.org/event/osdi04/tech/full_papers/dean/dean.pdf

Hadoop was an early MapReduce system that did away with old database standards, but in the 4 years after it's release, most SQL standards were brought back into the fold. The abstractions are important not just the tools.

MapReduce contributed the following the SQL

• Fault tolerance
• schema-on-read
• user defined functions that don't suck

The Abstractions of DS

• Matricies and linear algebra
• relations and relational algebra
• objects and methods
• files and scripts
• data frames and functions

Current practices don't scale (I'm skipping a lot here because I know the arguments).

The Course : 2

Actually working with data is hard. The real world problems - "handling data" - is the hard part, 90% of the problem.

Working in 'database' software get parallelism and scalability for free.