Data Science Notes: Difference between revisions
Jump to navigation
Jump to search
Adamcolton (talk | contribs) |
Adamcolton (talk | contribs) |
||
| Line 39: | Line 39: | ||
* schema-on-read | * schema-on-read | ||
* user defined functions that don't suck | * user defined functions that don't suck | ||
The Abstractions of DS | |||
* Matricies and linear algebra | |||
* relations and relational algebra | |||
* objects and methods | |||
* files and scripts | |||
* data frames and functions | |||
Current practices don't scale (I'm skipping a lot here because I know the arguments). | |||
=== The Course : 2 === | === The Course : 2 === | ||
Revision as of 16:24, 1 May 2013
https://class.coursera.org/datasci-001/class/index
Introduction
Context
Bill Howe venn diagram. "Danger Zone" is hacking + expertise without statistics.
Mike Driscolls 3 DS skills
- Statistics
- Data Munging
- Visualization
Dimensions
DS is not (or not quite)
- Business Intelligence
- Statistics
- Data(base) Management
- Visualization
- Machine Learning
If you're a → You need to learn
- DBA → unstructured data
- statistician → data that exceeds memory
- software engineer → statistical modeling
- business analyst → algorithms and tradeoffs at scale
The Continuums
- Tools vs Abstractions
- Desktop vs cloud
- hackers vs analysts
- structures vs statistics
The Course : 1
http://static.usenix.org/event/osdi04/tech/full_papers/dean/dean.pdf
Hadoop was an early MapReduce system that did away with old database standards, but in the 4 years after it's release, most SQL standards were brought back into the fold. The abstractions are important not just the tools.
MapReduce contributed the following the SQL
- Fault tolerance
- schema-on-read
- user defined functions that don't suck
The Abstractions of DS
- Matricies and linear algebra
- relations and relational algebra
- objects and methods
- files and scripts
- data frames and functions
Current practices don't scale (I'm skipping a lot here because I know the arguments).