摘要：溫習統計學的知識為更深層次的學習做準備在的演講中說就是我們理解但不知道另外的是如何的我在臺下想對于那可以理解的我好像都只懂了參考標準高效的流程課程用的是我不想再學一門類似的語言了我會找出相對應的和的來源流程什么是干凈的一個變

溫習統計學的知識, 為更深層次的學習做準備
Andrew Ng 在 2015 GTC 的演講中說, deep learning 就是 black magic; 我們理解50%, 但不知道另外的50%是如何work的. 我在臺下想, 對于那可以理解的50%, 我好像都只懂了5%.

參考"標準高效"的流程
mine: emacs org mode + emacs magit + bitbucket + python. There must be some room for improvement.

How

課程用的是R. 我不想再學一門類似的語言了, 我會找出相對應的numpy 和 scipy solution.

Getting and Cleaning Data

Website APIs

Databases

Json

Raw texts

Raw data --> Processing scripts --> tidy data (often ignored in the classes but really important)

Record the meta data

Record the recipes

--> data analysis (covered in machine learning classes)

--> data communication

Each variable you measure should be in one column, 一個變量占一列.

There should be one table for each "kind" of variable, generally data should be save in one file per table 為什么呢? 管理起來不會麻煩麼?

If you have multiple tables, they should include a column in the table thta allows them to be linked. 參見 dataframe.merge dataframe.join in pandas

代碼簿? (⊙o⊙)…

Info about the variables (including units!)
單位很重要! 沒有單位的測量是沒有物理意義的!
但測量時候必須要考慮的有效位數在課程中卻沒有提及. 大抵是因為python 和 R 對于有效位數handle地很好? 不需要像C 里邊一樣考慮 float 或者 double? 某些極端情況下也會需要像sympy這樣的library吧.

Info about the summary choice you made

Info about the experimental study design you used

代碼簿的作用類似于wet lab中的實驗記錄本. 很慶幸很早就知道了emacs 的 org mode, 用在這里很適合. 但是 Info about the variables 的重要性被我忽略了.

如果feature的數量很多, 而且feature本身意義深刻, 就需要仔細挑選. 記得一次聽報告, 有家金融公司用decision tree 做portfolio, 算法本身稀松平常, 但是對于具體用了哪些feature, lecturer守口如瓶.

"There are many stages to the design and analysis of a successful study. The last of these steps is the calculation of an inferential statistic such as a P value, and the application of a "decision rule" to it (for example, P < 0.05). In practice, decisions that are made earlier in data analysis have a much greater impact on results — from experimental design to batch effects, lack of adjustment for confounding factors, or simple measurement error. Arbitrary levels of statistical significance can be achieved by changing the ways in which data are cleaned, summarized or modelled."

Leek, Jeffrey T., and Roger D. Peng. "Statistics: P values are just the tip of the iceberg." Nature 520.7549 (2015): 612-612.

我通常都是直接用wget, 但是那樣就不容易整合到腳本中. 幾個很可能會在download時候用到的python function:

# set up the env
os.path.dirname(os.path.realpath(__file__))
os.getcwd()
os.path.join()
os.chdir()
os.path.exists()
os.makedirs()

# dowload
urllib.request.urlretrieve()
urllib.request.urlopen()

# to tag your downloaded files
datetime.timezone()
datetime.datetime.now()

# an example
import shutil
import ssl
import urllib.request as ur

def download(myurl):
    """
    download to the current directory
    """
    fn = myurl.split("/")[-1]
    context = ssl._create_unverified_context()
    with ur.urlopen(myurl, context=context) as response, open(fn, "wb") as out_file:
        shutil.copyfileobj(response, out_file)

    return fn

pandas.read_csv()

Here is a very good introduction

Below are my summaries:

python 標準庫中自帶了xml.etree.ElementTree用來解析xml. 其中, ElementTree 表示整個XML文件, Element表示一個node.

The first element in every XML document is called the root element. 一個XML文件只能又一個root, 因此以下的不符合xml規范:

recursively 遍歷

# an excersice 
# find all elements with zipcode equals 21231
xml_fn = download("https://d396qusza40orc.cloudfront.net/getdata%2Fdata%2Frestaurants.xml")
tree = ET.parse(xml_fn)
for child in tree.iter():
    if child.tag == "zipcode" and child.text == "21231":
        print(child)

JSON stands for Javascript Object Notation

lightweight data storage

JSON 的格式肉眼看起來就像是nested python dict. python 自帶的json的用法類似pickle.

Python makes a distinction between matching and searching. Matching looks only at the start of the target string, whereas searching looks for the pattern anywhere in the target.

Always use raw strings for regx.

Character sets
sth like r"[A-Za-z_]" would match an underscore or any uppercase or lowercase ASCII letter.

Characters that have special meanings in other regular expression contexts do not have special meanings within square brackets. The only character with a special meaning inside square brackets is a ^, and then only if it is the first character after the left (open- ing) bracket.

import pandas as pd
df = pd.DataFrame
# Look at a bit of the data
df.head()
df.tail()

# summary
df.describe()
df.quantile()

# cov and corr
# DataFrame’s corr and cov methods return a full correlation or covariance matrix as a DataFrame, respectively

# to calcuate pairwise correlation between a DataFrame"s columns or rows
dset.corrwith(dset[""])

# you can write your own analsis function and apply it to the dataframe, for example:
f = lambda x: x.max() - x.min()
df.apply(f, axis=1)

df.dropna()
df.fillna(0)
# to modify inplace
_ = df.fillna(0, inplace=True)

# fill the nan with the mean
# 或者用naive bayesian的prediction
data.fillna(data.mean())

Principles of Analytic Graphics

Show comparisons
If you build a model that can do some predictions, please come along with the performance of random guess.

Show causality, mechanism, explanation, systematic structure

Show multivariate data
The world is inherently multivariate

Integration of evidence

Describe and document the evidence with appropriate labels, scales, sources, etc.

Simple Summaries of Data

Two dimensions

scatterplots

smooth scatterplots

> 2 dimensions

Overlayed/multiple 2-D plots; coplots

Use color, size, shape to add dimensions

Spinning plots

Actual 3-D plots (not very useful)

Graphics File Devices

pdf: usefule for line-type graphics, resizes well, not efficient if a plot has many objects/points

svg: XML-based scalable vector graphics; supports animation and interactivity, potentially useful for web-based plots

png: bitmapped format, good for line drawings or images with solid colors, uses lossless compression, most web browers can read this format natively, does not resize well

jpeg: good for photographs or natural scenes, uses lossy compression, does not resize well

tiff: bitmapped format, supports lossless compression

rnorm:generate random Normal variates with a given mean and standard deviation

dnorm: evaluate the Normal probability density (with a given mean/SD) at a point (or vector of points)

pnorm: evaluate the cumulative distribution function for a Normal distribution

d for density

r for random number generation

p for cumulative distribution

q for quantile function

Setting the random number seed with set.seed ensures reproducibility

> set.seed(1)
> rnorm(5)