Exploring Mathematics and Data Science with Technology

(a)

Unfortunately, the datascience library doesn't provide a great out-of-the-box mechanism for computing \(r\text{.}\) Fortunately, we can add two features to our datascience tables by copy-pasting the snippet given in 5.1.4 into a Code cell. (This will need to be done in every notebook that uses these features.)

def with_correlated_columns(self,num_rows=100,x_range=5,slope=1,noise=1):
    from random import random
    table = self.select().with_columns("x",[],"y",[])
    for _ in range(num_rows):
        seed = random()
        x = seed*x_range
        y = seed*x_range*slope
        y = y + (-1+2*random())*noise
        table = table.with_row([x,y])
    return table
def correlation(self,col1_name,col2_name):
    import statistics
    # get column arrays
    col1 = self.column(col1_name)
    col2 = self.column(col2_name)
    # standardize units
    col1_s = (col1 - statistics.mean(col1))/statistics.stdev(col1)
    col2_s = (col2 - statistics.mean(col2))/statistics.stdev(col2)
    # correlation is the mean product of standard units
    return statistics.mean(col1_s*col2_s)
setattr(Table,"with_correlated_columns",with_correlated_columns)
setattr(Table,"correlation",correlation)

While we won't go into detail on how correlation is measured, it is related to the measures of center and spread studied in the previous chapter. Inspect the code shown in 5.1.4; what measure of center and what measure of spread appear to be used to compute correlation?

(b)

When 5.1.4 is run successfully, no output will be shown. However, you should now be able to create a table by running ctable = Table().with_correlated_columns(). (Note that this table is generated somewhat randomly, so re-running that line will create a slightly different table.)

Show ten rows and a scatter plot for this ctable.

(c)

Based on the definition of the correlation coefficient \(r\) and the illustration given in 5.1.3, would you expect the correlation of these pairs of data to be closer to \(-1\text{,}\) \(0\text{,}\) or \(1\text{?}\)

(d)

Run ctable.correlation("x","y") to confirm your intuition.

(e)

Show a scatter plot for ctable2 = Table().with_correlated_columns(slope=3). Guess a value for r based on the scatter plot, and then confirm your guess using technology.

(f)

Show a scatter plot for ctable3 = Table().with_correlated_columns(x_range=10,slope=-2). Guess a value for r based on the scatter plot, and then confirm your guess using technology.

(g)

Show a scatter plot for ctable4 = Table().with_correlated_columns(noise=5,slope=2). Guess a value for r based on the scatter plot, and then confirm your guess using technology.

(h)

Show a scatter plot for ctable5 = Table().with_correlated_columns(noise=100,slope=-4). Guess a value for r based on the scatter plot, and then confirm your guess using technology.

(a)

Display a scatter plot visualizing the table generated by the code given in 5.1.5.

cos_table = Table(["x","y"])
for _ in range(200):
    from random import random
    from math import cos
    seed = random()
    cos_table = cos_table.with_row([seed,cos(12.6*seed)])

(b)

Based on the definition of the correlation coefficient \(r\) and the illustration given in 5.1.3, would you expect the correlation of these pairs of data to be closer to \(-1\text{,}\) \(0\text{,}\) or \(1\text{?}\)

(c)

Confirm your guess using technology.

(d)

Regardless of the value of \(r\text{,}\) give one or more reasons based on the scatter plot for why it'd be fair to say that these data points are “related”.

In general, why might data be related even when \(r\) is close to zero?

(a)

Get a copy of testing_table from 4.3 available for use in this notebook, and use code cells to produce the scatter plots shown in 5.1.1 and 5.1.2.

Don't forget to use import matplotlib, %matplotlib inline, and .where("column_name",are.above(0)) appropriately.

(b)

Describe your intuition for why SAT math scores, college math placement exam scores, and high school sizes may or may not be correlated with each other based upon the scatter plots.

(c)

Test your intuition by using technology to compute appropriate correlation coefficients. Explain why those values confirm or contradict your intuition.

(d)

Is it true to say that SAT math scores are perfect predictors of college math placement exam scores? Give reasons why or why not based upon the scatter plot, the value of \(r\) found, and your intuition.