## Instructions
Upload a PDF file, named with your UC Davis email ID and homework number (e.g., `xtai_hw1.pdf`), to Gradescope (accessible through Canvas). You will give the commands to answer each question in its own code block, which will also produce output that will be automatically embedded in the output file. Each answer must be supported by any written statements as well as any code used. 

All code used to produce your results must be shown in your PDF file (e.g., do not use `echo = FALSE` or `include = FALSE` as options anywhere). `qmd/Rmd` files do not need to be submitted, but may be requested by the TA and must be available when the assignment is submitted. 

Students may choose to collaborate with each other on the homework, but must clearly indicate with whom they collaborated. 

**Please assign the pages with your answers to the corresponding questions when submitting your homework on Gradescope. Points will be taken off if you fail to do so.**

# Housing stock data
The data set [here](https://xhtai.github.io/statdatasci/homework/homework3/calif_penn_2011.csv) contains information about the housing stock of California and Pennsylvania, as of 2011. Information is aggregated into "Census tracts", geographic regions of a few thousand people which are supposed to be fairly homogeneous economically and socially. Each row in the data set is a Census tract. 

## Problem 1: Loading and cleaning (25 points)

a. Load the data into a dataframe called `ca_pa`. (Hint: one way is to use `read.csv()`.)

b. How many rows and columns does the dataframe have?

c. Run this command, and explain, in words, what it does:

```{r, eval = FALSE}
colSums(is.na(ca_pa))
```

d. Remove any row containing an NA value. There are many ways to do this; one possibility is using the function `na.omit()`, which takes a dataframe and returns a new dataframe, omitting any row containing an NA value. You may also use `dplyr` operations. 

e. How many rows did (d) eliminate?


## Problem 2: This Very New House (25 points)

a. The variable `Built_2005_or_later` indicates the percentage of houses in each Census tract built since 2005. Plot `Median_house_value` against this variable (`Median_house_value` should be on the y-axis). Is there overplotting? How can you improve on this scatterplot? Produce this plot. 

b. Make a new plot, or pair of plots, which breaks the plot in (a) out by state (use your improved version of the scatterplot), for just California and Pennsylvania. Note that the state is recorded in the `STATEFP` variable, with California being state 6 and Pennsylvania state 42. What do you learn from this figure? Is there a difference between the two states?

c. What is the median percentage of houses built in 2005 or later (in the entire data set, i.e., California and Pennsylvania)? Create a new binary variable for whether the Census tract has percentage greater or less than this median. Make a visualization for the median house prices, broken down by this new variable. What do you learn from this figure? 

## Problem 3: Nobody Home (25 points)
The vacancy rate is the fraction of housing units which are not occupied. The dataframe contains columns giving the total number of housing units for each Census tract, and the number of vacant housing units.

a. Add a new column to the dataframe which contains the vacancy rate. What is mean vacancy rate? 

b. What is the standard deviation? Calculate this using just basic arithmetic operations (`+` or `sum()`, `-`, ...) and `length()`, then use the `sd()` function to make sure that you get the same result. 

## Problem 4: County Investigation (25 points)
The column `COUNTYFP` contains a numerical code for counties within each state. 

a. We are interested in San Francisco County (county 75 in California), Yolo (county 113 in California), and Allegheny County (county 3 in Pennsylvania). Create a new data frame with just the relevant rows. What is the median home value in Yolo county? 

b. For San Francisco, Yolo and Allegheny Counties, what were the average percentages of housing built since 2005?

c. What is the (Pearson) correlation coefficient between median house value and the median household income in (i) the whole data, (ii) all of California, (iii) all of Pennsylvania, (iv) San Francisco County? First make scatterplots and guess, then compute these in R. What do you learn about the relationship between median house values and median household income? 

## Appendix

```{r eval = TRUE}
sessionInfo()
```
