Exploratory Analysis of Market Health Violations¶
by: Eddie Liu, Bryan Kim, Kei Hagiwara, Eddy Lara¶
1) Background Information
Introduction¶
"This dataset contains Environmental Health Inspection Results for Restaurants and Markets in the City of Los Angeles. The Los Angeles County Environmental Health is responsible for inspections and enforcement activities for all unincorporated areas and 85 of the 88 cities in the County. This dataset combines some of the fields from the County's inspection and violation data, and is filtered to include only facilities in the City of Los Angeles." https://data.lacounty.gov/Health/LOS-ANGELES-COUNTY-RESTAURANT-AND-MARKET-INSPECTIO/6ni6-h5kp
Guiding Thoughts¶
Our analysis will focus mainly on questions regarding health scores of establishments. Some guiding questions that inspired us to choose this dataset include: Is there a geographic pattern to restaurants receiving certain scores? What are the most common health scores? Is there a relationship between the Yelp rating of restaurants and the health scores? Another primary incentive that propels us to research and analyze this dataset is the notion of associating health scores with the probability of contracting a food-borne illness at restaurants. It is without much doubt that most of the general population would prefer not to contract such illnesses when going out to eat, which is why analyzing trends in health scores is important. A research article called the “Impact of Restaurant Hygiene Grade Cards on Foodborne-Disease Hospitalizations in Los Angeles County” found that there was a “significant decrease in foodborne-disease hospitalizations in Los Angeles County following the introduction of grade cards.” The authors found that the restaurant health cards were associated with a 13.1% decrease in the amount of people hospitalized with food-borne illnesses. From this, we see the importance in the role health cards have in association with the decreasing amount of food-borne illnesses. It is of much interest to us to analyze the trends in these scores in order to promote the wellbeing of the general population that frequents these establishments.
Overview of Health Inspections¶
The Los Angeles County Environmental Health acts as an enforcement agency for the Department of Health in the County of Los Angeles. Its enforcers are commonly known as “Health Inspectors” (though their official title is “Environment Health Specialist”) and these are the individuals that carry out the inspections of each establishment and assign them a letter grade. Each food inspection begins with 100 points. As a health inspector conducts an inspection, the appropriate sub-categories in the Critical Risk Factors and the Good Retail Practices Section on the Retail Food Official Inspection Report (FOIR) will be marked if appropriate. The card issued by the health inspector will be based upon the score received on the Retail Food Official Inspection Report. 100 - 90 points is an A, 89 - 80 points is a B, 79 - 70 points is a C, and 69 - 0 receive no score. A full description of the report can be found here http://publichealth.lacounty.gov/eh/docs/WhatsNew/FOIR-RefGuide.pdf. Our report will focus on those restaurants in 2018, with special emphasis on the establishments that received a B or C score as these locations may be more prone to carrying illnesses. Restaurants with a B score rating have “Generally good food handling practices and overall food facility maintenance” whereas restaurants with a C score rating have “Generally acceptable food handling practices and overall general food facility maintenance.”
According to LA Public Health: “A facility can be inspected from 1 to 3 times a year. The frequency is based on the public health risk associated with the food products served, the methods of food preparation, and the operational history of the food facility.” Thus we assume that in our dataset that the maximum number of times a restaurant is graded is 3 times per year. Each row in the dataset represents a unique instance of a violation. For example: a restaurant that has received 7 violations in an inspection will have 7 rows in the dataset, each with a unique identifier (record id).
Variable Discussion¶
1) "activity_date”. This column of our dataset represents the time and date that the restaurant was inspected. The date range for this data is from 2015 to 2018. For the purposes of our analysis, we will only look at restaurants with an activity date in 2018, and we will disregard the time as most entries have the standardized "12:00:00 AM" timestamp.
2) “facility_name”. This represents the name of the establishment that was inspected. We will be working with around 2,000 unique establishments in this dataset.
3) "violation_code". These violation codes are listed from the Los Angeles Department of Public Health website, with each code corresponding to a unique violation description.
4) "violation_description". These are the standards that all restaurants and markets are inspected to in Los Angeles. If a restaurant receives the violation as described by the violation description, it means the restaurant did not meet that standard. Ex: "# 14. Food contact surfaces: clean and sanitized" means a restaurant's food contact surfaces were not cleaned and sanitized to a satisfactory degree.
5) "points". This column represents the severity of the violation of the business. Each row represents a separate violation and those points correspond to a specific violation, regardless of whether or not the restaurant had multiple violations. The greater the point value, the more severe the violation.
6) "grade". This is the overall health grade that the restaurant ended up with at the time of the violation. Referring to the explanation above, a ranking of A represents a score of 100-90, a rating of B represents a score of 89-80, and a ranking of C represents a score of 79-70. Anything below 70 would not receive a score.
7) "facility_address". This is the address of the establishment registered with the City of Los Angeles that indicates the location of the restaurant.
8) "facility_zip". This is the zip code of the physical location of the establishment. Originally, the data included the city and state but every single entry in this particular dataset is Los Angeles, CA so these two columns were dropped from our dataset.
9) "pe_description". This is the type of establishment categorized by risk factor and size. For example, this includes whether the establishment is a restaurant, market, caterer etc. The size and risk factor are determined by the Los Angeles Department of Public Health.
10) "record_id". This represents each row in the dataset.
11) "score". This is the overall health score that the location earned at the time of the violation. The scores are calculated as follows: 100 - the sum of violation points found during the inspection. Major violations are 4 points and minor violations are 2 point deductions. A 0 point deduction doesn't constitute a deduction in the negative sense; it represents a field which the inspector marks down as a note. Major violations constitute an immediate health hazard that requires immediate correction and possible closure of the restaurant. Minor violations don’t constitute an immediate health hazard, but they do warrant necessary corrections in the future. If the inspector deems that the restaurant follows good retail practices despite the violations, the minor violation will only be a 1 point deduction. These are low risk violations that typically represent corrections to good health practices.
12) "rating". This represents the Yelp rating of the establishment. This is on a scale from 1.0 - 5.0 with 1.0 representing the worst end of the scoring spectrum and 5.0 representing the best end. This column was pulled from Yelp’s website.
13) "price_category". This is the price category of the establishment which was also pulled from Yelp’s website. A single dollar sign represents a generally cheaper location and a quadruple dollar sign represents a generally more expensive location, with a double and triple dollar sign representing the middle of the spectrum.
14) "type". This is the type of the establishment. This includes the ethnicities (ex: Chinese, Mexican, Hawaiian, etc.) or other descriptions (ex: Diner, Fast Food, Coffee, etc.), depending on the first Yelp description pulled.
15) "year". This is the year which the violation took place. We will be focusing solely on 2018 entries.
Limitations¶
As with any analysis, there are certain limitations that may inhibit inspection of certain areas. For our project, we are limited by how much the violation columns tell us about the specific violation each establishment received. We are only given the general category of the violation and not the specific violation (which can be found on http://publichealth.lacounty.gov/eh/docs/WhatsNew/FOIR-RefGuide.pdf). Another limitation is how the Yelp API does not allow us to scrape all of the establishments (we were only able to get about 2203 out of around 9000 establishments in Los Angeles). In addition to this, the "type" column extracted from Yelp only extracts the first description from a Yelp entry. Ex: El Carmen is labeled as 'Mexican' and 'Cocktail Bars' on Yelp but would only be classified as 'Mexican' under "type". Toca Madera is labeled as 'Cocktail Bars', 'Mexican', and 'Breakfast & Brunch' on Yelp but would only be classified as 'Cocktail Bars' under "type". This is due to how the establishment owner wishes to categorize their business. One owner may think that putting 'Mexican' first increases business while another owner may think the same by putting 'Cocktail Bars' first.
El Carmen: https://www.yelp.com/biz/el-carmen-los-angeles?osq=mexican+bar
Toca Madera: https://www.yelp.com/biz/toca-madera-los-angeles?osq=mexican+bar
Getting Data from Yelp's API¶
Note that the following data fetching code using the Yelp API has not been run for our submission output since it is VERY time consuming. We ran the code once in order to output a dataframe which we saved locally as a csv file. We then loaded the csv file for our analyses later on.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
violations = pd.read_csv("Restaurant_and_Market_Health_Violations.csv")
dropcols = ['serial_number','violation_status','facility_id', 'facility_city', 'facility_state',
'employee_id', 'owner_id', 'owner_name', 'program_element_pe', 'program_name',
'program_status', 'record_id', 'service_code', 'row_id']
df = violations.drop(columns=dropcols)
df["year"] = pd.to_datetime(df["activity_date"]).dt.year
df_2018 = df[df["year"] == 2018]
import requests
import json
tt = df_2018[["facility_name","facility_address"]]
yy = tt.drop_duplicates(subset='facility_name').reset_index(drop=True)
address = yy['facility_address']
names = yy['facility_name']
api_key='PYQ62khmD2Sjrfmd5ZCzI7tifXh5QhhdvjyZJB3IVdDaJWBnwyDeMFEa8xgfJv1ZsjU-pNJdLqNTKiK3yIbkFreySmVpnytNNWTa2VrcdNTAgDV-ZuapEGolJr3cXXYx'
headers = {'Authorization': 'Bearer %s' % api_key}
Here we find the IDs and addresses for as many of our businesses there are on Yelp.
url='https://api.yelp.com/v3/businesses/matches'
add = pd.Series([]) #restaurant address
rid = pd.Series([]) #restaurant id
for i in range(4000): #split the for loop into two because of Yelp API's daily fetching limit of 5000
if i % 50 == 0: #used to track progress of loop. Each loop took approximately TWO hours.
print(i)
params={'name':names[i],"address1": address[i],"address2": "","address3": "",
"city": "Los Angeles",
"state": "CA",
"country": "US"}
req=requests.get(url, params=params, headers=headers)
a = json.loads(req.text)
try:
b = a["businesses"]
rid[i] = b[0]['id'] #Get restaurant ID
add[i] = b[0]['location']['address1'] #Get restaurant address
except: #If an error occurs or the restaurant does not exist on Yelp, we label it as "Cannot Find"
rid[i] = "Cannot Find"
add[i] = "Cannot Find"
#2nd key
api_key2='xMEMOIYXVK3qxLtT_x0WbyQcah3ZP5dqmRM2cPYrFKZ0KJA5UvwDmHbJtZwJP2GS4HqlvQ63iKgUkfpvofRak9A0_lOpSUERY66rZxUfKb3dG6_5ZkcrHYJgB7_cXXYx'
headers2 = {'Authorization': 'Bearer %s' % api_key2}
#2nd loop
url='https://api.yelp.com/v3/businesses/matches'
for i in range(4000,8517):
if i % 50 == 0:
print(i)
params={'name':names[i],"address1": address[i],"address2": "","address3": "",
"city": "Los Angeles",
"state": "CA",
"country": "US"}
req=requests.get(url, params=params, headers=headers2)
a = json.loads(req.text)
try:
b = a["businesses"]
rid[i] = b[0]['id']
add[i] = b[0]['location']['address1']
except:
rid[i] = "Cannot Find"
add[i] = "Cannot Find"
url = 'https://api.yelp.com/v3/businesses/'+rid[8516] #Ethnicity always shows up as the first category
req=requests.get(url, headers=headers)
a = json.loads(req.text)
a
#3rd key
api_key='itYKMeAN8zm-Y5_bHjaPJPRgLyr5M74mrr1qO1cPwXEAn4gGYi0twN_dgCiimGc0Ni3xGUZWjA-QMkRDWsENiAnPZqYkxVFZgTNxbVA2tT0J1e-cBZB3_ATsqrXdXXYx'
headers = {'Authorization': 'Bearer %s' % api_key}
Now, using the IDs and address, we collect restaurant name, rating, price, and type from yelp's business search API
url = 'https://api.yelp.com/v3/businesses/'
restaurant = pd.Series([])
rating = pd.Series([])
price = pd.Series([])
typeofrest = pd.Series([])
for i in range(4000): #again split the loop into two
if i % 50 == 0: #used to track progress of loop. Each loop took approximately THREE hours.
print(i)
if rid[i] == "Cannot Find":
restaurant[i] = ("Cannot Find")
rating[i] = ("Cannot Find")
price[i] = ("Cannot Find")
typeofrest[i] = ("Cannot Find")
else:
url = 'https://api.yelp.com/v3/businesses/'+rid[i]
req=requests.get(url, headers=headers)
a = json.loads(req.text)
try:
b = a['location']['address1']
if add[i] == b: #Here we make sure that the business we fetch from Yelp API matches the one we collected earlier
try:
restaurant[i] = a["name"]
rating[i] = a["rating"]
price[i] = a["price"]
typeofrest[i] = a["categories"][0]["title"]
except:
restaurant[i] = ("Cannot Find")
rating[i] = ("Cannot Find")
price[i] = ("Cannot Find")
typeofrest[i] = ("Cannot Find")
else:
restaurant[i] = ("Cannot Find")
rating[i] = ("Cannot Find")
price[i] = ("Cannot Find")
typeofrest[i] = ("Cannot Find")
except:
restaurant[i] = ("Cannot Find")
rating[i] = ("Cannot Find")
price[i] = ("Cannot Find")
typeofrest[i] = ("Cannot Find")
#4th key
api_key='VistynnaqvQR5dCHJtJugLBsgMKYcUnkv0cxOXphGqZ3sgHS2QCrZokS9pWncR_zQgqefkVxjvzlz_BPKlQO7ch0Z8QypcQcIAZgxYW42QMNvdTb-ay7KnvmpGzYXXYx'
headers = {'Authorization': 'Bearer %s' % api_key}
#2nd loop
url = 'https://api.yelp.com/v3/businesses/'
for i in range(4000,8517):
if i % 50 == 0:
print(i)
if rid[i] == "Cannot Find":
restaurant[i] = ("Cannot Find")
rating[i] = ("Cannot Find")
price[i] = ("Cannot Find")
typeofrest[i] = ("Cannot Find")
else:
url = 'https://api.yelp.com/v3/businesses/'+rid[i]
req=requests.get(url, headers=headers)
a = json.loads(req.text)
try:
b = a['location']['address1']
if add[i] == b:
try:
restaurant[i] = a["name"]
rating[i] = a["rating"]
price[i] = a["price"]
typeofrest[i] = a["categories"][0]["title"]
except:
restaurant[i] = ("Cannot Find")
rating[i] = ("Cannot Find")
price[i] = ("Cannot Find")
typeofrest[i] = ("Cannot Find")
else:
restaurant[i] = ("Cannot Find")
rating[i] = ("Cannot Find")
price[i] = ("Cannot Find")
typeofrest[i] = ("Cannot Find")
except:
restaurant[i] = ("Cannot Find")
rating[i] = ("Cannot Find")
price[i] = ("Cannot Find")
typeofrest[i] = ("Cannot Find")
Output our dataframe to a csv file
pulled = pd.DataFrame(list(zip(restaurant, rating, price, typeofrest)))
pulled.index = rating.index
pulled.to_csv('yelp_all2018.csv')
First we load the original data set and verify it loaded correctly.¶
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.pyplot import figure
import seaborn as sns
from scipy import stats
from statsmodels.stats.proportion import proportions_ztest
from statsmodels.stats.proportion import proportions_chisquare
data = pd.read_csv('Restaurant_and_Market_Health_Violations.csv')
pd.set_option('display.max_columns', None)
data.head()
Next, we load the Yelp dataset and verify that it loaded cleanly.¶
yelp = pd.read_csv('yelp_all2018.csv')
yelp['0'] = yelp['0'].str.upper()
yelp = yelp.drop(columns = ['Unnamed: 0'])
yelp = yelp.rename(columns={"0": "facility_name", "1": "rating", '2': 'price_category', '3': 'type'})
yelp.head()
We merge the two datasets on the 'facility_name' column using a left join.¶
#Combine the two datasets and sort alphabetically by facility name
data1 = pd.merge(data, yelp, on='facility_name', how='left')
data1 = data1.sort_values(by = ['facility_name'])
data1.head()
Notice that rating, price_category, and type have NaN's, meaning they were not listed/were not accessible on Yelp.
Cleaning the Dataset¶
- Dropping columns
- Adding new columns
- Reindexing
- Tidying up fields
- Dropping duplicates
- Drop NaN's
- Fixing zipcodes with 4 extra numbers (e.g. 90024-xxxx)
- Variable creation
#Reindex
data1 = data1.reset_index()
#Create date column
data1['year'] = data1['activity_date'].str[6:10]
#Remove unnecessary columns
data1 = data1.drop(columns = ['index','serial_number','violation_status','facility_id', 'facility_city', 'facility_state',
'employee_id', 'owner_id', 'owner_name', 'program_element_pe', 'program_name',
'program_status', 'service_code', 'row_id','service_description'])
#Tidy year column
data1['year'] = pd.to_numeric(data1['year'])
data1['year'].dtype
#Drop duplicates
data1 = data1.drop_duplicates(subset=['facility_name','activity_date','violation_code','facility_address'])
#Standardize zipcodes
data1['facility_zip'] = data1['facility_zip'].str[0:5]
data1['facility_zip'].unique()
We will only be focusing on entires in 2018 that our yelp API could pull¶
data = data1.loc[data1['year'] == 2018]
data = data.reset_index()
data = data.drop(columns = 'index')
#Drop NaN's
data = data.dropna()
data = data.reset_index()
data = data.drop(columns = 'index')
data.shape #dimensions
We decide to drop any NaN values in order to facilitate our analyses as 19,788 entries is plenty for the purposes of this project.
#Create a new column 'major' which indicates if a row is a major violation or not (1:major and 0:minor)
data['major'] = np.where(data['points']>=4, 1, 0)
#Create a new column 'num_rating' which is simply the rating column but in type float
data['num_rating'] = [float(i) for i in list(data["rating"])]
Here is a preview of our finalized dataset.
data.head()
data.columns
data.info()
It is important to note that the "Score" column contains Health Scores extracted from the Los Angeles Department of Public Health whereas the "Rating" column contains ratings from Yelp users extracted from the Yelp website.
2) Exploratory Data Analysis
Variable Distributions¶
Let's examine the distribution of our variables as a high level overview before our in-depth analysis. These distributions represent the amount of violations by variable since each row of our dataset is a violation.
Violation Code¶
fig= plt.figure(figsize=(8,9))
data.violation_code.value_counts().plot(kind = "barh")
plt.ylabel('Violation Code')
plt.xlabel('Violations')
plt.title('Count of Violations by Violation Code')
plt.tight_layout()
plt.gca().invert_yaxis()
plt.show()
data.violation_code.value_counts().head()
The most common violation given out in 2018 is violation F044 which signifies "Floors, walls and ceiling: built, maintained and clean" on the reference guide (http://publichealth.lacounty.gov/eh/docs/WhatsNew/FOIR-RefGuide.pdf). This category is marked as out of compliance if anything about the structure of the building is unsatisfactory including but not limited to: chipped walls, missing outlets, unapproved floor boards on the ground, dirt, etc. F033 is the second most common violation and this is marked if 'nonfood contact surfaces' are unclean. This may include: dirty fan guards, food debris on shelves, and chemical buildup in the mechanical warewasher unit. F035 is the third most common violation and this category signifies anything out of the ordinary with respect to equipment and utensils. A violation can be handed out under this category for any of the following reasons: using uncertified equipment, using 'household only' wares, or using lead glazed containers. For a full list of violation reasons, please refer to the above reference link.
Points¶
data.points.value_counts().plot(kind = "bar")
plt.xticks(rotation = 0)
plt.xlabel('Violation Points')
plt.ylabel('Violations')
plt.title('Count of Violations by Violation Points')
plt.show()
data.points.value_counts()
Most violations handed out were for minor violations (indicated by a 1 point violation). A 0 point violation indicates that the health inspector took down a note of some non-negative attribute for the establishment. An 11 point violation indicates a shutdown of that establishment due to a major health concern.
Major¶
data.groupby(['major'])['record_id'].count().plot(kind = 'bar')
plt.xticks(np.arange(2), labels = ('No', 'Yes'), rotation = 0)
plt.ylabel('Violations')
plt.xlabel('Major Violation')
plt.title('Count of Violations by Major Violations')
plt.show()
data.groupby(['major'])['record_id'].count()
We notice here that most violations were minor. This is a more categorized representation of the count of violations by violation points.
Description of Establishment¶
fig= plt.figure(figsize=(12,8))
data.pe_description.value_counts().plot(kind = "barh")
plt.xticks(rotation = 0)
plt.xlabel('Violations')
plt.ylabel('Establishment Description')
plt.title('Count of Violations by Establishment Description')
plt.gca().invert_yaxis()
plt.show()
data.pe_description.value_counts().head(4)
Most violations were handed out to high risk restaurants with seating capacities of 150 or below, with more violations handed out to restaurants with lower seating capacities. This is most likely due to the high risk nature of smaller capacity restaurants with regards to their smaller infrastructure and reduced operating area.
Price Category¶
price = data.price_category.value_counts()
price_df = pd.DataFrame(list(price)).plot(kind = "bar", legend = None)
price_df.set_xticklabels(["\$$","\$","\$$\$","\$$\$\$"])
plt.xticks(rotation = 0)
plt.xlabel('Price Category')
plt.ylabel('Violations')
plt.title('Count of Violations by Price Category')
plt.show()
data.price_category.value_counts()
Most violations were handed out to establishments with a 2 dollar and 1 dollar sign on Yelp. Pricier establishments may maintain higher standards including health standards which could explain the lower amount of violations for them.
Establishment Type (count of violations)¶
fig= plt.figure(figsize=(23,8))
data.type.value_counts().plot(kind = "bar")
plt.tight_layout()
plt.ylabel('Violations')
plt.xlabel('Establishment Type')
plt.title('Count of Violations by Establishment Type')
plt.show()
data.type.value_counts().head(10).plot(kind = "barh")
plt.gca().invert_yaxis()
plt.tight_layout()
plt.xlabel('Violations')
plt.ylabel('Establishment Type')
plt.title('Count of Violations by Establishment Type')
plt.show()
Above we examine the number of violations by establishment type. Most violations handed out in 2018 were to Mexican, Chinese, Coffee & Tea, Korean, and Bakery type establishments with Mexican establishments having a noticeable top spot. Let us examine this again by normalizing this through looking at the ratio of violation count to establishment count by each establishment type.
fig= plt.figure(figsize=(29,8))
plt.tight_layout()
#number of violations
(data.groupby(['type'])['record_id'].count().sort_values(ascending = False)\
/data.groupby(['type'])['facility_name'].nunique().sort_values(ascending = False)).sort_values(ascending = False).plot(kind = 'bar')
#number of establishments
plt.ylabel('Violations/Establishments')
plt.xlabel('Establishment Type')
plt.title('Violations/Establishments by Establishment Type')
plt.show()
(data.groupby(['type'])['record_id'].count().sort_values(ascending = False)\
/data.groupby(['type'])['facility_name'].nunique().sort_values(ascending = False)).sort_values(ascending = False).head(10).plot(kind = 'barh')
plt.gca().invert_yaxis()
plt.xlabel('Violations/Establishments')
plt.ylabel('Establishment Type')
plt.title('Violations/Establishments by Establishment Type')
plt.show()
When we examine the proportion of violations to the amount of establishments by establishment type, we see a new set of rankings. Establishments categorized as "Stadiums & Arenas" rank the highest with a value of 39. This means that for every Stadium & Arena establishment, there are 39 violations associated with it. We also see "Fast Food" ranked in the top 10 as this category was not present when we only examined the amount of violations by establishment type. Mexican establishments have a value of 8.39 here which is not extreme when compared to the median. This means that although Mexican establishments have a very high number of individual violations, there is a relatively higher amount of Mexican establishments in general.
Establishment type (sum of violation points)¶
fig= plt.figure(figsize=(29,8))
plt.tight_layout()
data.groupby(['type'])['points'].sum().sort_values(ascending = False).plot(kind = 'bar')
plt.ylabel("Sum of Violation Points")
plt.xlabel("Establishment Type")
plt.title("Sum of Violation Points by Establishment Type")
plt.show()
data.groupby(['type'])['points'].sum().sort_values(ascending = True).tail(10).plot(kind = 'barh')
plt.xlabel("Sum of Violation Points")
plt.ylabel("Establishment Type")
plt.title("Sum of Violation Points by Establishment Type")
plt.show()
These are the top 10 establishment types with the highest number of violation points. Mexican, Chinese, and Coffee & Tea establishments have the highest count of violation points in 2018. The pattern here matches that of the top establishment types with the highest number of individual violations so we see a correlation between violations and violation points. Let's examine the data again if we normalize it by taking the ratio of violation points to number of establishments by each establishment type.
fig= plt.figure(figsize=(29,8))
plt.tight_layout()
#sum of violation points
(data.groupby(['type'])['points'].sum().sort_values(ascending = False)\
/data.groupby(['type'])['facility_name'].nunique().sort_values(ascending = False)).sort_values(ascending = False).plot(kind = 'bar')
#number of establishments
plt.ylabel('Violation Points/Establishments')
plt.xlabel('Establishment Type')
plt.title('Violation Points/Establishments by Establishment Type')
plt.show()
#sum of violation points
(data.groupby(['type'])['points'].sum().sort_values(ascending = False)\
/data.groupby(['type'])['facility_name'].nunique().sort_values(ascending = False)).sort_values(ascending = False).head(10).plot(kind = 'barh')
#number of establishments
plt.gca().invert_yaxis()
plt.xlabel('Violation Points/Establishments')
plt.ylabel('Establishment Type')
plt.title('Violation Points/Establishments by Establishment Type')
plt.show()
The establishments with the top proportions of sum of violation points to number of establishments generally match the establishments with the top proportions of individual violations to number of establishments. There are some changes however. Singaporean establishments ranked 5th with regards to individual violations:number of establishments but ranked 3rd in sum of violation points:number of establishments. This means that although they had relatively less individual violations, they had more severe violations which is measured by their higher sum of violation points.
It is important to note that there are some establishment types, such as Men's Clothing, Day Spas, and Home Decor (to name a few), listed that do not correlate with the selling of food/ drinks. The reason why they appear in our data is due to these establishments selling food/ drinks in one form or another, whether it be through primary or secondary means. As an example, the establishment categorized as "Home Decor" is Cost Plus World Market, which mainly sells furniture but also sells snacks and alcohol. As long as these establishments sell perishables they are subject to health inspections as well.
Score¶
data.score.hist(bins=30)
plt.xlabel('Health Score')
plt.title('Distribution of Health Score by Violations')
plt.ylabel('Violations')
plt.show()
Here we notice something interesting with the distribution of health scores by violations. There are peaks at health scores of 90 and 80. We hypothesize here that health inspectors are more likely to give these borderline scores in order to prevent the establishment from receiving the next grade down (ex: a 90 would be a borderline A and an 80 would be a borderline B). This could be a reflection of bias on the health inspectors' part.
Grade¶
grades = data.groupby(['grade'])['record_id'].count().sort_values(ascending=False)
grades.plot(kind = 'bar')
plt.title('Count of Violations by Grade')
plt.ylabel('Violations')
plt.xticks(rotation=0)
plt.xlabel('Grade')
plt.show()
We may want to see how the city of Los Angeles fairs on a holistic scale with regards to grade distributions. Here, grade A contains the most number of inspections (17713) followed by grade B (1861) and grade C (214). Thus for the year 2018, around 90% of inspections resulted in a grade A ranking which gives us some relief in choosing where to eat.
Rating¶
data.rating.value_counts().plot(kind = "bar")
plt.xticks(rotation = 0)
plt.ylabel('Violations')
plt.xlabel('Yelp Rating')
plt.title('Count of Violations by Yelp Rating')
plt.show()
data.rating.value_counts()
The majority of violations were handed out to establishments that had Yelp ratings between 3.0 and 4.5 stars. This is mainly due to how the majority of establishments fall between this range in terms of Yelp ratings.
Zipcode¶
fig= plt.figure(figsize=(25,7))
data.facility_zip.value_counts().plot(kind = "bar")
plt.ylabel('Violations')
plt.xlabel('Zip Code')
plt.title('Count of Violations by Zip Code')
plt.show()
data.groupby(['facility_zip'])['violation_code'].count().sort_values(ascending=False).head(10).plot(kind = 'bar')
plt.xticks(rotation = 30)
plt.ylabel('Count of Violations')
plt.xlabel('Zip Code')
plt.title('Count of Violations by Zip Code')
plt.show()
Right off the bat, we notice the top 10 areas (defined by zip codes) that contained the highest number of violations in 2018. 90012, 90048, 90020, 90005, and 90028 were the top 5. 90012 is the area with the highest amount of violations and we see that this is the zip code for Chinatown and Japantown. This inspired us to examine the facilities by the amount of violations received and to investigate the relationships between our other variables. Before doing so, we decide to normalize our analysis by examining the ratio of violations to number of establishments by zipcode.
fig= plt.figure(figsize=(25,7))
(data.groupby(['facility_zip'])['record_id'].count().sort_values(ascending = False)\
/data.groupby(['facility_zip'])['facility_name'].nunique().sort_values(ascending = False)).sort_values(ascending = False).plot(kind = 'bar')
plt.ylabel('Violations/Establishments')
plt.xlabel('Zip Code')
plt.title('Violations/Establishments by Zip Code')
plt.show()
(data.groupby(['facility_zip'])['record_id'].count().sort_values(ascending = False)\
/data.groupby(['facility_zip'])['facility_name'].nunique().sort_values(ascending = False)).sort_values(ascending = False).head(10).plot(kind = 'bar')
plt.ylabel('Violations/Establishments')
plt.xlabel('Zip Code')
plt.title('Violations/Establishments by Zip Code')
plt.xticks(rotation = 30)
plt.show()
The zipcode with the highest ratio of number of violations to number of establishments by zip code is 90103. The zipcode for Westwood and UCLA (90024) is listed here, which prompts us to investigate this are further below. Chinatown/Japantown is not listed in the top 10 here; the zipcode for this area is ranked 21 out of 78 unique zipcodes.
Chinatown and Japantown¶
chinatown = data.loc[data.facility_zip == '90012']
chinatown.facility_name.nunique()
97 unique establishments in Chinatown were handed health violations in 2018. Let us see which establishments received the most and least violations.
chinatown.groupby(['facility_name', 'facility_address'])['record_id'].count().sort_values(ascending=False).head(5)
chinatown.groupby(['facility_name', 'facility_address'])['record_id'].count().sort_values(ascending=False).tail(5)
table = pd.pivot_table(chinatown, values='record_id', index=['facility_name'],
columns=['grade'], aggfunc='count').fillna(0).sort_values(by = 'B', ascending = False).head(5)
table
Lucky Deli appears to be the establishment that received the most violations that resulted in a B rating. Let's examine the types of violations that were given.
facilities = "LUCKY DELI"
lucky = chinatown.loc[chinatown.facility_name == facilities]
lucky.violation_description.unique()
A full reference guide for these violations can be found on http://publichealth.lacounty.gov/eh/docs/WhatsNew/FOIR-RefGuide.pdf. Unfortunately, our dataset does not list the specific reason for the violation under each code so we cannot examine this more closely. From this we move on to the rest of Los Angeles and examine the breakdown of inspections by grade type.
Westwood¶
We remember that 90024 is the zip code encompassing the Westwood area. Therefore we decide to examine this further as many students and faculty from UCLA are likely to go to these locations.
westwood = data.loc[data.facility_zip == '90024']
westwood.facility_name.nunique()
68 unique establishments in Westwood were handed health violations in 2018. Let us see which establishments received the most and least violations.
westwood.groupby(['facility_name', 'facility_address'])['record_id'].count().sort_values(ascending=False).head(5)
westwood.groupby(['facility_name', 'facility_address'])['record_id'].count().sort_values(ascending=False).tail(5)
table = pd.pivot_table(westwood, values='record_id', index=['facility_name'],
columns=['grade'], aggfunc='count').fillna(0).sort_values(by = 'B', ascending = False).head(5)
table
We notice that there were no establishments in Westwood that were given a C grade in 2018. Therefore we examine the establishments that were given the most violations that resulted in a B rating. Interestingly, the Carl's Jr. and Kikka Sushi in Ackerman Union received the least amount of health violations in 2018 with respect to the 90024 area. These UCLA establishments manage to maintain good health practices while servicing hundreds of college students each day. We see that Barney's Beanery, a very popular bar for UCLA students, is listed as one of the establishments with the most violations in this area and decide to examine the types of violations they were given.
facilities = "BARNEY'S BEANERY"
barneys = westwood.loc[westwood.facility_name == facilities]
barneys.violation_description.unique()
A full reference guide for these violations can be found on http://publichealth.lacounty.gov/eh/docs/WhatsNew/FOIR-RefGuide.pdf. Unfortunately, our dataset does not list the specific reason for the violation under each code so we cannot examine this more closely. From this we move on to the rest of Los Angeles and examine the breakdown of inspections by establishment type.
Proportion of violations summarized by establishment type and grade¶
table = pd.pivot_table(data, values='record_id', index=['type'],
columns=['grade'], aggfunc='count').fillna(0)
total = data.groupby(['type'])['record_id'].count()
table['total'] = total
table = table.divide(total, axis=0)
table['total'] = total
table = table.round(2)
table.sort_values(by = 'total', ascending = False)
From a preliminary look, we see that Mexican, Chinese, Coffee & Tea, Korean, and Bakery establishments had the most number of individual violations in 2018. Let us examine this further below by examining the proportion of grades by establishment type.
Proportion of grades by establishment type¶
Top 20 'B' violations¶
table.sort_values(by = 'B', ascending = False).head(20)
Here we see the top 20 establishment types that had the highest proportion of violations resulting in a B grade. 100% of violations for Columbian establishments resulted in a B rating, but only there were only 11 violations for those establishments in 2018. We note that Chinese establishments had a relatively high number of violations handed out to them (1009 violations) and 20% of those resulted in a B rating. Bakeries and Korean establishments also had relatively higher amounts of violations handed out to them (869 and 937 violations respectively), each with 20% and 19% of those violations resulting in B ratings.
Top 20 'C' violations¶
table.sort_values(by = 'C', ascending = False).head(20)
Here we see the top 20 establishment types that had the highest proportion of violations resulting in a C grade. From the top, we see that Buffet type establishments had 33% of their violations result in a C grade. However, only 48 violations in total were handed out to them in total. It is interesting to note that 'Buffet' type establishments have a relatively higher proportion of violations in the 'B' and 'C' ranking range as seen in both tables. After getting a preliminary sense of the violation points and the amount of violations by establishment type, we wish to gain a better understanding of the health score by establishment type. This analysis is conducted below by examining the average health score by establishment type.
What is the average health score by establishment type?¶
data.groupby(['type'])['score'].mean().sort_values(ascending=False).head(10)
data.groupby(['type'])['score'].mean().sort_values(ascending=True).head(10)
From the above analysis, we see the top and bottom establishments by average health score. We notice again that Buffet type establishments have the lowest average health score rating and Chinese establishments appear in the bottom 10 list again as well. We have looked at individual violations, violation scores, and health score averages by establishment type. These are all important factors to keep into mind when choosing a place to go out to, but arguably one of the most important factors individuals take into account when going out is the Yelp rating of these establishments. Below we start our analysis on this highly coveted variable.
Distribution of violations and grades by rating¶
table = pd.pivot_table(data, values='record_id', index=['rating'],
columns=['grade'], aggfunc='count').fillna(0)
total = data.groupby(['rating'])['record_id'].count()
table['total'] = total
table = table.divide(total, axis=0)
table = table.drop('total', axis=1)
table = table.round(2)
total = data.groupby(['rating'])['record_id'].count()
table['total'] = total
table
We see that 1 star restaurants had the highest proportion of A rating inspections, but it also had the least amount of inspections total. Establishments between the 2 to 3 star rating zone contained a moderate amount of B rating inspections compared to other rating categories. A surprising proportion of 5 star restaurants (19%) had B rating inspections but it is also accompanied by a lower sample size. The proportion of C rating inspections remains low for all rating categories.
Distribution of violations and grades by Zipcode¶
table_zip = pd.pivot_table(data, values='record_id', index=['facility_zip'],
columns=['grade'], aggfunc='count').fillna(0)
total_zip = data.groupby(['facility_zip'])['record_id'].count()
table_zip['total_zip'] = total_zip
table_zip = table_zip.divide(total_zip, axis=0)
table_zip = table_zip.drop('total_zip', axis=1)
table_zip = table_zip.round(2)
table_zip['total_violations'] = total_zip
table_zip.sort_values("total_violations", ascending = False).head(10)
We take another look at the zipcodes with the highest number of violations except this time we break it down by grade type. 90012, the zipcode for Chinatown and Japantown, had the most number of individual violations with 86% of those resulting in an A rating and 14% of those resulting in a B rating. Interestingly, 90048, the zipcode neighboring Beverly Hills and West Hollywood had the second highest number of violations but 100% of those violations resulted in an A rating for those establishments.
Rodents?¶
One of the least desirable health violations is that involving rodents or insects. Violation code F023 encompasses this category. A full description is as follows: This subcategory is marked MAJOR if any of the following violations are observed:
- A rodent infestation as evidenced by any of the following: fresh droppings, gnaw marks, nesting materials, grease marks, live or dead rodents in the food preparation, food storage, warewashing areas, indoor customer dining area, or in the restroom. (CLOSURE)
- A cockroach infestation as evidenced by one or more live cockroaches in the food preparation, food storage, warewashing, indoor customer dining area, or restrooms. (CLOSURE)
- Fly infestation as evidenced by numerous flies in food preparation areas contacting food, food-contact surfaces or both. (CLOSURE)
- A food employee caring for or handling animal(s) that is present, such as patrol dogs, service animals, or pets, in areas that are used for food preparation, storage, or display.
As listed in the background information, a violation of 2 points is counted as a minor violation. A violation of more than 4 points is marked as major. Here we wish to see which establishments had a major violation in this category.
data.loc[(data['violation_code'] == 'F023') & (data['points'] > 4)].shape[0]
53 violations in 2018 were for a major violation for F023. Let's examine this more closely.
rodents = data.loc[(data['violation_code'] == 'F023') & (data['points'] > 4)]
rodents.groupby(['type'])['record_id'].count().sort_values(ascending=False)
We see that 6 major rodent violations were handed out to Mexican establishments in 2018. The next establishments to receive these violations were Thai, Korean, and Pizza places. Next we compile a list of these establishments
rodents.groupby(['type', 'facility_name'])['record_id'].count().sort_values(ascending=False)
3) Significant Difference between Mexican and Chinese Violations?
We wish to examine if there is a significant difference in the proportion of violations for Mexican and Chinese Restaurants in terms of A's, B's, and C's. We will use the two proportion z-test to evaluate our findings.
table = pd.pivot_table(data, values='record_id', index=['type'],
columns=['grade'], aggfunc='count').fillna(0)
total = data.groupby(['type'])['record_id'].count()
table['total_violations'] = total
establishment_count = data.groupby(['type'])['facility_name'].nunique()
table['establishment_count'] = establishment_count
table.sort_values(by = 'total_violations', ascending = False).head(2)
A - Violations¶
counts = np.array([1655, 767])
nobs = np.array([1830, 1009])
zscore, pvalue = proportions_ztest(counts, nobs, alternative = 'two-sided')
print('zscore = {:.3f}, pvalue = {:.3f}'.format(zscore, pvalue))
Our results indicate that there is a significant difference in the proportion of Mexican violations resulting in an A (90.0%) and the proportion of Chinese violations resulting in an A (76.0%). The proportion of Mexican violations resulting in an A is significantly higher.
B - Violations¶
counts = np.array([158, 197])
nobs = np.array([1830, 1009])
zscore, pvalue = proportions_ztest(counts, nobs, alternative = 'two-sided')
print('zscore = {:.3f}, pvalue = {:.3f}'.format(zscore, pvalue))
Here, our results indicate that there is a significant difference in the proportion of Mexican violations resulting in a B (8.6%) and the proportion of Chinese violations resulting in a B (19.5%). The proportion of Chinese violations resulting in a B is significantly higher.
C - Violations¶
counts = np.array([17, 45])
nobs = np.array([1830, 1009])
zscore, pvalue = proportions_ztest(counts, nobs, alternative = 'two-sided')
print('zscore = {:.3f}, pvalue = {:.3f}'.format(zscore, pvalue))
Lastly, our results indicate that there is a significant difference in the proportion of Mexican violations resulting in a C (1.0%) and the proportion of Chinese violations resulting in a C (4.5%). The proportion of Chinese violations resulting in a C is significantly higher.
It is interesting to note that there are almost 3 times as many Mexican establishments as Chinese establishments, but Chinese establishments have significantly higher proportions of B and C violations and a significantly lower proportion of A violations. Mexican establishments may have an overall higher amount of violations in general, but the majority of those violations (90%) resulted in an A grade.