311 Blocked Bike Lane Requests Exploratory Analysis; Updated 12/24/24
#
By Abu Nayeem (anayeem1@gmail.com); published 12/6/24
#
# [Sustainabu GitHub Link](https://github.com/sustainabu/OpenDataNYC)
# # Table of contents
# * [Purpose](#d0)
# * [About the Data](#d1)
# * [Descriptive Statistics](#d2)
# * [Borough Service Overview](#d2a)
# * [Borough Community Board Overview](#d3)
# * [Top Frequent Requested Areas](#d4)
# * [Temporal Analysis](#d5)
# * [NYPD Response Breakdown](#d6)
# * [NYPD Response Breakdown by Borough](#d7)
# * [NYPD Response Time Breakdown (Unbinned)](#d8)
# * [NYPD Response Time Breakdown (binned)](#d9)
# * [NYPD Response Time Breakdown by Borough (binned)](#d10)
# * [NYPD Response Time Breakdown by Resolution (binned)](#d11)
# * [Geo-Spatial Analysis](#d13)
# * [NYPD InAction Rate Density Map (Interactive)](#d14)
# * [NYPD Median Response Density Map (Interactive)](#d14a)
# * [NYPD Response Bubble Map (Interactive)](#d15)
# * [NYPD Median Response Time Bubble Map (Interactive)](#d16)
# * [311 Blocked Bike Lane Data Dashboard](#d17)
# * [Conclusion](#d18)
# * [Next Steps](#d19)
# * [Programmer Notes](#d20)
# * [Run Code](#d2)
# # Purpose
#
# I’m a Queens native, and a cyclist. I’m greatly appreciative of the existing bike infrastructure in Queens, but the frequency of obstruction and lack of enforcement within bike lanes are frustrating and life-threatening. It should not be the case that cyclists are risking their life when riding their bike even with dedicated lanes. NYC has a 311 reporting system to report violations, which the NYPD enforces.
#
# However, recent studies have shown the NYPD are doing an inadequate job with [only 2% of service requests receiving summons compared to 16% of other similar complaints](https://nyc.streetsblog.org/2023/04/06/nypd-tickets-fewer-than-2-of-blocked-bike-lane-complaints-analysis). In 2021, Streetblogs did [an investigative report](https://nyc.streetsblog.org/2021/10/21/ignored-dismissed-how-the-nypd-neglects-311-complaints-about-driver-misconduct) highlighting the sharp increase of NYPD unrealistic response time of less than five minutes as shown in the graph below. Finally, in 2024, there was [a surveillance study](https://nyc.streetsblog.org/2024/10/29/study-exposes-nypds-systemic-failure-to-enforce-safety-related-parking-violations) using AI to empirically verify the lack of police enforcement, and false reporting in the service 311 records.
#
# 
#
# This exploratory report reviews 311 service request data for Blocked Bike Lane violations within the last two years (starting January, 1st, 2023) to provide baseline insight, highlight inequities, and provide the tools for community members to take action. I’ve created an **[INTERACTIVE data dashboard](https://sustainabu.shinyapps.io/blockedbikelanedashboard/)** to monitor progress and hold local enforcement. I provide some recommendations at the end to address this issue, including my interest to build a civic-tech app that crowds-source validation to increase transparency and hold NYPD accountable.
#
#
# **Why does adequate enforcement of blocked bike lanes incidents matter?**
# * Unlike vehicle-drivers, cyclists can face severe bodily harm if there is a vehicle collison. Obstruction can force cyclists to veer into a vehicle lane. Vehicle drivers may not be paying attention and/or willful negligence.
# * Unsafe bike trips lead to less ridership
# * Lack of enforcement encourage violators
# * Misallocations of public funds [i.e. invest in infrastructure without basic safety
#
# **How does NYPD prioritize blocked bike lanes requests?**
# * These are low-priority requests, but it's still required for NYPD to review them. Within a police precinct, the increased frequency of higher-priority crimes and traffic violations may lead to slower response times.
# * Bike lane obstruction is expected to be fleeting. A prolonged response reduced liklihood of incidc data.
# # About the Dataset:
#
# The dataset, [311 Service Requests from 2010 to Present](https://data.cityofnewyork.us/Social-Services/311-Service-Requests-from-2010-to-Present/erm2-nwe9/about_data) was obtained from NYC Open Data Portal. The report covers data from January 1st, 2023 to December 6th, 2024. Each record is a 311 blocked bike lane service request, which is handled by the NYPD. The NYPD is expected to investigate the request, and can give the violator a warning, summon (i.e. ticket), or take no-action. Once a resolution is made, the request is closed and registered in 311. The **police response time** is the time elapsed between the opening & closing of the request.
#
# Here is the **list of resolution, frequency, and classification** for the last two years:
#
# 
#
#
# **Is there variation of biking infrastructure in NYC?**
# * There is extensive and varying biking infrastructure within Manhattan, Brooklyn, and parts of Bronx, and Queens. You can view the [NYC bike map](https://www.nyc.gov/html/dot/downloads/pdf/nyc-bike-map-2024.pdf).
# * For this report, I did not classify/ evaluate the impact of different cycling infrastructure.
#
# **Is the data reliable?**
# * Only reported service requests are shown in the dataset. There can be many **unreported** blocked bike lane violations. Several reasons include: inconvenience, lack of time, intimidation from violators, and infractions may be committed by NYPD.
# * There is **no external validation** that the NYPD physically investigated the request. However, they are required to close the request.
# * It is possible that there may be "double-counting" where multiple service requests can be made for the same violation.
# * The 311 records do not link to the NYPD police precinct that handled service. A geo-validation method must be used to retrieve this information.
#
# **How were the 311 service requests made?**
# * About 60% of request were made online, 38% via phone app, and 2% over the phone.
#
# **Data Prep Decisions:**
# * The data is updated daily, and I excluded any "open" status service requests and requests not geo-located to any borough or community board.
# * The NYPD response was broken down to "MinutesElapsed" and placed into bins.
#
# # Descriptive Statistics
# ## Borough Service Overview
#
# 
# ## NYC Borough Community Board (CB) Overview
#
# * For Manhattan, service requests are distributed more evenly. CB3 has the most requests (19%) followed by CB10 (14%).
# * For Brooklyn, service requests are distributed unevenly. CB2 has the most requests (42%) followed by CB6 (24%).
# In[2]:
SideImage('Screenshots/F2Man.png','Screenshots/F3Brook.png')
# * For Queens, service requests are distributed more evenly. CB3 has the most requests (19%) followed by CB10 (14%).
# * For Bronx, CB5 covers 52% of all service requests with the remainder distributed more evenly.
# In[3]:
SideImage('Screenshots/F4Queens.png','Screenshots/F5Bronx.png')
# ## Top Frequent Requested Areas
# * This suggests vigilance of neighbors or cyclists reporting in that area.
#
# 
#
# ## Temporal Analysis
#
# * Compared to previous year, there is less service requests except during the summer months
# * There is seasonal variance with a drop during the summer months. I speculate that it could be related to school not being in session, and less vigilance
#
# 
#
# * There are nearly twice more service requests during the weekday than the weekend. Wednesday has the most service requests.
# * Service requests peak around rush hour times: 8AM, 2PM, and 6PM.
# In[4]:
SideImage('Screenshots/F7DayofWeek.png','Screenshots/F8Hour.png')
# # NYPD Response Breakdown of Service Requests
#
# * As explained in the data section, NYPD responses were categorized into [Miss, Pass, Action, and Summon]
# * About 30% of service requests led to enforcement
#
# 
# ## NYPD Response Breakdown by Borough
#
# * For Bronx, about 22% of service requests led to enforcement though increased number of summons
# * Both Brooklyn and Manhattan follows the average
# * For Queens, about 40% of service requests led to enforcement
#
# 
# # NYPD Response Time Breakdown (Unbinned)
#
# * Note: Response times considers once the entire interaction with the violator is completed. If the violator is present, the response time should be longer.
#
# 
# ## NYPD Response Time Breakdown (binned)
#
# * Nearly 2/3 of all service requests are resolved within 1 hour.
# * At the extremes, 5% of service requests are resolved within 5 minutes, while 3% of service requests are resolved after 6 hours
#
# 
# ## NYPD Response Time Breakdown by Borough (binned)
#
# * For Bronx, 57% of responses took at least over 1 hour with 11% taking over 6 hours.
# * For Queens, 42% of response took at least over 1 hour
# * Manhattan has the most favorable response times with only 21% of response took at least over 1 hour
#
# 
# ## NYPD Response Time Breakdown by Resolution (binned)
#
# * When summons were given, over 40% of requests were within 1 to 6 hour response time. I predict that it's likely the result of multiple persons filling a service request for the same violator. In addition, the process of writing a summons and discourse with violator can increase response time
# * When NYPD took "Action" against the violator, they had faster response times even considering extra time for dialogue.
# * When NYPD "Miss" the violators, 40% of the time they investigated at least one hour or longer
# * When the NYPD took "No-Action", 7% of service requests were completed in 5 minutes or less. This seems suspicious
#
# 
# # Geo-Spatial Analysis
#
# ## NYPD InAction Rate Density Map by Police Precinct (Interactive)
#
# The **inaction rate** is simply the rate that NYPD took no action (i.e. Miss or No-Action). HOVER for more details including hotspot.
#
#
# ## NYPD Median Response Time Map by Police Precinct (Interactive)
#
# * HOVER for more details including on hotspot.
#
#
#
# ## NYPD Blocked Bike Lane Service Response Bubble Map (Interactive)
#
# * **Note:** The color indicates increased frequency, and the scale was balanced to increase visibility
# * **Scale:** Green: 5 or less; Orange: 5 to 25; Purple: 25+
# * ON Click: NYPD Response
# In[398]:
Bub_CB()
# ## NYPD Median Response TIME Bubble Map (Interactive)
#
# * **Legend:** The color indicates median response time. Green: =< 30 mins; Orange: 30 to 60 mins; Red: > 60 mins
# * **Scale:** All locations must have at least six entries.
# * **ON Click:** Median Response and Response Distribution
# * Some areas of Brooklyn, Queens, and Bronx are deeply concerning where there is greater frequency of violations, and slow police response time
# In[397]:
Bub_Med()
# ## 311 Blocked Bike Lane Dashboard
#
# With the dashboard you can select date range, select community board, and monitor existing police responses. Please open with web browser. [CLICK HERE for DashBoard](https://sustainabu.shinyapps.io/blockedbikelanedashboard/)
#
# 
#
# # Summary/ Conclusion
#
# From the data analysis, there is reason for citizens to be concerned about NYPD enforcement of blocked bike lane requests.
# * The NYPD enforces only 30% of service requests.
# * For resolutions where NYPD took no action, 7% of requests were responded in 5 minutes or less, varying greatly from the average. This is suspicious.
# * When NYPD issues a verifiable summons, only 1% of requests were responded in 5 minutes or less, and often responded later
# * When NYPD "misses' the violator, their response time is considerably slower than all actions
# * Comparing boroughs, NYPD response time for Manhattan is better across the board. For the Bronx, and Queens, the response times are very slow.
# * From the maps, there are hotspots within a neighborhood of sub-par NYPD response that communities can focus on.
#
#
# **How do we measure improved progress of NYPD enforcement?**
#
# The most obvious metric is reducing NYPD response time. This goal is easy to achieve if they don't actually review the violation. They can falsely review requests earlier and provide a "No-Action" or "Miss" resolution, to reduce their recorded response time. This coincides with previous investigative reporting. The data corroborates foul behavior by the stark difference of response times where summons (confirmable resolution) were given.
#
# **Here are three recommendation strategies:**
# 1) **Administrative:** New administrative leadership prioritizing cutting down poor practices. This could create short-term change, but can revert to the previous state. This is not sustainable.
# 2) **Geo-validation:** Since traffic violations are location-specific, it could be required for the responding personnel to geo-validate when confirming their investigation. This process assures "ghost" enforcements are not made, but cannot confirm if proper enforcement was taken.
# 3) **Crowd-Source Validation:** This process involves citizens (i.e. crowd) to verify law enforcement. Citizens would file a 311 request, and enter data independently through an app. From the discrepancy between the 311 request and crowd-source data, citizens can hold local law enforcement accountable. I'm interested in creating a civic-tech app. If you are a developer, biking/ transit advocate, let’s connect.
#
# # Next Steps/ Opportunities
# * Partner with local community organizations for an intentional campaign to improve biking violation enforcement. You can measure real-time progress with community dashboard
# * A dedicated team to build a civic tech app and launch in appropiate community
# * Data Request: If anyone has a geo-validation method to link geo-coordinates to NYPD police precincts, this will greatly expand precision, and analysis.
# * Contact me: anayeem1@gmail.com
# # Programmer Notes
#
# ### 12/14/24 Update
# * Able to access police precincts. Police precincts and community board boundaries are nearly 1 to 1
# * Created a midpoint algoritm to set zoom at optimal location. Manual dataframes is no longer needed
# * For adjustable bubble maps, I removed the creation of columns by pivot table, because crash of missing data. Instead used dummy count.
# * Within shiny app, I added as much customization without breaking the app. There was no simple solution adjusting the column row.
# * Within shiny app, I removed three pieces where info seemed bare. Added a summary table, history line chart, and only median response time graph
# * Another piece could be added about distribution of resolutions but nothing seemed intutive
# * Chat GPT has been a huge asset in making adjustments
# * Updated Flourish graphs, and points; included many
# * The QA testing seems pretty decent; I have not tested in very scarce communities
# * A visual freshup can be helpful. I don't think ChatGPT can help, and it seems good enough for basic functionality
#
# ### 12/19/24 Update
# * Report Edits: Added "InAction" Rate Map, and removed the intensive geo-mapping code. Flourish seemed better.
# * I tried to use StreamLit as alternative python dashboard. It was quite bad as it cannot handle data pulls, and it was hard to set up
# * I did an overhaul for community dashboard making it more mobile-friendly. A few notes:
# * A lot of edits for reducing space so title are displayed for graphs. Corrections were made to remove menu coming up when click
# * The menu format was finally selected to reduce information overload on screen
# * SHINY is designed primarily for Desktop, not Mobile
# * Chat GPT hallucinated frequently
# * I try to build the app from custom, and given up as the web development is confusing because there are two packages SHINY, and SHINY
# * Chat GPT- It hallucinates to provide a solutions and then hallucinate again to come up with alternative
# * Things that cannot be adjusted: Horizontal Scroll-bar and expanding output for folium map
# # Run Code
# In[3]:
import pandas as pd
import numpy as np
import warnings
warnings.filterwarnings('ignore')
import json
from urllib.request import urlopen
from IPython.display import HTML
from IPython.display import display
import requests # library to handle requests
import folium
import seaborn as sns
from plotly import graph_objects as go
import plotly.express as px
import geopandas
import branca
#Placing Side by side Image
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
from matplotlib import rcParams
get_ipython().run_line_magic('matplotlib', 'inline')
def SideImage(Img1, Img2):
rcParams['figure.figsize'] = 11 ,8
img_A = mpimg.imread(Img1)
img_B = mpimg.imread(Img2)
# display images fig,
fig, ax = plt.subplots(1,2)
ax[0].imshow(img_A)
ax[0].axis('off')
ax[1].imshow(img_B)
ax[1].axis('off')
return ax
#regular coding
#1.Read local files
#dataframe
df = pd.read_csv("dfc_out.csv")
df.cboard=df.cboard.astype(int)
print(df.shape)
def Bub_CB(CB="All"):
if CB=="All":
B=df
zo= 11.25
default_location = [40.7128, -74.0060]
else:
B= df[(df['cboard'] == CB)]
zo=13
# Find Midpoint of Latitude and Longitude of all points
LaMax=Z.latitude.max()
LaMin=Z.latitude.min()
Latitude=(LaMax+LaMin)/2
LoMax=Z.longitude.max()
LoMin=Z.longitude.min()
Longitude=(LoMax+LoMin)/2
# Use default location for "All"
if CB == "All":
map_location = default_location
else:
map_location = [Latitude,Longitude]
# generate a new map
FG_map = folium.Map(location=map_location, zoom_start=zo,tiles="CartoDB positron") #tiles="OpenStreetMap")
Index =['incident_address','longitude','latitude', "index_",'Miss','No-Action','Action','Summon']
B17=B[Index].groupby(['incident_address','longitude','latitude']).sum().reset_index()
# Less than 5 count
BA=B17.query('index_<=5')
for index, row in BA.iterrows():
popup_text = "Address: {} Total: {} Response_Miss: {} Response_No-Action: {} Response_Action: {} Response_Summons: {}"
popup_text = popup_text.format(row['incident_address'],row['index_'], row["Miss"],row['No-Action'], row['Action'], row['Summon'])
folium.CircleMarker(location=(row["latitude"],row["longitude"]),
radius=row['index_'] +2,
color="#007849",
popup=popup_text,
fill=True).add_to(FG_map)
# Between 5 and 25 count
BB=B17.query('index_<=25 and index_>5')
for index, row in BB.iterrows():
popup_text = "Address: {} Total: {} Response_Miss: {} Response_No-Action: {} Response_Action: {} Response_Summons: {}"
popup_text = popup_text.format(row['incident_address'],row['index_'], row["Miss"],row['No-Action'], row['Action'], row['Summon'])
folium.CircleMarker(location=(row["latitude"],row["longitude"]),
radius=row['index_']/5 +7,
color="#E37222",
popup=popup_text,
fill=True).add_to(FG_map)
# Greater than 25 count
BC=B17.query('index_>25')
for index, row in BC.iterrows():
popup_text = "Address: {} Total: {} Response_Miss: {} Response_No-Action: {} Response_Action: {} Response_Summons: {}"
popup_text = popup_text.format(row['incident_address'],row['index_'], row["Miss"],row['No-Action'], row['Action'], row['Summon'])
folium.CircleMarker(location=(row["latitude"],row["longitude"]),
radius=row['index_']/15 +12,
color="#800080",
popup=popup_text,
fill=True).add_to(FG_map)
title_html = f"""
