CITY OF NEW YORK, NY
Mayor Rudolph Giuliani

"Project Scorecard Evaluates Cleanliness of New York's Streets"

Since 1973, the City of New York has used a litter measurement tool called "Project Scorecard" to evaluate and measure the cleanliness of its streets and sidewalks. Trained evaluation teams use the Scorecard to "rate" the degree of surface litter by comparing actual street conditions to photographic standards. This measurement tool provides objective information about street and sidewalk cleanliness. Project Scorecard is managed by the Mayor's Office of Operations because they are responsible for tracking and monitoring the City's cleanliness over time.

Purpose of Project Scorecard

Scorecard issues semi-monthly and monthly reports to the Department of Sanitation and releases periodical information to the public. Each month's performance is compared to that of the prior month and to that of the same month last year. A trend report that includes data for 13 months for each command is also produced monthly. The data contained in these reports serves several different purposes:

The Mayor's Office of Operations monitors Scorecard to ascertain if street cleaning resources are being utilized to maximum potential. This reports also influence the Mayor's Office of Operations to support or deny requests for additional resources to provide street cleaning services.

Scorecard provides Sanitation Department managers with information to help them make optimum use of limited resources by identifying areas where increased concentration of City services will have the greatest impact.

The Department of Sanitation uses the reports to evaluate citywide operations. Scorecard's database is analyzed to answer policy planning questions, such as assessing the effectiveness of different types of cleaning and collection strategies. Community and public interest groups obtain the reports to learn about cleanliness conditions in their local neighborhoods. By publicizing the results to these groups, it promotes an increased and better informed citizen participation in the effort to clean up New York City.

Business Improvement Districts (BIDS) use the report to evaluate the conditions of neighborhood shopping and central business districts.

Project Scorecard's Litter Measurement Methods

Project Scorecard is an objective measurement system developed by the Urban Institute in Washington, D.C., and modified for New York by the Fund for the City of New York in cooperation with the Department of Sanitation. The Scorecard ratings were determined by studying the public's perception and expectations of street and sidewalk cleanliness.

To develop objective ratings, seven photographs depicting increasingly dirty streets are assigned a numerical value ranging from 1.0 (cleanest), 1.2, 1.5, 1.8.2.0, 2.5, and 3.0. When a street is under construction the raters give the street a 5.0 unrated score. Actual street conditions are then compared with photographic standards and assigned a numerical rating. Ratings below 1.5 are considered "acceptable" according to both the consensus arrived at in public surveys conducted by Scorecard and the Department's own operating standards.

The Scorecard figures are based on a fixed sample of streets in each Sanitation section and district. The sample of streets is statistically representative because the average ratings for sample streets are equivalent to average ratings for all streets in a section. The sample of streets is geographically representative because the samples are distributed evenly through all parts of a section. District and zone figures are weighted by street mileage. Sections with more mileage along the streets have more weight in determining overall district, zone, borough, and citywide figures.

Scorecard analysis is presented in terms of the percentage of streets in each neighborhood rated "acceptable." This unit of measurement has proved to be more readily understandable for evaluators and more useful for comparing the relative cleanliness of neighborhoods than absolute numerical ratings. This is because average ratings, such as the 1.29 citywide average rating for July 1999, have a small numeric range and are in a form that Scorecard's wide audience of community groups, City government managers, and oversight organizations may not readily grasp. By contrast, the meaning of the percent of acceptably clean streets is immediately apparent, and this figure has a wide range - ratings for individual neighborhoods have been reported from zero to one hundred percent. The latest citywide rating of acceptably clean streets was 89.3 percent. For the purposes of internal program evaluation, including reliability analysis of inspector performance, the Department of Sanitation uses the average rating because of its greater stability as a statistic. However, in either case, current cleanliness conditions can be compared with earlier reports of the same area to determine the trend over time.

Determination of Routes

All inspections are conducted Monday through Friday between 9:30 a.m. and 4:00 p.m. The daily route for each rater/driver inspection team is assigned by the operations supervisor, who considers a variety of factors in developing the schedule for daily inspection routes. For purposes of statistical reliability, the operations supervisor must base route determination on the following variations when assigning teams:

Rotation of inspection teams for each district

Variation in time of day the ratings are conducted in each district

Rotation of day of the week the ratings are conducted in each district

Rotation of the week during the month the ratings are scheduled in each district

Each day Scorecard sends out three teams to rate 20 to 25 of the City's 230 Sanitation sections.

Frequency

The frequency of inspections varies according to the degree of variation in cleanliness levels. In areas where the level is fairly consistent, Scorecard makes fewer inspection trips than in sanitation sections with great variation in the level of cleanliness. Each of the 230 sanitation sections is rated from one to three times a month.

Streets and sidewalks are rated separately. When a street is rated, it is divided into two to four segments depending on the length of the street. Each segment is rated and the average of those ratings is the street's rating. On average, about five percent of the blockfaces in a district are rated. The average district sample is approximately 120 blockfaces, and the Citywide total sample size is 6,659 blockfaces.

Training

A special program teaches raters to disregard secondary objects (parked cars, garbage pails, trees, leaves, people) on a blockface and to evaluate the area exclusively in terms of loose litter and surface dirt. A system of internal controls has been devised to assure consistency and reliability in the application of the Scorecard rating scale. The program collects daily data on mileage and rating time for each inspection team, and a variety of quality control procedures identify inconsistent raters who need further training. Inspections are continuously monitored to detect potentially biased ratings. Managers conduct daily checks of ratings against standards, make field checks, and direct monthly all-team cross checks of the same route. During these cross checks, the Department of Sanitation may determine that some ratings are in error. These ratings are dropped and the streets are evaluated again, if necessary.

Data Processing

Scorecard raters record their numerical ratings on pre-programmed laptop computers, which are then returned to the office so that data can be uploaded to the City's mainframe computer. Data is stored on a monthly and daily level in a hierarchical database, permitting generation of regular reports and extraction of data on an ad hoc basis for special analyses.

Results

Scorecard's monthly reports show the historical trend of the average cleanliness rating and the percent of all streets and sidewalks rated acceptably clean. Although minor variations and changes from month to month can be influenced by a variety of factors, accurate comparisons can generally be made by comparing the most recent monthly ratings to the same month of the previous year. The Department of Sanitation also calculates the quarterly averages for the percent of streets and sidewalks rated acceptably clean and compares district ratings to other districts in the Sanitation zone.

Measuring the amount of litter on neighborhood streets and sidewalks is important both for evaluating effective use of limited city resources and developing targeted programs for neighborhoods that need additional cleaning services and support. Project Scorecard has effectively assisted the City of New York in improving their services and keeping neighborhoods clean. The dramatically improved percentage of 'acceptably' clean districts in the City during the past 22 years prove its effectiveness and success:

District 1976 1998

Manhattan 50.5% 77.4%

Bronx 61.4% 80.9%

Brooklyn 58.7% 80.1%

Queens 83.6% 87.5%

Staten Island 90% 94.9%

CITYWIDE 72.5% 85%

The cleanliness of New York City is not determined by the performance of the Department of Sanitation alone. Many factors contribute to the cleanliness of streets in neighborhoods. Weather conditions and the collective behavior of pedestrians, motorists, homeowners, tenants, landlords, building superintendents, shopkeepers, commercial establishments and private carters are all major factors affecting street and sidewalk cleanliness. Scorecard is designed to measure actual litter conditions that are the product of the millions of "sanitation transactions" that take place every day. It is not a simple measurement of only the Department of Sanitation's efforts. However, by analyzing the ratings taken in each of the City's 230 sanitation sections every month, Scorecard is able to provide extensive information that can be useful to the public.

For more information regarding Scorecard contact:

Anthony Longo, Associate Director

New York City Mayor's Office of Operations

100 Church Street, 20th Floor

New York, NY 10007

Phone: (212) 788-1677

E-mail: tlongo@cityhall.ci.nyc.ny.us

For general street cleaning information contact:

Rocco Sabatella, Chief

City of New York, Cleaning Operations

125 Worth Street, Room 805

New York, NY 10013

Phone: (212) 788-4064

Fax: (212) 788-3739



Home Search jdelong@usmayors.org

The United States Conference of Mayors

J. Thomas Cochran, Executive Director
1620 Eye Street, NW, Washington, DC 20006
Telephone (202) 293-7330, FAX (202) 293-2352

Copyright 2000, U.S. Conference of Mayors, All rights reserved.