Update, 4:38 PM CST, August 5, 2014:
This afternoon the FCC published this blog post: FCC Makes Open Internet Comments More Accessible to Public. Snip:
Because of the sheer number of comments and the great public interest in what they say, Chairman Wheeler has asked the FCC IT team to make the comments available to the public today in a series of six XML files, totaling over 1.4 GB of data.
The addresses in the files released by the FCC today are accurate to the zip code level. If you are looking for exact addresses of the subset of comments entered by users directly into the Electronic Comment Filing System, the information below may be of use to you.
* * *
tl; dr: below is a bulk download of all Net Neutrality comments published to the FCC’s Electronic Comment Filing System as single file, along with the complete text of those comments. There are 475,280 comments available as of August 4, 2014. More comments are being published and will be available here as time goes by.
- https://www.google.com/fusiontables/DataSource?docid=1KNueemGVwbYsVsErQLrgTAvKliuGo-3y4e5wSYt6 (~50MB): this is a single file with a row for every comment. With this file, one can find the original comment on the FCC website and also refer to the PDF containing the content of the comment itself
- https://drive.google.com/file/d/0BzehoDP_J3kCbUU1cTdEUWhhZlk/edit?usp=sharing (~1.3GB): this is the compressed file containing all of the PDFs referenced in the comment spreadsheet
Scott Robbin and I are working with some friends to extract the content of the PDF files and integrate all of the data into one usable file, and perhaps a service. More to come on that.
We wanted to make this stuff available now so that it could be useful to others and so that we could dive further into what it means to “comment” during an FCC rulemaking process. It is a fascinating study in civic engagement and is far more complex that it first seems.
A quick visualization
Our main goal with this bulk download (and, hopefully, all of the remaining 600,000 comments we haven’t seen yet) is to stimulate visualizations, charts, and other analyses around this important content.
But because maps and lists are nice, here’s a quick, initial look at comments filed, by state:
Starting with the question
The Federal Communications Commission is considering new rules around Protecting and Promoting the Open Internet in the context of the Communications Act of 1934. They start with a fundamental question:
What is the right public policy to ensure that the Internet remains open?
Background on the rulemaking action
These actions were prompted by a January 14, 2014 decision in Verizon v. FCC, where the United States Court of Appeals for the District of Columbia Circuit vacated and remanded parts of the Commission’s December 2010 Open Internet rules (“In the Matter of Preserving the Open Internet Broadband Industry Practices“). FCC Chairman Tom Wheeler also wrote a blog post on the topic on the day of the decision, “Ensuring an Open Internet Now and for the Future“.
On February 19, 2014, the FCC created a “new docket”— 14-28— “within which to consider how the Commission should proceed in light of the court’s guidance in the Verizon v. FCC opinion”. The Commissioners also published a series of statements, a fact sheet, and a explainer, all of which highlighted the importance of public comment. On April 24, 2014, the FCC established a new inbox for Open Internet comments at a openinternet@fcc.gov. In an April 29, 2014 blog post, “Finding the Best Path Forward to Protect the Open Internet” Chairman Wheeler cued up the context for the comments:
We are asking for comment on a proposed a course of action that could result in an enforceable rule rather than continuing the debate over our legal authority that has so far produced nothing of permanence for the Internet.
Even before the filing of the proceeding, interest in the topic was high. In a May 7, 2014 blog post, “Preserving An Ever-Free and Open Internet “, FCC Commissioner Mignon Clyburn wrote that “over 100,000 Americans have spoken” and “during the past few weeks, tens of thousands of consumers, companies, entrepreneurs, investors, schools, educators, healthcare providers and others have reached out to ask me to keep the Internet free and open”. Under the long-established “notice and comment” rulemaking process at the FCC, the Commission set up a page for this proceeding, chronologically numbered 14-28, in the Electronic Comment Filing System on the FCC Web site directly after the filing was made on May 15, 2014.
The comment period
The volume of comments was immense. In a July 14, 2014 blog post, “Keeping Track of the Open Internet Comments Submitted to the FCC“, FCC Chief Information Officer Dr. David A. Bray provided some data about the number and rate of submissions. In a July 16, 2014 blog post, Chairman Wheeler wrote of the need to modernize IT systems at the FCC, citing “the antiquated IT capabilities of the agency that have not been able to handle the surge of comments” in proceeding 14-28 via their Electronic Comment Filing System (ECFS). On July 18, 2014, Gigi Sohn, Special Counsel for External Affairs, Office of the Chairman, Federal Communications Commission tweeted about the number of comments:
Specifically, as of noon, 1,030,000 #NetNeutrality comments have been filed at @FCC. — Gigi Sohn (@GigiBSohnFCC) July 17, 2014
Scott and I were interested in these comments because we think that the people who have raised their voices about an Open Internet are exactly the kind of people we want to hang out with. This list of people can constitute a grass-roots, natural social network, one we’d like to be in.
More about the data
The final tally of “comments”, at the close of the initial comment period (11:59 p.m., Friday, July 18) was 1,067,779. All of the data in this bulk download file was taken directly from the Protecting and Promoting the Open Internet detail page of the Electronic Comment Filing System maintained by the FCC. 446,843 were filed through this system by the close of the initial comment period of the Open Internet proceeding at 11:59 p.m., Friday, July 18. According to a statement by the FCC sent to reporters, “the comment and reply deadlines serve to get public input to the FCC in a timely and organized way to provide more time for analysis. However, comments are permitted in this proceeding any time up until a week before a vote is scheduled at an Open Meeting (the “Sunshine” period under the Sunshine in Government Act).” The system is still open to new comments and the number of comments continue to rise on this system. This dataset contains no records of the 620,936 comments filed through the Open Internet inbox set up by the FCC in April 2014.
Method of obtaining data
Scott Robbin created a scraper to obtain the data. Here it is:
Explanation of data
The bulk download is in the form of a spreadsheet with nine fields.
- Filing ID, which is a unique number used to keep track of each filing. When prepended with the string “http://apps.fcc.gov/ecfs/comment/view?id=”, it will display the full record (http://apps.fcc.gov/ecfs/comment/view?id=6017840634)
- Name, which is first and last name in one field
- PDF ID, which is a unique number used to keep track of each PDF that is associated with each filing. For instance, on the record referenced above (http://apps.fcc.gov/ecfs/comment/view?id=6017840634), you can see that the PDF which contains the text of the comment starts with “7521332360.txt”. In this instance, “7521332360” is the PDF ID
- Filing type (“comment” is the most common, but there are others). The ECFS has a pulldown menu for “filing type”, with comment pre-chosen. More details on filing type, below
- Exparte?, which is a a field that indicates if the filing was exparte or not. At the FCC, “ex parte” describes a communication directed to the merits or outcome of a proceeding that, if written, is not served on all of the parties to the proceeding and, if oral, is made without giving all the parties to the proceeding advance notice and an opportunity to be present. “Yes” means that it is exparte, “no” means it is not. More on exparte, below
- Date received, which is the date that the filing was made, in day/month/year format
- Date posted, which is the date that the filing was published to the Electronic Comment Filing System, in day/month/year format
- Address 1, which is street address. Includes directional, street number, street name, and street type in one field
- Address 2, additional information (if applicable)
- Address 3, which is city, state, zip in one field
A note about filing types
When people talk and the huge interest in this matter being considered by the FCC, they talk in terms of “comments”. However, comment is just one of the many types of filings offered in the FCC’s Electronic Comment Filing System:
In the data we have available in this bulk download, vast majority of the filing types are comments. There are a number of stray filing types (Testimony, Submission for the Record, Statement, and so on) that are basically substitutes for comments.
There are some instances, like with the “Opposition” filing type, where the user wanted to make more clear their statement.
There is a special filing type with documents that have been filed using the ECFS: “Notice of exparte“. Here’s the way they explain it:
At the FCC, “ex parte” describes a communication directed to the merits or outcome of a proceeding that, if written, is not served on all of the parties to the proceeding and, if oral, is made without giving all the parties to the proceeding advance notice and an opportunity to be present. Our ex parte rules play an important role in protecting the fairness of the FCC’s proceedings by assuring that FCC decisions are not influenced by impermissible off-the-record communications between decision-makers and others. At the same time, the rules are designed to ensure that the FCC has sufficient flexibility to obtain the information necessary for making expert decisions.
Examples of entities which filed items under this type are the City of San Antonio, Texas, Microsoft Corporation, Cogent Communications Group, Inc., Verizon. Here’s a complete list of all ex-parte filings, along with links to the actual filings. I obtained this through a simple search on the FCC’s comment filing search interface.
Another type of possible filing is a petition, Here’s their definition:
A person outside of the FCC files a Petition to suggest new rules or changes to existing rules. Unless directed otherwise in the FCC’s Public Notice seeking comment on the petition, the public has 30 days from the date of the Public Notice to submit comments on whether the FCC should grant or deny the petition.
The only true petition out of the hundreds of thousands of entries in this data is Mozilla’s “Petition to Recognize Remote Delivery Services in Terminating Access Networks and Classify Such Services as Telecommunications Services Under Title II of the Communications Act”.
The most famous example of this type of filing is covered in the book, Petition Against God. Here’s a summary:
“In 1974, two broadcasters filed a petition with the Federal Communications Commission, asking that there be a “freeze” on applications for noncommercial religious radio and television stations. Within six months of the filing, more than 4,000,000 letters of protest had been sent to the FCC. Over the years, an estimated 50,000,000 letters dealing with this petition have clogged up the government mail rooms.”
The power of the response to this tongue-in-cheek, satirical petition (which was never seriously put forward as a matter of policy) is a seminal example of the power of grassroots outrage. It presaged the rise of Ronald Reagan and the effectiveness of unity and discipline within the conservative movement through the 1980s and beyond. There are many lessons here. This was 4,000,000 *letters*— actual letters delivered by the U.S. Postal Service.
History of the FCC Electronic Comment Filing System
Lastly, we’re interested in the cultural, political, and technological histories of the FCC’s Electronic Comment Filing System. Again, it’s easy to make fun of its dated interface, its lack of raw data, and the fact that comments are published to individual PDFs. According to an October 14, 2009 press release about version 2.0 of their , the FCC made some significant upgrades to the system:
With the 2.0 upgrade, ECFS will including many new features, including fully Section 508 compliance; the ability for users to file multiple documents to multiple rulemakings in a single submission; advanced search and query of rulemakings; ability to extract comments; RSS (Really Simple Syndication) feeds; and the ability to export data results to Excel or PDF formats.
I’m currently working on a deeper look into this. If you have any insights into how this tool was created, please let me know. More to come!
More
If you are going to do some work with this data, we’d love to hear from you! Hit us up on Twitter here: @srobbin and here: @danxoneil.