North American Congress for Conservation Biology

Transcript for Public Comments Support Public Lands: Humans, Form Letters & Bots Agree on National Monuments

Title Slide: Photo of the Authors standing on a prairie in Iowa

CAITLIN: I’m Caitlin McDonough MacKenzie. This research was a group project with the Class of 2017 Smith Fellows along with Sara Kuebbing from the class of 2016, and Mike Dombeck the executive director of the Smith Fellowship program. Our paper was just accepted in Conservation Letters so here’s a sneak peek!

Slide 1: A letter to the Department of the Interior sits next to a cartoon pencil and a cartoon envelope addressed to Dept. of Interior. Below there is a photo of five of the coauthors raising their hands in celebration.

The Letter text: Dear Secretary Zinke, Thank you for the opportunity to comment on the proposed “Review of Certain National Monuments Established Since 1996”. We are a group of David H. Smith conservation fellows; as conservation scientists we value the protection of national landscapes for historic, cultural, scientific, and conservation value, and we submit this comment to assist you in the review process. We believe that all 27 National Monuments currently proposed for review were appropriately designated or expanded in accordance to the Antiquities Act of 1906, are important to our work as scientists, and should remain at their current size and designation as National Monuments. Briefly, we provide evidence and support that the National Monuments proposed for review meet the following three of seven criteria under consideration: (criteria i) the requirements and original objectives of the Act, including the Act’s requirement that reservations of land not exceed “the smallest area compatible with the proper care and management of the objects to be protected”; (criteria ii) whether designated lands are appropriately classified under the Act as “historic landmarks, historic and prehistoric structures, [or] other objects of historic or scientific interest”; and (criteria iv) the effects of a designation on the use and enjoyment of non-Federal lands within or beyond Monument boundaries.

CAITLIN: In June 2017 twenty Smith Fellows submitted a seven-page comment on the proposed “Review of Certain National Monuments Established Since 1996” to regulations.gov, and then patted ourselves on the back for being such good applied conservation researchers, confident that our well-researched public comment would be placed on the top of the pile and imminently inform federal conservation policy…

Slide 2: a photo of Secretary Zinke with a speech bubble that reads, “Comments received were overwhelming in favor of maintaining existing monuments and demonstrated a well-orchestrated national campaign organized by multiple organizations.” Below the speech bubble there’s a photo of the New York Times headline ‘Trump Slashes Size of Bears Ears and Grand Staircase Monuments.’

CAITLIN: Instead, the Department of Interior’s summary of public comments dismissed all of the public sentiment in support of National Monuments as a well-orchestrated campaign and two monuments were significantly downsized.

Slide 3: a panoramic photo of lakes and mountains in Katahdin Woods and Waters National Monument from the national monument’s flickr feed. The words in the three questions flash over this photo in white text.

CAITLIN: We asked ourselves: What was the real public sentiment towards National Monuments — not in form letters, but in individual comments? And did this align with our letter and our work in conservation? Have form letter campaigns overshadowed individual comments?

Slide 4: coauthor Tony Chang sits in a canoe with a speech bubble floating over his head. As Caitlin narrates, the speech bubble fills in with the words “Yeah, we can totally do that!”

CAITLIN: We realized that to address the gaps in the DOI’s summary of public comments, we would have to analyze the database of comments collected by regulations.gov. Tony, our resident machine learning expert said, “yeah, we can totally do that!”

Slide 5: Two clip art computer monitors sit next to each other under the title “754,707 public comments.” On the each monitor, there is the text of a public comment. Left: “I support Bears Ears and all the other lands designated as national monuments. They have been set aside for future generations and for all species to use as nature intended. Please continue to protect them from short-sighted exploitation to line the pockets of just a few individuals. They belong to all of us and to our children and their children. We are truly idiots if we destroy them for short term profit.” Right: “I fully support this review, and furthermore support overturning the establishment of all these lands that were taken from the American public and placed under the restricted designation ‘National Monuments’.” As Caitlin narrates, the phrases around “support” in each comment are highlighted.

CAITLIN: While other groups have attempted to report the public sentiment in the public comments, we were determined to automate an analysis of the entire database of comments submitted during the open comment period, and precisely differentiate between comments that used “positive” language to voice support for National Monuments and those that used “positive” language to voice support for the review of National Monuments.

Slide 6: A black and white photo of a group of Smith Fellows coding public comment sentiment together late at night in a conference room. Photo by Smith Fellow Sean Anderson.

CAITLIN: We hand-coded over 10,000 comments (our artisanal training dataset) and categorized these into ‘support’ or ‘oppose’ the review of national monuments Then we used machine learning and natural language processing to “read” and classify the other 740,000 comments.

Slide 7: A screenshot of the Wikipedia entry for “Natural Language Processing”, which reads, “Natural language processing is a subfield of linguistics, computer science, information engineering, and artificial intelligence concerned with the interactions between computers and human (natural) languages, in particular how to program computers to process and analyze large amounts of natural language data.” Above this, the acronyms of our Natural Language Processing model, ULMFiT and AWD-LSTM appear as Caitlin names them.

CAITLIN: For our natural language processing model we used a bleeding edge Recurrent Neural Network approach. If you are into the nitty-gritty acronyms, we used ULMFiT method on an AWD-LSTM model to encode sentiment within the National Monuments public comment database.

Slide 8: There are three line-drawn tablet screens. The first is labeled ‘Language Model Pre-training’ and the screen is the Wikipedia entry for “Natural Language Processing”. The second is labeled ‘Language Model Fine-tuning’ and the screen is the text of the Smith Fellows’ public comment from Slide 1. The third is labeled ‘Classifier model training’ and the screen has two columns (‘Support’ and ‘Oppose’) with tick marks underneath.

CAITLIN First, you give the model 28,595 Wikipedia articles to read and this is how it learns English. Then, you give it the artisanal selection of hand-coded comments, and this is how it learns English in the context of public comments. It starts making connections between words and phrases, and notes the subtleties of word order. It’s almost like practicing a word association game with a robot.

Slide 9: A photo of a yellow tree against a red rock canyon from Grand Staircase Escalanta National Monument (flickr user dru!). White text reads “Across all public comments: 94.9% comments opposed the review, 2.9% supported the review, 2.2% unknown/neutral sentiment.” The numbers are added in handwriting as Caitlin narrates.

CAITLIN: We found 94.9% of comments opposed the review, 2.9% of comments supported the review, and 2.2% of comments were classified as unknown or neutral sentiment.

Slide 10: A line drawing of a person is labeled ‘Human’, a line drawing of two people sharing letters is labeled ‘Form Letters’, and a line drawing of a robot face is labeled, ‘Bot.’ As Caitlin narrates, more robot faces appear. Then the text: “Among humans, form letters, and bots, sentiment was overwhelmingly opposed to the review” appears with ‘opposed’ underlined in yellow. The percentages of comments opposed to the review in each category appears in yellow handwriting under the line drawings — humans: 97.4%, form letters: 96.4%, bots: 99.6%.

CAITLIN: Human comments were defined by their complete uniqueness from other comments. 20% of the comments were from humans. Form letter comments were collections of very similar comments that contained small differences from one another; typically because of the addition of a submitter’s name or a custom sentence. 11% of comments were form letters. Bot comments were complete duplicates of comment text. 69% of all comments submitted were complete duplicates. Some of these were submitted tens of thousands of times! Among humans, form letters, and bots, sentiment was overwhelmingly opposed to the review

Slide 11: On the right, the text: “I am appalled that our treasured national monuments are up for review at all. Every single one of our parks, monuments and cultural or historic sites is worthwhile and belongs as a part of the American story. I am adamantly opposed to any effort to eliminate or diminish protections for national monuments, and I urge you to support our public lands and waters and recommend that our current national monuments remain protected. An attempt to attack one monument by rolling back protections would be an attack on them all. National monuments have already been shown to be tremendous economic drivers. The $887 billion outdoor recreation economy and businesses in gateway communities rely on the permanency of these protections when making decisions about investing in these communities. This is a vital economic driver for our rural communities that cannot be ignored or hamstrung. Our national monuments should remain protected for future generations to enjoy - they are a gift that belongs to all. Please make sure you side with the people who support national parks, monuments, historical and cultural sites.” On the left, the logos for the League of Conservation Voters (highlighted in green) and the Sierra Club (highlighted in blue), at the bottom the text: “This comment was submitted 34,276 times!”. As Caitlin narrates, the sentences pulled from each form letter are highlighted in the text in the respective colors.

CAITLIN: This comment included text from the League of Conservation Voters and the Sierra Club form letters. It was submitted 34,276 times. Automatic software bots developed to mimic human participation in online activities are a growing manipulation used to disrupt the public comment process. For example, during the Federal Communications Commission’s net neutrality comment period in 2017, researchers determined that 94% of comments were submitted by bots!

Slide 12: A photo of snow-covered mountain slopes in alpenglow from the Cascade-Siskiyou National Monument flickr account. In white text: “In Conclusion…We find serious flaws in the DOI Summary Response. We strongly recommend that agencies employ bot deterring & bot detecting technology and report on bot activity in their summary response to public comments.”

CAITLIN (from inset video): The official DOI summary response to National Monuments public comments obscured the overwhelming negative sentiment by labeling all comments that opposed the review of National Monuments as “a well-orchestrated national campaign”. In contrast, our analysis demonstrates that unique human comments were also overwhelmingly opposed to the review. Additionally, the DOI failed to mention the potential disruptive use of bots in the comments which were super obvious to detect in our analysis.

Slide 13: A photo of the five coauthors and Smith Fellows Program Director, Shonda Foster. Shonda, Sara, and Mike (in full color) are obviously photoshopped into a black and white photo of the Class of 2017 Fellow. In white text: “Thank you! Shonda Foster, Our artisanal comment coders, our friendly reviewers, our peer reviewers, Mike Anderson, Phil Hanceford, Susan Jewell.”

CAITLIN: Thank you to the Smith Fellow Conservation Research Fellowship program and everyone who contributed to this work.

POST-CREDITS PRODUCTION LOGO/VANITY CARD: ‘Mara’s Mom Productions’ written over a black and white shot of Caitlin reading her lines under a tree.

OFF-SCREEN CHILD: Mom! Mommy!

CAITLIN: Significant eco-[breaking] ecological impacts. Oh my! [Laughing]