Moving the September 11 Digital Archive to Omeka

It’s hard for me to comprehend that the September 11 Digital Archive has been in operation for over nine years. I was brought on near the beginning of the project, charged by the Center for History and New Media to make sense of the growing amount of contributed material. As demand to preserve reactions to the attacks increased beyond our initial capacity, we began to accept CD-ROMs and Zip drives filled with heterogeneous collections. In addition, as our reputation grew, other 9/11-related projects donated their digital content, often in the form of fully functional websites.

Quickly we realized that our original database was insufficient to the task, so we designed one better suited to store a multitude of files and complex directory trees. Over the years, this database and accompanying website (designed primarily for researchers) have done their job satisfactorily. We donated a copy of the Archive to the Library of Congress, and, as contributions waned, the Center focused on building other websites and digital tools.

The Center’s work on the Archive and other collections websites greatly inspired the development of Omeka, an open source, standards-based web publishing platform designed especially for cultural heritage institutions. Our mission was to provide a generalized CMS that would benefit a wide variety of collections projects, to facilitate the ingestion, description, organization, and presentation of digital content. It’s fitting, then, that we’ve chosen Omeka as the next home of the September 11 Digital Archive.

By far our biggest job is moving a massive amount of content to Omeka. The Archive contains hundreds of collections and well over 100,000 digital objects. Each collection has particular characteristics and many have unique requirements for import. To avoid laborious and redundant work, I’m working on a framework that systematizes the import process. With this framework in place, all I need to do is evaluate the Archive’s holdings, write an import strategy for every collection or class of collections, and run a script. If something goes wrong during import, the script picks up where it last left off.

Data loss during import is unacceptable. Thankfully, mapping the Archive’s data model to Omeka is proving to be straightforward, given Omeka’s database pedigree and overall extensibility. When the need arises, I simply create element sets, item types, and plugins to accommodate data structures and behavior that are not native to Omeka. This is a testament to the success of Omeka as a generalized CMS, able to adapt to virtually any circumstance.

It continues to be a mixed blessing working on the September 11 Digital Archive. The work offers exceptional methodological and technological use cases for working with digital archives, but the material is a constant reminder of the September 11, 2001 attacks and the response to them. We’re certain that the move to Omeka will enrich the historical record by making such noteworthy material even more usable by scholars and accessible to the general public.

State of the Archive on the Eve of the 10th Anniversary

Many of us still find it difficult to believe that ten years have passed since the September 11 attacks. Every person who lost a loved one or who lived through the aftermath of the events experienced something unique. It was in the wake of 9/11, we at CHNM together with our friends at the American Social History Project (ASHP) at the City University of New York Graduate Center built the September 11 Digital Archive to preserve some of those responses to the traumatic events in the months and years that followed.

When we launched the Archive in 2002, we could not have imagined how it would grow to become the world’s largest public collection of born-digital materials related to the events of September 11, 2001. At 150,000 items, the Archive is a uniquely rich collection of the feelings, hopes, and fears of Americans and people around the globe as they attempted to process the world-changing events of that late summer morning. Collected materials range from recordings of Manhattan residents’ voice mails on that morning, to drawings of the attacks by children in Los Angeles, to photos of impromptu memorials from around the world.

We also published digital collections shared with the Archive by large organizations such as the Red Cross, the National Guard, the New York Fire Department, the Smithsonian Institution, and the Library of Congress and smaller groups serving flight attendants, the Chinese community of lower Manhattan, and Arab-American groups.

The September 11 Digital Archive is significant not only because of its contents and use, but also because of the way it was produced. At the time it launched, the Archive was a unique example of how the Internet could be used to collect history from ordinary people, and it has served as a model for countless other digital archives projects. In 2001 most individuals were not regular users of the Web, nor were they as comfortable sharing personal materials online as they are today.

As we approach the 10th anniversary, we will direct our efforts towards preservation. A Saving America’s Treasures (SAT) grant, jointly-administered by the National Park Service and National Endowment for the Humanities, will help pay for our preservation efforts as we transfer the aging collection to the Omeka platform, a more stable and standardized archival system. This is an essential step to making the contents of the Archive more accessible to scholars, students, policy makers, and the general public in the coming years.

We will use this blog to update you on our progress and detail some of the work required to transfer a large digital collection using one data model to another system with a different one. We also plan to highlight some of the collections and items that have intrigued us as we sort through the Archive.

Finally, we are re-opening the collecting portal and want to hear how your life has changed since September 11, 2001. By collecting reflections at this commemorative moment, we hope to further the life of the Archive as one that not only includes the most immediate reactions to the attacks, but also shows change over time as individuals reflect at different points in the post-9/11 world.

When asked what did you do to commemorate the 10th anniversary of September 11th, we hope that includes submitting a quick reflection to the September 11 Digital Archive.