07-F-0258 Data Mining Iraqi Documents 12-22-2005

DEFENSE INTELLIGENCE A:GENCY WASHINGTON, D.C. 20346- , INFO MEMO P: ~I U-0820/DR 22 December 20. 05 FOR: SECRETARY OF DEFENSE THRU: DEPUTY UNDER SECRET WARFIGHTING SUPP T ENSE FOR INTELLIGENCE & JAN 0 9 2006 (b)(3):10 usc §424 SUBJECT: Data Mining Iraqi Doc?ments This memo is in response to your direction to develop a proposal to ensure captured Iraqi documents arc properly data mined. There are two parts to DIA's data mining proposal. The first is making the information in the records digitally searchable. The second part calls for enabling access to the data by creating an archive at an academic institution. This memo also address precedent for implementing this proposal. Creating Digitally Searchable Records The current U.S. government data base, called HARMONY, contains nearly 700,000 documents with more than 7,000 of these being media files captured in Iraq. Of the 700,000 documents, fewer than 47,000 records have been translated. The records in the HARMONY database are images of the original Arabic documents; they are not text documents. These images are stored in HARMONY in portable document format (PDF), with an attached text synopsis. Due to limitations of the PDF software the documents themselves cannot be digitally searched; key word searches can only be performed on the synopsis which is a text document. DIA 's proposal to remedy this problem is a process which can be implemented within a few months using existing technology. The process begins with a conversion of the images to text using Arabic typists to transcribe the PDF files to Arabic text files. The transcribed Arabic text documents can then be digitally searched for terms of interest, or commercial machine translation programs can be used to produce rough English translations. The rough Enesh translations can be used to Prioritize dlcuments for full translation. DIA and the I I are prepared to develop a proof of concept to determine if this transcription process is feasible and if output meets expectations. (b)(3):10 usc §424 OSD 00425-06 J Enabling Access to Academia and the U.S. Government The va1ue of properly data mining captured materials is demonstrated by the Iraqi Perspectives Project (IPP), a post-Operation IRAQI FREEDOM (OIF) study commissioned by the U.S. Joint Forces Command's Joint Center for Operational Analysis & Lessons Learned (JOAC-LL) and created by the Institute for Defense Analyses (IDA) (attachment 1). As a baseline for its research, IDA data mined HARMONY for document, audio, and video files then used the baseline research to conduct a series of interviews with key Iraqi regime and military leaders. The result was a presentation ofOIF events from the adversary's perspective. The IDA subsequently developed a proposal, Publishing Key Documents and Transcripts from the Regime ofSaddam Hussein (attachment 2), to allow broader access to the records currently in the HARMONY database. This proposal calls for the U.S. government to establish a sponsored activity, chartered by an interagency board at an academic institution, to open the records to subject matter experts in academia and the U.S. government The IDA proposal is a valid "way ahead" to enhance data mining of the materials captured in Iraq and to expand the pool of experts and disciplines that have access to the materials. These experts can then provide expanded analysis of the captured materials. DIA endorses IDA's proposal which includes provisions to ensure the Intelligence Community has reviewed documents before release to an academic institution. Precedents for Establishing Archives Precedent for housing a digital archive at an academic institution was set after Operation DESERT STORM. During the March 1991 Kurdish uprising in Iraq, Kurdish opposition groups captured large quantities of Iraqi documents. The Non-Governmental Organization, Human Rights Watch, assisted in transferring these documents to the U.S. where the Senate Foreign Relations Committee took charge of them. Senate staffers contacted U.S. Departments of State and Defense for assistance in archiving the documents. DIA document exploitation personnel, with technical assistance from the National Archives and Records Administration, created a digital database of the original documents. Human Rights Watch first examined these documents, which were subsequently given, in digital fonnat, to the Iraq Research and Documentation Project (IRDP). The resulting database was turned over to Harvard University and can be viewed at http://www.fas.harvard.edu/-irdp. The original documents were returned to the Patriotic Union ofKurdistan (PUL) and the Kurdistan Democratic Party (KDP). The Iraqi Memory Foundation (IMF) serves as a second example of precedent. Deputy Secretary of Defense Wolfowitz was personally involved with the receipt and digitizing of the records obtained in Iraq by the I.MF. His staff led the effort to transfer the documents and funding for their digitization to the!._ __________ _ ___. 2 I (b)(3):10 lJSC §424 ! I I J (b)(3):10 usc §424 '-:-:---:-:-- --'I Th~ ~ontracted with Information Manufacturing Corporation (IMC) to digitize nearly eight million records. The IMF has temporarily retained the original . documents and digital copies. It intends to post all the digital files to the internet as soon as a suitable electronic document management system is completed. This wilr enable rapid and complete searching of the ftles. The IMF currently possesses numerp.u.s..--., document collections from Iraq which it intends to combine with the digitizedL__J files to become the Iraqi National Archives. These two examples set the stage for a larger effort to utilize the expertise across the U.S. government and academia to effectively exploit this cache ofinfonnation. This two part proposal will provide a means to properly data mine the captured Iraqi documents currently housed in the HARMONY data base. (b)(3):10 usc §424 (b)(3):10 usc §424 Attachments: a/s 3 J ' ':a r;· Ja. ~. 1 ·.,.- r. • JUH»T HW.'t. l,;o, . ~' tl\fH :V\'n Institute for Defenae Analyses 4850 M~rk Center Drive • Alexandria, Virginia 22311-1882 US Joint Forces Command Combat Study: Iraqi Perspectives Project (IPP) Unclassified Manuscript Overview ' 7 December 2005 Project Leader f._(b><_s) __ __, This briefing is · ·1 · · ' . · .... · . .. :. ~ .~ · ; . • . 1 V2.0 Background • The IPP combines extensive interviews of key regime and military leaders with hundreds of captured documents to present the events surrounding OIF from the enemy's perspective. • Project began in Sep 03 as an extension of the ongoing US Joint Forces Command lessons learned program to better understand US capabilities and effects. • Project Deliverables to date: - Over 120 classified briefs to senior defense, intelligence, and JPME audiences since Jan 04 - Support to related research (commissions, agencies, ~nd education) Classified report (May 2005) • An unclassified version of the IPP report was prepared with the following guidance: . - Remove any information that could aid an adversary (descriptions of specific US lessons, capabilities, vulnerabilities, etc) - Focus on the adversary perspective and the revealed history of the regime based on their own words. - Review with the appropriate authorities 2 ----- · -· .... -- - - · -·--- Sources • Captured Iraqi tapes: over 500 audio/video tapes from a library of over 7000 • Selected secondary sources 3 Manuscript Chapters • Nature of the Regime - Historical views of the regime from within. Establishes key aspects of Saddam's personality and its macro effect on Iraq's national security • Skewed Strategy - Strategic context of OIF from the regime's perspective • Military Effectiveness - Chronic effects of sanctions, internal security focus, and centralized authority • Crippled Operational Planning - Evolution of Iraqi national defense after Desert Storm and the dramatic changes on the eve of OIF • The Regime Prepares for War - Overview of Iraq's defensive preparations in the months prior to OIF. • Doomed Execution - A revealing narrative of the major combat phase of OIF from the point of view of Iraqi senior staff as well as selected corps and division commanders. • Annex: About the Project - A detailed unclassified background on the origins of the Iraqi Perspectives Project, its primary purposes, methodology, and utility. 4 ----·· -·-.-- ·-·- ·--·-- - ---·-- Nature of the Regime Ma~r subheadings : "The Setting", "Seizing Power", "Flawed ecisions", "War Planning", "Saddam's Leadership Style", and "Distorted World View" , ~ample Extracts: • On the challenges resistance in Kuwait after the 1991 invasion: - Minister: t(Your Excellency, what ifwefmd that some of the rebellious ones have little brothers and sisters that may avenge them? - Saddam: "Kill them all" • When in 1990 a leader of the Palestinian Islamic Jihad encouraged Saddam to declare himself the Caliph, Saddam responded with "it is too earlyfor that." • Tariq Aziz "If a military leader disappeared one did not ask to know what happened, since it was known that the security services dealt with the unfortunate individual. " · • Saddam on his analytical gifts H ••• / don't want either intelligence organization to give me analysis- that is my specialty ... which is what I used with the Iranians, some of it out of deduction and some of it through 5 invention {by} connecting the dots, a/ll1Jithout having hard evidence. " ·-·-- - -------·-·-·----·-- ... . ~- uuernal vs external threats", "Security Restrictions", .. '""' r•atrona-1-ln"""""!ffi.-.uence Campaign", "American 'paper tiger'", "Optimistic to the end" )ample Extracts: According the senior kaqi military personnel Saddam's priority for the armed forces was, in order, security of the regime (internal threats), security from regional threats (Iran and Israel)~ and defense against a Western coalition. Senior Iraqi Minister "I believe if any incident oc<;urs, the Americans will utilize their air strike methods, which they prefer and used recently, instead of r:;ending troops, based on their horrific experience in Somalia." )irector of Military Intelligence ((We thought the Coalition would go to Basra. taybe Amarah, and then the war would end." Chemical Ali opined that merica would quit the region, after they had to deat with thP Jublesome Shia on a regular basis. Military Effectiveness Major subheadings "Endemic Lying", "Dectining readiness", II Lesson Learned", "Guidance from Sad dam", "Private Armies", "Relatives and Sycophants", "Security and Command Limitations", and "Reflections of a Corps Commander" Sample Extracts: • An Iraqi Commander recalls "The army continued to fight the schemes of the Military · industrial commission, which played an important role. in promising secret weapons it would never deliver while most types of things we needed were neglected. ., • • • "After the liberation of our land and despite the fact that more than thirty countries headed by the occupation forces of the US rushed madly upon our Republican Guard, our performance was heroic. " RG Lesson. Learned Conference 1995 ' A 1999 Fedayeen Saddam planning document signed by Uday directs "special operations, assassinations, and bombings for the centers and traitor symbols in London, Iran, and the self-ruled areas." The Minister of Defense on the qualifications to be the Commander of the SRG ''He was Saddam 's cousin, but he had two other qualities which made him the best man for the job. First, he was not intelligent enough to represent a threat to the regime and second, he was not brave enough to participate in anyone else's plots." 7 ----------······ - Crippled Operational Planning Major subheadings "Operational Concept before December 2002"· and ''Operational Concept after 18 December 2002" Sample Extracts: • The focus after 1991 was to balance the traditional defense in depth concept of national defense with the requirement to prioritize internal security - a concept of pessimistic pragmatism. • On 18 December 2002, in a surprise meeting with the senior military commanders a ucrude and ugly" plan was unvei'led. The new concept focused . on an urban centric ring defense, it was tightly controlted by Saddam, and there would abe no changes. " • The Minister" of Defense noted of the urban centric plan that ~~There were a number of tasks to complete to ensure the city defenses, but there was no agreement as to what they were or how to get them done. " 8 The Regime Prepares for War Major subheadings liThe Diplomatic Campaign", "The role of ·wMD'", "Countering Coalition Psyops", "Logistical Preparations", "Oil Wells", "The Ba'ath Party", .. The Regular Army", "Defending a City", "The Republican Guard", "Final Events Prior to 19 March 2003'' Sample Extracts: • • • • In a post war critique of Iraq's diplomatic efforts Tariq Aziz noted "France and Ruissia did not help Iraq} they helped themselves ... We attempted to win favor with [them} through our oil contracts ... " The Republican Guard order to clean-up "old" WMD. sites before renewed UN Inspections. (Resulting act;ons described by US Secretary of State on 5 Feb 2003 to UN). Water, food, fuel, and munitions were dispersed at all levels in preparation for a "long air campaign. 'J "Saddam directed [lzzat Al-Dunj not to do anything to damage or destroy the oil.fields ... Saddam felt that destroying the oil fields would affect the morale of the soldiers and the people. " Secretary of the Republican Guard 9 ---- - - ·--··- ·-------- ~. · • it~...._ __ • •• • ··-- •• - ( I I Doomed Execution Maior subheadings ~~overview", "Effects of Precision Weapons", "The Air Campaign", "The Defense of the Southern Cities", ~·confusion Out West", and "The Ba'ath Regime Ends With a Whimper" Sample Extracts: • The combined effects of planned and unpfanned psychological operations messages, combined with the effect of precision strikes were "the bullet that hits the heart before hitting the body ... " • Defensive operations in most major cities were going very well through 4 April. ~~The Fedayeen and the heroic men of the party fired up two tanks ... the routed force of the enemy is estimated to be more than fifty tanks '' (F edayeen report from Basra) • On 2 April, Saddam declared the military actions in the South a {(strategic trick, and directed all forces to prepare for the main attack into Western Baghdad. 10 - - ----- ------·- · ------- Annex: About the Project • US Joint Forces Command Lessons Learned Program - Integrated with Joint Forces "Blue" assessments - Dynamic diagnosis versus st~tic post-mortem - The Iraqi Perspectives Project contribution • Historical Precedent and the "other side of the hill" - WWII Experience - Learning from contemporary history • Research Methodology and notes of sources used in this manuscript - Primary Sources • Oral Histories • Documents and electronic media - Historical Case-Study 11 Attachment 1 Institute for Defense Ana/vses (!DAJ Proposal Publishing Key Documents and Transcripts (rom the Regime o[Saddam Hussein Proposal: Publish, with limited analysis and commentary, a set of original and translated documents and transcripts from the former Saddam Hussein regime (collection includes data from 1 l)79 through Apri12003). Current US Government data base includes almost 700,000 documents and over 7,000 media fiies captured from the regime. Purpose: There are several near and long tenn purposes for this project; . a. Public Diplomacy and National Security Value: These documents will help c1arify the nature of the adversary and the complicity of others in perpetuating Saddam's regime. To put it simply- let words and documents of the former regime remind the free world why it was in a state of war with this regime from 1991 until2003. More specifically for the Islamic world - being confronted with the actual nature of the regime that dominated the Middle East for 30 years may deflate the growing myth of Saddam as a defender of the Muslim peoples. Some portion of world opinion will not be moved even when confronted with the starkest examples of conspiracy to conunit crimes against humanity. But that is not the target population. The politically undecided and morally ambivalent may take a new look at the larger issues involved in OIF and GWOT. This material is potentially a large piece of a strategic infonnation campaign. This campaign simply based on the nuth as represented in the words and actions of the adversary. b. Intelligence Value: Using an open source release of this material will provide to the unofficial "virtual" intelligence community a treasure trove of material to research and analyze. This is an untapped resourte and includes for example global media professionals, bloggers, and academics. The opportunity to generate feedback on this material and open new avenues of understanding could dwarf any official (traditional) analysis effort. There can be official and unofficial feedback mechanizes built into such an enterprise. This portion of the program can also be "contracted" out to a NGO that specializes in this kind ofinfonnation environment. The sheer volume of material in the possession of the US Government means that near-term value will be limited if only government intelligence analysts are available to study the material. c. Historical Value: These documents and transcripts will illuminate an area of the world most shroud~ in secrecy and myth. This material will help enlighten not only world opinion but also Iraqi opinion. The documents captured in OIF can be combined with other collections of documents from the Saddam regime to fonn the basis of a much deeper understanding of a volatile part of the world. Issues: a. Speed of Action. It is essential that this project be completed as soon as possible in order to have any eff ect on the current fight. .. :; ... ... ~ . b. Security: Most of the documents are Unclassified (For Official Use Only). This will require some level of clearance before allowing public access but there is no requirement to get the material "declassified". Documents suspected of having value to ongoing operations should be withheld. If the data released is limited to the period ofSaddam's regime this will be a minor issue. c. Legal Issues (Privacy Act, etc): For documents and transcripts where US citizens, corporations, bank accounts, etc) are mentioned there rna y be issues with open publication. However, these documents are historical in nature, are essentially "public records,", and every attempt can be made to protect personal data (i.e. pb<?ne numbers, bank accounts, etc) but names and dates should be included regardless of citizenship or affiliation. d. Embarrassing US officials and or past US policies: There will be cases v.ilere the documents indicate or imply errors of commission or omission with regard to US policy. Titis kind of criticism is inevitable. After careful study of this data for the past 18 months t.he vast majority either justifies US policy or provides clear rational for US actions. ln fact, the very content of Saddam 's conversations and docwnents provide many of the reasons for the policy errors (i.e. purposeful duplicity, obfuscation, and lying on the part ofSaddarn's regime). It should be remembered that US policy is under assault with or without these documents. These documents provide what is missing- the other side of the hill. e. Embarrassing foreign (allied and not) officials and or policies: This is potentially a more difficult issue that with US officials and policy. The default position should be maximum release. That said- care should be given to the ongoing strategic campaign in Iraq and exceptions to the maximum release policy should be expected. f. Credibility: This material should not be published or released directly by a US government agency. It should be published or released through a credible academic institution. The chain of custody for original material cannot be hidden but the efforts at publishing the archives should be as visible as possible. A combined interagency, ~ytical, historical, and subject matter expert panel should be empowered to screen the available material for inclusion. This panel should include like specialists from selected allies (lil< and AUS at a minimum). Metbod.-;: There seem to be three basic options. All options carry with tbem the same policy issues, opportunities noted above a. Internet Option: Develop a screening process for the existing Harmony database to purge do~uments that should not be released. Establish an archival management scheme to track documents back to original data (current metadata sheers contain information on capturing units, etc). Develop an interactive website where archival data is posted and updated. Invite users to comment, retranslate, and or debate the issues. Data posted should include original Arabic and full or partial translated pdf files. b. c. National Defense University (or equivalent) Option: Create a small activity (-10 staff) under the auspicious ofNDU. This staff would include archivist, data managers, historians, and subject matter experts. The organization would develop products for immediate publication, invite scholars from a wide variety of disciplines to conduct research, and mruntain an internet portal for rapid publication of analysis or release of documents (for example see Iraq Memory Foundation site). Civilian Academic or NGO Option: Much the same as above except the material would be provided by the government to a non-government activity for analysis and publication. Sponsorship would be predominately non-governmental. Precedent: In 194S the American and British armies captured the archives of the German Foreign Office which had been evacuated from Berlin. Use of the archives for intelligence purposes began inunediate!y. Later, it became evident that the documents concerning the aims and methods of German foreign policy should be published for the enlightenment of world opinion. including German opinion. In Jum: l946 the Department of State and the British Foreign Office agreed to sponsor jointly the publication of approximately twenty volumes of documents illustrative ofGennan foreign policy from 1 :J J 8 to 1945. The French Government subsequently became a party to this agreement. The docume:lts were to be printed in the original German, and the more important were also to be printed in English translation. It was agreed that the selection and editing were to be perfonned on the basis of the highest scholarly objectivity and that, to secure an authoritative and scholarly documentary record of German foreign policy, the services of private scholars should be enlisted, as well as the services of scholars in government service. Each Government reserved the right to publish separately any portion of the documents.