|
Most of the data used in determining baseline conditions and the effects of sanctions are gathered from existing sources, whereas original data is usually generated sparingly, to fill gaps.
Existing sources of data—which are referred to as secondary data sources—include international, national and local institutions. National governmental agencies are usually the dominant source of information upon which many international (UN, World Bank, etc.) publications depend. Yet national sources of data are frequently biased or inaccurate or fail to reflect the entire population comprehensively.
The periodicity or frequency of updating national statistics will powerfully determine and constrain the value of data sets found. As a general rule, the more emergencyaffected and poor the country, the less likely it is that reported data will be accurate. Even population statistics—the size of the population, income, vaccination rates—may be many years out of date.
Humanitarian agencies generally collect information on the services that they provide and the number of beneficiaries served. This process data about their activities and beneficiaries is of limited value in providing a sufficient picture of a population for detailed monitoring. However, some nongovernmental humanitarian agencies also conduct, on occasion, more statistically rigorous surveys of the broader changes in the population. But these surveys are usually limited in scope to a small geographical area—a district or camp, not a whole country.
Similarly, much of the data that assessments draw on are process data from government departments or agencies. The number of civic services provided by central and local government institutions or the number of people served can serve as important indicators of the production or demand for services. In Serbia for example, the health system provided a stable number of emergency consultations, but reduced wellchild visits and increased the number of immunizations provided during sanctions. This was because demand for immunizations rose when families knew fewer medicines and routine visits would be available to them.
In principle, information about how conditions are changing can be gleaned from agencies intimately involved in meeting changing demands on a daily basis since service providers usually count the number of people seen each day. Data of this type are typically the most widely available. This information is available in institutional files or annual reports, but it cannot be used to establish rates for the population as a whole. They can be used to track demand, but not need.
Thus, any time indicators derived from the number of services provided are used they will likely provide an incomplete description of the general condition of the population. Private services are seldom included in such counts, quality is difficult to assess, and the population’s need for such services cannot be determined. For example, the average number of medical visits in Cuba from conception through one year of age in 1990 was 22— far more than could actually be useful. But since the system could not respond to some needs (such as higher quality foods and medicines) it continued to raise the number of services it could offer. Populationbased surveys of prevailing conditions are the best way to get around the limitations of servicedelivery data.
With this in mind, UN organizations and the World Bank, often in partnership with national Governments, engage in occasional largescale surveys of economic and social conditions in many countries. Often this data is available on the sponsoring organizations’ web sites. Prime among these organizations are the World Bank, UNICEF, the World Health Organization (WHO) and the Pan American Health Organization, UN Development Programme (UNDP), UNESCO, the Office of the UN High Commissioner for Refugees (UNHCR), the Office of the United Nations High Commissioner for Human Rights (OHCHR), the World Food Programme (WFP), the UN Environment Programme (UNEP), the United Nations Population Fund (UNFPA), and the Food and Agriculture Organization of the United Nations (FAO). UN sources are frequently combined and made available via the web site of the UN Statistics Division or Common Country Assessments (CCAs). Specialized web sites also collate detailed data from different sectors or subsectors. For example the UN Administrative Coordinating Committee/SCN publishes an electronic summary four times a year that draws together malnutrition and mortality data from a range of agencies working in emergencies.
Outside the UN system, human rights organizations and civil society monitoring agencies including (among many others) Human Rights Watch, SIPRI, Transparency International, the Norwegian Refugee Council and the International Crisis Group collect information on many countries. The number of groups and electronic databases prepared by such organizations is growing rapidly.
When a database is uncovered, methodologic introductions, qualifiers or footnotes should be read carefully. Were these data collected by the organization or are they reprinted from another source? The original source should provide information on the time period examined, data definitions, information collection methods and population included. Potentially, the best data sources come from either universal population counts (censuses) or representative sample surveys covering all groups and areas of a country. Many countries have a census to count the population or households every 10 years; few do them more often and some have not had a census for more than two decades. A national census is often unavailable in detail except in the planning office of a government.
There are currently only two widely available sources of representative sample information from surveys about important health and demographic indicators in most developing countries. The first is UNICEF’s Multiple Indicator Cluster Sample Survey (MICS), which measures conditions of child and maternal health and wellbeing in more than 60 countries. A recent round of MICS surveys, comparable to the first group of surveys in 1996, was carried out during 1998-2000 in 55 countries.
The second is the series of “Demographic and Health Surveys” (DHS), which are nationally representative household surveys with large sample sizes of about 5,000 households. The sample sizes are carefully calculated to be statistically significant and representative of the country as a whole. DHS surveys provide data for indicators of population, health, women’s status, fertility, children’s status and nutrition. Many countries have carried out DHS surveys every five years; periodic surveys are available online for 30 countries and others are available offline or via government planning departments.
For most countries additional surveys from international organizations or estimates and projections are available from UN organizations, economic research groups, and newspaper or encyclopedia “fact books”. Most of these sources are now available on the Internet, or can be accessed in library reference collections, including UN, donor and university archives.
Unlike secondary data sources, in original, or primary, data collection it is possible to select whom to include in the study and what is to be studied. The advantages of primary data collection therefore are: (1) the timeliness of the data can be controlled; (2) the representativeness of the data can be ensured; and (3) the type of information desired can be directly determined by the design of the survey questions.
If the goal of the primary data collection is to glean information about the larger population, statistical methods require that the sample (typically of people) be drawn as randomly as possible from the whole population, which means it will include dispersed people around the country. UN agencies and NGOs are increasingly making use of twostage cluster sampling techniques that provide a reasonable degree of representativeness in circumstances where census information and lists of citizens are inaccurate or biased to exclude groups of peoples systematically. When done poorly, the conclusions from such studies have “gone beyond their data” to make generalizations which could not be justified.
When embarking on the collection of primary data, standardization of definitions for key variables should be established with unambiguous operational definitions in order for the data to be understood by others who might review the raw data afterwards. Supervisory efforts have to be made to assure that all participants are in fact using standard approaches and definitions in the field. In fact, almost every term or variable requires attention since even the most common terms take on different shades of meaning from culture to culture, from researcher to researcher and from respondent to respondent.
For example, many investigators assume that their staff all share a common understanding of what constitutes a “household”, while in reality, there can be many interpretations of this. Pretest surveys will typically reveal the range of understandings that the target population have as well as the range of options to consider in establishing a definition. Establishing a common operative definition for key variables prior to the main survey is essential to ensure the quality and comparability of the resulting information.
Three types of studies are frequently used to gather original data on humanitarian conditions: crosssectional studies, panel studies and longitudinal studies.
The simplest type of original or primary data collection is a onetime survey. Sometimes called crosssectional studies, the objective is to collect information to characterize the humanitarian situation at a specific point in time. In other words, cross-sectional studies take a snapshot of how things appear and relationships at that moment, but do not capture patterns of change. In countries under sanctions, this has been the most common approach. Such a study provides potentially useful information about differences between groups but cannot be used to track trends over time. A good example was a survey by an independent group of scholars, the International Study Team, in Iraq eight months after the 1991 Gulf War.22 The data they collected provided the first nationallevel indicators for child malnutrition; all subsequent studies refer back to that source.
Thus, the quality of any causal model that only draws on crosssectional data study will be weak because by itself crosssectional data does not reveal dynamics or temporal relations.
Various sets of crosssectional information from different time periods can reveal changes over time. At a minimum, crosssectional surveys help to establish a baseline, to be followed using more powerful study models as sanctions progress.23
A better approach than a crosssectional study is a panel study, where crosssections are taken periodically using a common, systematic method. In Iraq, for example, nationallevel household demographic and nutrition surveys were carried out each five years from 1983-1993. The information gathered on sources and levels of income, family formation and child bearing are excellent examples of sensitive, longitudinal monitoring indicators. Unfortunately, after 1993, monitoring was interrupted.
The Government of Iraq failed to carry out a subsequent survey until 2000 and survey data was not available again until 2002. Similarly, despite an apparent demographic emergency documented in 1991, the results of another demographic survey only became available (thanks to UNICEF expertise and funding) in 2000. These special surveys have provided most of the useful information available on humanitarian conditions during sanctions in Iraq.
When there are only a few panels over a period of years, or where there is little continuity in the information gathered or the approaches used to gather it, a panel study will net few benefits.
Gaps in the regular collection of data, particularly in times of crisis, reduce the ability to make sense of data later when surveys are restarted. For example, although hundreds of individuals and groups had visited Iraqi hospitals in the 1990s, only one group used a list of standard questions and observation goals to set a baseline level for comparisons. With a little coordination, others could readily have done the same, which would have contributed to far better identification of changes in conditions around the country.
Panel studies that do not follow up with the same individuals during each panel have to examine whether the people in different panels are indeed comparable. Migration, attrition or mortality change the composition of the communities from which samples are drawn. Therefore, the question to be examined is whether substantial peculiar changes occur due to any of these forces. Seasonal cycles for phenomena such as malnutrition need to be understood if samples are to be taken in a manner that permits valid comparisons. For example, significant changes in malnutrition rates observed over the course of several months could be found to replicate a regular cycle of increase and decrease that occurs each year due to rain, climatic conditions and harvests.
A study of conditions at an initial point in time, preferably prior to the imposition of sanctions—a baseline study (see section 5.3)—is key in avoiding these problems. A good baseline can be prescriptive, suggesting the frequency with which future panels should be taken, the key information to collect in each panel, and the groups to include in those panels. It is almost impossible to go back and reinvent the questions or correct ambiguous definitions after this information is collected. Orientation on how to coordinate from the start cannot be substituted later.
Another challenge with panel studies is that they sometimes focus too heavily on collecting information on the outcomes of interest rather than relevant process information.
Panels are good for special studies on subgroups of the population. If, for example, it is suspected that children in one region, in a new rural settlement or of one ethnic group are doing worse than others, the normal panel procedures for a nutrition study in the whole country can be utilized on a onetime basis among the population subgroups to learn about their status as compared to national trends. Depending on the results, it will be clear if further panels of this subgroup should be undertaken when the routine cycle of periodic panels continues.
Whereas a crosssectional study looks only at one point in time and a panel study repeats periodic crosssections, sometimes it is possible to do ongoing monitoring in a continuous manner. This is a longitudinal study. When longitudinal studies are properly controlled and track the same individuals over time, they provide statistically powerful results.
The best studies follow the changes that occur to people throughout the period of sanctions. When it is not the same individuals tracked throughout, statistical validity and significance is lost. Unfortunately, most of the information available at regular intervals is institutional data from service statistics systems, which tracks a lot of people but does not track the same individuals throughout. For example, data may be available on the number of children seen in clinics for malnutrition each month. The limitation with this kind of data is that different individuals are included in each panel, so it is not a true longitudinal study.
A longitudinal study selects a group of individuals at baseline and follows these same individuals forward, observing how they are affected by sanctions. For example, families could provide income summaries each month. Changes in buying power among those living on public salaries, compared to those in the private sector, can be recorded and compared over time if the same families are followed throughout. This approach will potentially provide the best quality of comparable information.
Institutional data usually yields information on services provided but not on the population from which users come. If, however, everyone is included in the data, and if the population is not changing, such service statistics can be used to estimate population rates. For example, if almost all health care is provided via government hospitals, then changes in the number of hospital visits more closely represents a true change in the overall national pattern of use of medical care services. This was the case in Cuba and Serbia, allowing assessment teams to draw conclusions about how medical needs changed over time.
| 
