|
What set of indicators will adequately represent the humanitarian situation in a country? The portrayal of these conditions depends on the investigator’s ability to define a unique set of indicators that collectively characterizes, and is sensitive to changes in, each local situation. There are, however, commonalities across countries regarding the major threats to wellbeing and the likely places where information on these indicators will be found.
By applying the “4 + 4” human security subject area framework (section 2.2.1) one can ensure that the status of core humanitarian needs and conditions can be captured. But the decision regarding which measures are most relevant in each subject area, and the plan to pursue the available information with which to characterize these measures, will require creativity and opportunism in each individual case. A priority set of indicators to use as a starting point for capturing humanitarian conditions is presented in box 6.
Lacking confidence that one has covered all the key areas, investigators frequently collect too much information to “cover all the bets”. This frequently results in the collection of an unwieldy amount of information, leading authors to delayed and overly complex reporting, which can seriously dilute the ability to communicate their main points. The audience must be kept foremost in mind in presenting data and analyses; otherwise consumers of the information may suffer from information overload. Focusing information collection to the key areas identified in “4 + 4” human security subject areas allows for greater efficiency and effectiveness.
To be useful, information must be:
Definable—If there is an assessment that education has deteriorated, there must be a way to specify what deterioration means. Is it fewer children going to school (as in Haiti), lack of new textbooks (as in Serbia), or deterioration of physical plant (as in Iraq) and declining literacy (also in Iraq)?
Comparable—Continuing the above example: a useful operational definition of deterioration in education would require criteria that can be used in multiple locations, or that multiple informants can respond to, or both. A standard operational definition, and examination of literacy levels around the country and over time, is such an example.
Measurable—There is a wide range of precision in measurement, from the more qualitative (do you think education is “bad” or “good”) to the most quantitative (the percentage of 12-year-olds scoring about 500 points on the standard exam fell from 62 per cent to 58 per cent).
Accessible—If original data is to be collected, it should be easy to collect. If a secondary source it used, the information should be routinely available.
Representative of a defined population—If measurement is precise, but one does not know who does and who does not contribute to that collected information, the information is not useful. Does the information come from children in 2001, children aged 8 in 2001, children aged 8 in three districts in March of 2001, or children aged 8 in three districts that attended school on 3 March 2001. Each of these groups represents unique opportunities and limitations for comparing the results to other information.
The best way for investigators to know what the information represents is simply to ask:
-
Who was included when information was collected? Who was not included?
-
How was the information collected?
-
By whom and under what condition was it collected?
Information does not need to be quantitative or be available on the entire population of interest to be useful. Indeed, it is seldom viable to include all possible informants when information is collected. Were some kinds of people more likely to be included than others? If the group included is implied to be representative of a wider population, what would that wider group be? These are the types of questions that the investigator must answer to ensure that he/she has a good appreciation of the source, veracity and utility of the information.
Qualitative methods can provide useful impressions of what sources’ information best reflects the actual situation. Moreover, a qualitative method called triangulation assists in making judgement calls where information is inadequate. Triangulation is a process of taking the information that one has, to compare with information from a new source or a new informant. It is useful with both imperfect quantitative and qualitative information.
To compare information from multiple imperfect sources, the investigator must assess the potential biases from each source. This can be qualitative (“He is from the rival tribe”) or quantitative (“They counted 10 per cent fewer people from villages where malnutrition was higher”). To do this, one must identify the original source for the information and determine how it was collected. Each independent data source should only be included once, no matter how many people refer to it, unless there is reason to believe that it is better than other available sources.
Bias is a systematic error in the information gathered. Clinic exit surveys in Iraq, for example, consistently showed better nutrition than representative household surveys. Young children who receive vaccinations are slightly healthier than the general population of young children and this was reflected in nutrition status. A measuring scale that is offcalibration may consistently underweigh commodities. The amount of bias can be determined by seeing how off of true the scale is. For example, bias would exist if 30 per cent of babies are born in hospitals, but their weights are assumed to represent average birth weights nationally.
The biases that affect quantitative studies can be summarized in two areas:
- Threats to internal validity (are we really measuring what we think we are measuring?); and
- Threats to external validity (do these results accurately represent the wider population of interest that we think they represent).
Methods exist to examine and reduce the influence of many of these biases. The most important is recall bias. There are many forms recall bias may take. As one example, in charged political situations people are likely to recall the past as being either better or worse than it actually was. They are also more likely to want to remember politically important events in their lives and will tend to report those events even if they occurred outside the time frame asked about in the survey. It is usually much more effective to ask for opinions about the present time than the past or future. When questions need to be about the past, it is helpful to pinpoint a hard historical milestone that is hard for the respondent to be confused about. In contrast, when asking many people about events that happened during the last two years, they will frequently make mistakes recalling whether an event happened “20 months ago” or “26 months ago” and therefore may be more prone to overremember it as occurring more recently if it was important to them, and as being further in the past (or not think to mention it at all) if it weighed as less important.
Two important methods to reduce recall bias are to conduct a pretest for the survey questions in order to reveal the kinds of recall problems that occur, and to ask different questions about the same thing, each coming from a different angle that forces the respondent either to curtail their own bias or to provide answers that bound the true event within an interval.
In analysing results, bias can be understood and therefore filtered out through triangulation. For example, it is possible to conduct independent checks on recall: information on wages gathered in homes can be compared to data from employers, landlords, neighbours, credit unions, central banks, or planning ministries. “Leading questions” prejudice the responses gathered, and so differing ways to ask a question can be used to find the most effective approach. For example, good recall studies on mortality never mention the word “death”; they ask instead about people ever born and ask where they are now.
| 
