About two thirds of deaths worldwide are not recorded. A network of hundreds of health and population researchers hopes to change this with a new dataset of 110 000 deaths recorded over two decades in low- and middle-income populations across Africa and South-East Asia.

A shortage of health care practitioners and no legal requirements for death certificates mean that the majority of deaths do not make it into official registries, and information about who died and what they died of is based largely on estimates. This affects the ability of health care policy makers to reliably design the best possible health care interventions for their countries; and has wider implications for citizenship and social development.

The INDEPTH dataset, which is the largest ever of its kind, shows that in Africa, while the number of deaths from HIV/AIDS has recently declined due to the availability of antiretroviral treatment, the number of deaths from non-communicable diseases such as cancer, diabetes and heart disease has increased, intensifying concerns about risky lifestyle behaviours such as drinking, smoking and poor diet. In addition, the high prevalence of both HIV and non-communicable diseases in the same areas may be the most extreme coexistence of chronic infectious and non-infectious conditions ever, and places a huge burden on health care systems as well as home carers.

The data was collected by 22 research sites across 13 countries including South Africa, Ghana, Kenya, Bangladesh and Vietnam. In South Africa, data was collected from the MRC/Wits Rural Public Health and Health Transitions Research Unit in the Agincourt area of Mpumalanga, and the Africa Centre for Health and Population Studies in the Umkhanyakude district of KwaZulu-Natal.

The process involved an unprecedented degree of collaboration between health and demographic surveillance systems, all members of the INDEPTH Network, and represents a breakthrough in demonstrating how to provide actual records for deaths on-the-ground, rather than relying on mathematical models.

A method of "verbal autopsies" was used to interview care-givers as many people die at home or away from hospitals and clinics. Verbal autopsies were conducted by local field staff who entered homes where there had been a death and administered a standardised questionnaire, lasting some 15-30 minutes, about the signs and symptoms of illness, or details of an accident, before the person died. The information was analysed by a software system that generated a probable cause of death. The method standardises assessment across different settings, is reliable and cost-effective, and could be used by national health authorities.

"Every health authority in the world has a shortage of resources. Knowing about cause of death really helps you to priorities your resources, to make things better for the future. Obviously knowing cause of death doesn't make a difference for the person who has died, but it can make a big difference to the people who have not yet died," says Professor Peter Byass, Director of the Umeå Centre for Global Health Research in Sweden and Honorary Professor in the School of Public Health at Wits University.

The data from all 22 sites had to be harmonised and made publicly available, a mammoth task funded by the Wellcome Trust and others, and findings have been published in a special issue of the journal, Global Health Action. There are six multisite papers dealing with specific causes of death: HIV/AIDS, malaria, pregnancy-related, external causes (including accidental death and suicide), adult non-communicable diseases (such as stroke, diabetes and cancers), and childhood illness. Other papers describe methods, and many sites have contributed papers with local details.

Professor Kathleen Kahn, Senior Scientist at the MRC/Wits Rural Public Health and Health Transitions Research Unit (Agincourt), Health and Population Division of the Wits School of Public Health, says: "The power of being able to look at all these numbers is that it lifts the lid on the unknowns, an essential starting point in the battle to address global inequities. There is now a publicly accessible database, available to any scientist or analyst in the world. This is unprecedented for low- and middle-income countries."

Professor Steve Tollman, Director of the MRC/Wits Rural Public Health and Health Transitions Research Unit, and Head of the Health and Population Division of the Wits School of Public Health, says: "Now that we have done it, it will not be too difficult to continue doing it over time, and the data should become an important barometer of the impact of national policies - how to intervene most effectively against the most important problems."