This article looks at methodological considerations in relation to researching ethnic/ religious minorities, in light of Channel 4 and ICM’s study entitled ‘What British Muslims Really Think?’ It also highlights the wider implications of methodological shortcomings on policy development and society as a whole.
In the last few weeks, Channel 4 and ICM have received some criticism about their survey methodology. In their defence, they have in large part been transparent about their approach and the challenges involved with researching British Muslims. As someone who specialises in researching Black, Asian and Minority Ethnic (BAME) communities, I do somewhat sympathise. Given that Muslims make up a relatively small proportion of the population (4.8%; 2.71 million people in England /Wales), achieving robust sample sizes through traditional channels is difficult. Online sample providers have few on their research panels and telephone samples do not flag religion. This currently leaves face-to-face data collection as the most effective way to research Muslims on a large-scale. However, given that approximately just 1 in 20 people are Muslim, tracking them down becomes extremely time consuming, making an already expensive method even more so.
ICM went with a face-to-face data collection method and to negotiate time and budget constraints, opted for a clustered sampling approach: focusing on areas with high concentrations of Muslims (20% or more of the local population). Doing so essentially enabled them to increase the likelihood (incidence rate) of finding Muslims to around 1 in 5. All in all, having a targeted sampling approach makes this sort of study more feasible; otherwise it may be too expensive for clients or beyond the scope of many research companies. As an ICM spokesperson stated to the Huffington Post, the study was “as rigorous as we think is possible within the parameters of realistic time and budget considerations….Nobody’s claiming that this is perfection, no survey ever has been or ever will be…. We couldn’t have done it otherwise, it simply can’t be done.”
ICM are correct in highlighting that, currently, there are a number of trade-offs to make when researching British Muslims (and other minority groups). Furthermore, they may even be right in claiming that this is the ‘most comprehensive’ study of British Muslims to date, and should be commended for their achievement. However, what this article will go on to demonstrate is that, given their methodological choices, they cannot claim to represent ‘What British Muslims Really Think,’ and why doing so could have harmful consequences for policy development and community relations. It does so by looking first at the survey’s target audience and then at its content.
Target Audience: Sample vs. Society
Looking at the sample structure in detail, it is evident that the study fails to include key sub-groups within the British Muslim population.
The 2011 Census showed that British Muslims have a remarkably young age profile, with 33% 15 or under (19% in the overall population) and Muslims accounting for 8.1% of all school aged children (5 to 15 yr olds). Given the sheer size of the young population (representing 1 in 3 Muslims) and the influence they will have on future societal outcomes, surely understanding them is vital to policy development? Yet the study only included those aged 18 and over.
The study also fails to pay attention to cultural differences between British Muslim. Although 68% are of Asian origin, the sample includes 83%, leaving little room for other prominent ethnic groups (i.e. Arabs and Africans). This makes it almost impossible to disentangle religious vs. cultural factors. Having worked in this area for some years now, I am aware of the impact of cultural differences on policy initiatives. A more nuanced understanding is therefore imperative.
Perhaps the most commonly noted criticism of the study is its focus on areas with high concentrations of Muslims (20% or higher). It is unclear exactly which geographic locations were covered, but ICM state that they account for approximately 50% of the Muslim population. Based on Census 2011 data, there are approximately nine local authorities that would classify. However, it should be noted that some of these are amongst the most deprived in the country (rankings based on ONS Indices of Deprivation 2015): Birmingham (6th), Bradford (11th), Blackburn and Darwen (12th), and Tower Hamlets (24th). Without controlling for such social and economic circumstances, can the results really represent segments of the Muslim population who do not face them?
Importantly, when comparing the 2001 and 2011 Census data, it is evident that there have been substantial improvements in socio-economic outcomes for Muslims. These include a reduction in the proportion with no qualifications (39% in 2001 vs. 26% in 2011), an increase in the percentage (of over 16s) with ‘Degree level and above’ qualifications (24% Muslims vs. 27% in the general population), and a similar proportion in ‘Higher professional occupations’ when compared to the overall population (5.5% and 7.6% respectively).
The increased prominence of British Muslims in public life also highlights the need for a more detailed socio-economic assessment. Notable examples include: Sadiq Khan, the current front-runner for London’s next Mayor; Sajid Javid, the Secretary of State for Business, Innovation and Skills; Baroness Sayeeda Warsi, a former Cabinet Minister; and the acclaimed lawyer and now wife of a Hollywood superstar, Amal Clooney.
By failing to look at Muslims who are progressing within society, the picture is only half painted. It may be that they hold the exact same views as the sample; this would go against Trevor Phillips’ suggestion that increased mixing and integration are the solution, potentially making the story of the study even worse. Or, it could demonstrate that their views are more in-line with mainstream society. In the case of the latter, when formulating policy, would we not benefit from insight into what actually works and why?
Content: To Tell or Not to Tell?!
Being a British Muslim myself, I could not identify with many of the headline findings, as these are not views I personally hold nor are they prominent within my immediate circle of family/ friends. However, this is not to say that the results do not reflect the population that was surveyed. This disparity simply highlights that we need to exercise caution when reporting and interpreting these results, especially in light of the methodological constraints.
The study claims to reveal attitudes that will ‘surprise’ you and, having analysed the data tables, I certainly agree. However, many of things I found particularly surprising were not reported in very much (if any) detail. Here are just a few:
My analysis suggests that, despite certain challenges (i.e. harassment), Muslims do feel part of British society. However, disappointingly, when these positive statistics were reported by Trevor Phillips, it was insinuated that the primary reason is the ability to practice religion free from the persecution:
“More than eight in 10 Muslims say that they are happy living here, and feel British. Their preoccupations aren’t that different from most people’s: family life, their children’s future, economic security. But Muslims also prize the British way of life for a reason increasingly unimportant to non-Muslims: freedom to practice their religion any way they see fit. In the Indian subcontinent, Muslims are subject to Hindu persecution. In Nigeria, north Africa and the Middle East, the brutal Islamists of Boko Haram, Isis and al-Qaeda make the slightest deviation a potential suicide mission.” (‘What Do British Muslims Really Think?’, The Sunday Times Magazine, April 10th 2016)
Without having the necessary data or qualitative insights to establish the strength (if any) of the relationship between belonging in Britain and fears of religious persecution elsewhere, it seems a bit of a stretch to make such connections. Furthermore, these inferences are even less legitimate given that approximately three-quarters (72%) of the survey sample are from Pakistan (55%) or Bangladesh (17%), predominately Muslim countries (in the Indian Subcontinent) where people are unlikely to face the sort of religious persecution referenced.
Finally, whilst it is useful to have a control group to compare and contextualise the Muslim data, the sample is representative of the general population as whole; it does not match the characteristics of the Muslim sample (e.g. including a high proportion of deprived areas). But perhaps the most important representation that is lacking is of other religious groups who could be classified as ‘devout’ or ‘conservative’. Without including them (be they Christian, Jewish, Buddhist, Hindu or Sikh), when interpreting certain findings, we are in many ways left comparing apples with mangos.
Lessons for the future:
Having specialised in this field since starting my career, I have seen and completely believe in the enormous potential of research when it comes to giving lesser-heard groups a voice. However, I am also well aware of the challenges and limitations we face as an industry. As such, we may often need to use more pragmatic approaches than we would in general population research. Doing so is fine, as long as we clearly acknowledge the constraints of our work and the extent to which findings can be extrapolated to the wider population of interest. It’s when we fail to do this that we risk presenting a skewed picture that, instead of empowering communities, could well have the opposite effect.
For now, perhaps the way forward is to accept the constraints we face and develop new, bespoke methodologies for researching minority groups. Though the result may not be as statistically robust as we’re able to achieve when looking at larger groups, it doesn’t mean they won’t be insightful and even game-changing.
————
About the author:
Sania Haq is Head of Research at AudienceNet. For further information on best practices when researching minority groups, email sania@audiencenet.co.uk
