No clarity yet but more questions on HDB Survery

Last week, I wrote about how the HDB Sample Survey 2008 was misleading and contained serious statistical issues. Leong Sze Hian did an independent analysis and came to a similar conclusion.

http://theonlinecitizen.com/2010/02/hdb-95-per-cent-happy-with-their-flats-%e2%80%93-really-or-just-half-the-story/

What I had written was even featured in the Reach forum. An anonymous poster posted the entire thread from TOC into the Reach discussion thread on HDB.

http://app.reach.gov.sg/reach/YourSay/YourDiscussionCorner/tabid/117/ptid/414/page/59/totrecs/591/threadid/2427/forumtype/posts/Default.aspx

It is strange that up till now there isn’t any reactions to what I have written. The formidable PAP Internet team which is supposed to “counter” unfavourable, untrue articles by the Opposition did not show up. Instead among the people who commented, there was almost universal agreement that there was something “fishy” about the statistics.

As I had previously written, I do not understand the results of the press release. In particular, the following seems to contradict the daily experience of those that live in the HDB heartlands.

96.4% of all HDB households surveyed said they were satisfied with their flats, while 95.1% were satisfied with their neighbourhood.

I therefore wrote the following email to HDB to ask them for the survey form and the methodology of the survey.

Dear Sir/ Madam,

I have read with interest about your recent release on HDB Sample Survery 2008. I would like to have a copy of the survey questionnaire and understand more about the survey methodology applied in this survery.

I am particularly interested on how the 94% of HDB dwellers satisfaction rate was derived from the survey. I would like to know the sample size and the sampling methodology used in this survey.

I would appreciate if you could send me the survey questionnaire and necessary information as soon as possible. Thank you.

Goh Meng Seng

I received the following reply:


Dear Mr Goh,

HDB SAMPLE SURVEY 2008

We thank you for your interest in HDB's SHS 2008 and your enquiries on the survey.

2. Close to 8,000 households across the island were sucessfully surveyed, yielding an overall sampling error of +1.2% at 95% confidence level. A set of weight was used to generalise the survey data to the population level, so that the findings reported are representative of all HDB households. A dual-modal data collection method was used, encompassing Internet survey (e-survey) as well as the conventional face-to-face survey at residents' home.

3. Residents' satisfaction was measured on a 4-point scale, ranging from "Very Satisfied", "Satisfied", "Dissatisfied" to "Very Dissatisfied". The proportion of households who were satisfied consists of those who indicated either they were "Very Satisfied" or "Satisfied".

4. We are unable to forward you a copy of the survey questionnaire. However, you can look forward to more details in our monographs, available for purchase at end Mar/early Apr 2010.



Yours sincerely,

GOH LI PING
SENIOR RESEARCH OFFICER
RESEARCH & PLANNING DEPARTMENT


While polite, the reply unfortunately does not provide any of meaningful answers. It did not answer some fundamental question on the sampling methodology applied when the face to face interviews were carried out. It does raise some doubts technically but I will confirm and comment on my doubts after I get more information and clarity once I get hold of the monographs. On the other hand, the email raises 3 additional questions:

1) The survey was conducted in 2008. It is very surprising to read that the results will not be ready until Mar/Arp 2010. This is very unusual as the normal time to complete a survey of this size is about 3 to 6 month, or at most 1 year. If this data is still not ready, then what data has Mr Mah been using to make policy? The HDB website shows that the most recent data release is 2003. Was Mr Mah using vintage 2003 data to make policy decisions?

2) It is very strange that HDB has refused to release the questionnaire. This is a basic requirement in all research so that those reading the numbers can understand how the numbers were derived. The Singapore Department of Statistics releases all questionnaires from their surveys. Questionnaires cannot be classified as being Confidential or Secret. They belong in the public domain since to use them to gather information, you have to show them to the public. This is especially so when it is claimed that internet survey was carried out.

3) The 4 point scale is a non-standard method to generate a satisfaction score. The more standard scale is to use a 5 point scale or a 10 point scale. It is unknown why HDB chose to use such an unorthodox scale. A known research problem with such a scale is that it tends to give inflated results. This is because respondents who are indifferent (neither satisfied nor dissatisfied) are not given a choice. When forced the tendency of such indifferent respondents is to give a 3 rating, inflating the results.

Mar/early April is round the corner. I will wait for the monographs to come out and do a more detailed analysis of the findings.

Goh Meng Seng