Purpose
The purpose of this evaluation is to assess the Family Start programme’s impact and effectiveness. To achieve this purpose, the evaluation was undertaken through two workstreams:
- an impact evaluation, the purpose of which was to assess the impact that Family Start participation is having on the wellbeing of New Zealand children and their whānau. The evaluation examined a range of health, educational, and social outcomes for children, including separate analyses for Māori and Pasifika children.
- a process evaluation, the purpose of which was to explore the effectiveness of the programme’s design and delivery, including how client whānau experience the programme, and how Family Start providers can optimise positive impacts for children and their whānau.
Methodology
The impact evaluation was conducted using data from Statistics New Zealand’s Integrated Data Infrastructure (IDI), using two quasi-experimental methods: Propensity Score Matching (PSM) for the period from 2009 to 2015 and Difference-in-Differences (DiD) for the period from 2003 to 2015.
As DiD measures change across a given area (e.g., a Territorial Local Authority, TLA), results from the DiD analyses should be treated with caution; this is because the number of Family Start participants is a small proportion of the total population of each TLA, which makes it difficult to detect any changes. The evaluation therefore relies more heavily on the PSM results, which are more reliable in this context.
The process evaluation, conducted in 2019, drew strongly on qualitative data from five providerlevel case studies (three kaupapa Māori providers, one Pasifika provider and one Pākehā provider) which included in-depth interviews with Family Start managers, workers, referrers and whānau that were clients of Family Start. Other data collection methods included interviews with Oranga Tamariki officials and other key stakeholders, an electronic diary activity during which Family Start workers recorded their experiences and perceptions in their role working with whānau, a review of key documents and analysis of Family Start administrative data.
Key Results
Overall, the results of the evaluation indicate that Family Start is having a positive impact on the wellbeing and safety of participant children and their whānau, and improving engagement with health-promoting public health services.
Findings suggest that:
- Participation in Family Start improves child safety and analysis estimated that children’s participation in Family Start is associated with a reduction in deaths from all causes in their first year of life
- Whānau received an effective, reliable and safe service from the Family Start Programme and whānau consider that Family Start has improved their parenting skills
- Family Start increases participation in health services such as enrollment with a Primary Health Organisation and the likelihood of being fully imminused by their first birthday
- Participants in Family Start are more likely to experience interactions with Oranga Tamariki, which may reflect the programme’s success in identifying and addressing family violence
- Some aspects of the Family Start model do not align with Māori worldviews and practices
- Family Start workers are feeling stretched by the challenge of working with whānau who need intensive support
- Family Start providers identified a need for nationally-consistent training.