This report summarises the findings of the evaluation of the Avenues pilot. Avenues tested a modified Domestic Purposes Benefit application process and ran from August 2001 to January 2002. The pilot tested the assumption that enhanced case management would alter the number of people who pursued a Domestic Purposes Benefit. The pilot did not significantly alter the numbers applying.
Purpose
The evaluation objectives were:
• to assess the extent to which Avenues was implemented as intended, including a description and comparison of the intended and actual operation of Avenues
• to assess the extent to which Avenues achieved its intended objectives
• to assess the outcomes for Avenues DPB-SP applicants, including the income support options taken up and any other outcomes
• to assess the impact of Avenues on DPB-SP applicants, on Work and Income staff and on Work and Income administration of the DPB-SP application processes.
Methodology
A mixed-method approach was used in the evaluation of the pilot, reflecting the evaluation’s dual focus on design/implementation and outcomes/programme impact. Each method is described below.
2.3.1 Databases
An Excel database was developed for the pilot to record information about the Avenues process that could not be captured via the Work and Income benefit administrative database (SWIFTT). This included referral information and interview details such as duration and location. SWIFTT was used to provide information about client demographic characteristics and benefit uptake trends.
2.3.2 Interviews
Semi-structured face-to-face interviews of approximately one hour duration were conducted in three regions: South Auckland, Central and Bay of Plenty. Interviewees included Avenues clients (n=22), and Work and Income client advisors (n=7), case managers (n=5), and call centre customer service representatives (n=4).
2.3.3 Telephone survey
A telephone survey was conducted to explore alternative options pursued by Avenues participants who did not proceed with a core benefit application. The sample was drawn from the Avenues and the SWIFTT databases: clients selected were those who had not applied for a core benefit before 14 November, 2001. When these clients were contacted in the first week of December 2001, 59% of those contacted reported having applied for a core benefit which raised concerns about the rigour of the sampling technique and the possibility of a non-representative sample. Consequently, with one exception (see 3.16.1), findings from the telephone survey have not been included in this report (see section 7.1.2 for details).
2.3.4 Impact analysis
Impact analysis was conducted to assess whether Avenues altered participants’ decision to apply for the DPB-SP. A discontinuity with a baseline control was used which involved modelling the monthly number of applications, grants, and declines in the pilot sites. This used information on the number of applications, grants, and declines in non-pilot sites for two years prior to the introduction. Using this model, it was possible to estimate the expected number of applications, grants and declines in the pilot sites, if Avenues had not been implemented, and therefore its impact.
2.4 Evaluation resources
MSD Operational Policy requested ERE to produce this report.. They reviewed and analysed Avenues documentation and, interviewed Avenues project managers. ERE staff analysed quantitative data from the Avenues operational database and from SWIFTT. A budget provided by Operational Policy was used to contract independent external researchers to undertake the in-depth interviews and a telephone survey.