Muriel Cummins

Establishing the National Disability Insurance Scheme (NDIS) Act 2013 was a big brave step towards progressive disability policy and support in this country. Disability Discrimination Commissioner Dr Ben Gauntlett recently made the point that the NDIS and its underpinning legislation is comparable to the Sydney Opera House in terms of it being an Australian icon. Dr Gauntlett expressed concern that the latest NDIS Independent Assessment pilot – which assessed functional capacity using clinically unproven methods – may place Australia at risk of breaching its obligations under the United Nations Convention on the Rights of the Persons with Disability.

Occupational Therapy Australia (OTA) is the peak body representing occupational therapists all over Australia. Assessment of function is a central focus of the occupational therapy profession. OTA has looked very carefully into the plans for NDIS assessments, and they have strong and valid concerns.

OTA has made submissions to both the NDIA Consultation and the Australian Parliament’s Inquiry into NDIS Independent Assessments. Myself and another OTA representative appeared as witnesses at the parliamentary inquiry’s first public hearing. We explained why the NDIA’s process around assessing functional capacity is problematic – and what needs to change to keep it ethical and appropriate.

“It is entirely inappropriate to determine a person’s eligibility for NDIS supports using a set of tools which were neither designed nor validated for this purpose and population.” Read more in OTA’s latest submissions to the NDIA here: https://t.co/L28X2dN7bH pic.twitter.com/hmzMksIm8D

— OTA (@otaust) March 9, 2021

After hearing our concerns Senator Carol Brown asked us to provide an answer to one of the biggest questions everyone is asking right now. She asked “Do you see the tools that have been selected by the NDIA – such as the WHODAS and PEDI-CAT – as being appropriate measurements to inform a funding decision in a NDIS plan?”

Or in plainer language …

“Independent Assessments are made up of a number of assessment tools. They will be used to calculate plan funding. Is this ok?”

The short answer is no. (The long answer is at the bottom of this page).

OTA does not think the Independent Assessment (IA) tools are capable of informing funding decisions – either individually or bundled together.

There are two colossal elephants in the NDIA policy room

Both are based on unproven assumptions. That is – things the NDIA think are true, but without any evidence to prove it.

The first assumes that a selection of IA tools can accurately assess functional capacity. OTA does not believe they can.

The second assumes that IA scores can somehow translate into an accurate support-budget. OTA does not believe they can.

Short-cutting assessment of functional capacity cannot result in the creation of equal opportunity for people with disabilities. Functional assessment must be done with integrity of process, and it must honour the rights and lived experience of the person with disability.

OTA believes people with disability deserve better than unproven assumptions directing their NDIS journey. OTA states that:

“people with a disability in Australia have a right to an evidence-based, robust, and safe process for assessment of functional capacity to determine access to the NDIS, and to determine funding for reasonable and necessary supports.”

The proposed Independent Assessment tools are not meant to be used like this

OTA holds a clear position on the inappropriateness of the IA assessment tools, individually or as a bundle, to accurately determine functional capacity. These assessment tools are only accurate when used for their intended purpose, with the disability groups they are researched and designed for. They lose their accuracy, validity and reliability when used in other ways. Validity means, can we trust that the tools can actually measure functional capacity? Reliability means, can we trust that the tools can measure functional capacity consistently, for all disability types?

For example, the Vineland-3 does not have proven validity for use with people with psychosocial disability. It is entirely possible for a person with a psychosocial disability to have substantially reduced functional capacity for self-care. And that could easily slip through the cracks in the IA and be missed entirely.

The IA will likely be more costly – both in terms of dollar amounts and human suffering – than a more robust alternative that includes assessment of individual support needs.

There is no single tool or bundle of tools proven to accurately assess the functional capacity of people with disabilities. It’s not for lack of trying – Australian researchers and the World Health Organisation have been researching this for years.

The IA does not have the proven ability to measure the true impact of disability or substantially reduced functional capacity. The recently closed pilot included a survey which asked people about their satisfaction and experience of having an IA. It did not measure the IA’s specific ability to measure functional capacity.

The NDIA recently acknowledged in an update to their submission to the Australian Parliament’s Inquiry into Independent Assessment, that the IA toolkit lacks validity (meaning accuracy of assessment) and reliability (meaning consistent assessment across disability types) in the NDIS context, stating:

“The tools selected have proven reliability and validity in the contexts for which they are designed but this cannot be extended with great confidence to other contexts” (p.16).

Their proposed solution is to roll out the IA regardless, with “progressive evaluation”. No further explanation is offered on “progressive evaluation”. No detail on methodology (how), or ethical or clinical (responsible and professional) oversight. This demonstrates dangerous disdain for integrity of functional assessment and denies the clinical reality that the IA has not been researched or evaluated for fitness-of-purpose. A plan to do a “progressive evaluation” while acknowledging problems with validity (accuracy) and reliability (consistency), also demonstrates a willingness to take short-cuts on best practice, and a willingness to mandate thousands of people with disabilities to undergo an unproven assessment process.

Best practice

OTA describes best practice functional capacity assessment as more detailed, in order to build an accurate picture. It should involve a combination of things – self-report tools, observational tools, clinical reasoning and interpretation by skilled therapists, and the inclusion of carer, participant, and existing provider perspectives and cultural considerations. The NDIA regularly seek these best-practice occupational therapy functional capacity assessments for their legal purposes, such as in preparation for an Administrative Affairs Tribunal (AAT) hearing. These assessments are in stark contrast to the IA, which relies almost completely on self-report tools. The IAs run a high risk of under or over-rating functional capacity and are too simplistic.

Calculating the participant budget using the assessment scores

Importantly, we cannot provide an evaluation of the NDIA process for translating IA scores into a participant budget, because the details of how this will happen has not been published by the NDIA. It will likely include weighting or tallying of assessment scores – we can only guess. What is becoming more clear, is that the IA is an ‘input’ into what is likely an automated or artificial intelligence-driven system that creates participant budgets based on persona’s, or stereotypes. The use of automation or artificial intelligence has led to the disability sector and media naming the mechanism ‘robo-NDIS’ and ‘robo-planning’. Individual support needs are not considered in this process.

A few weeks into a parliamentary inquiry and still no transparency on how proposed #independentassessments will be translated into NDIS participant budgets. What’s in the ‘ black box’?? An unproven weighting of assessments? An algorithm? A bunny rabbit? pic.twitter.com/7QcdVoPCZn

— muriel cummins (@muriel_cummins) May 14, 2021

Back to the drawing board

The National Disability Insurance Scheme may be older now, but it is arguably less mature. Occupational therapists join participant and carer groups, advocacy groups, and the disability sector broadly, in asking for a return to the drawing board to co-design method for robust functional capacity assessments and creating participant budgets. It must both meet the needs of NDIS participants and the future sustainability of the NDIS. There are many ways to approach this that would be more accurate and inclusive than what is currently planned.

It is disappointing to discover via leaked marketing documents that NDIA views stakeholders, including allied health peak bodies, as risks to be managed. This approach is the opposite of collaboration and co-design. Building co-designed best-practice approaches can only happen if the NDIA undertakes genuine, inclusive consultation with stakeholders.

We need to return to the basic principles underpinning the NDIS. Without re-commitment to basic principles, we can’t progress. We can’t rebuild multi-partisan support – or make legislative changes that will sustain the NDIS into the future. Without agreed core principles, the NDIS will not reach its potential, and will deny people with disabilities the opportunity to reach theirs.

And it is unthinkable that we may wake one morning in late 2021, to find the Sydney Opera House has been replaced by something dark and dated, with a 1940’s brick veneer.

Muriel Cummins is an occupational therapist and member of Occupational Therapy Australia’s NDIS Taskforce.

Occupational Therapy Australia’s answer to the Question on Notice

Download the PDF

Ms Bonnie Allan
Committee Secretary
Joint Standing Committee on the NDIS

Via email:

21 May, 2021

Dear Ms Allan

Re: Occupational Therapy Australia response to question taken on notice at public hearing in Melbourne on 23 April 2021

Occupational Therapy Australia (OTA) thanks the Joint Standing Committee on the National Disability Insurance Scheme (NDIS) for the opportunity to appear before the Committee on 23 April 2021, and to respond to the following question from Senator Carol Brown taken on notice at the hearing:

Do you see the tools that have been selected by the NDIA – such as the WHODAS and PEDI-CAT – as being appropriate measurements to inform a funding decision in a NDIS plan?

OTA holds a clear position that people with a disability in Australia have a right to an evidence-based, robust, and safe process for assessment of functional capacity to determine access to the NDIS, and to determine funding for reasonable and necessary supports. For this reason, OTA welcomes Senator Brown’s question regarding the appropriateness of the tools selected by the NDIA to inform NDIS plans and funding decisions.

OTA has considered the relative merits and limitations of the Independent Assessment (IA) measurement tools proposed to inform funding decisions and NDIS plans. This includes close examination of their relevance, utility, and psychometric properties (WHO 2020).

While these tools have reasonably sound intrinsic measurement properties when used for the purpose for which they were specifically designed, their reliability and validity is profoundly compromised when they are used for other purposes. OTA does not believe the NDIA is using the measures for the purpose for which they were intended. These tools were not designed to specifically assess functional capacity to inform funding decisions or plans, and they lack sufficient relevance, sensitivity or specificity to be used in this way.

OTA offers the following specific observations in response to Senator Brown’s question.

Are these tools designed to sufficiently inform funding for NDIS plans?

The IA toolkit is primarily based on the use of global self-report measures related to levels of health and disability in the general population. They have reasonably sound measurement properties when they are used for their intended purpose. However, they are not designed as functional assessment tools and do not provide sufficient detail to effectively determine individual functional capacity or support needs.

A functional assessment identifies what the person can and can’t do, due to disability-related impairment. A support needs assessment identifies what the person needs to address the impairment, so that they can capacity-build, or compensate for the impairment, thereby reducing the impact of the disability on their ability to participate in their lives and community. Delivering standard plans and budgets, sometimes known as ‘roboplans’, based on an IA provides limited detail and effectively bypasses the essential step of individual support needs identification. By neglecting disability support needs, the NDIA runs the real risk of rendering the assessment process more costly in the long term, both in terms of human suffering and dollar amounts. OTA recommends support-needs identification be an essential step in the determination of participant plans and the budgets that support them.

Are the IA standardised assessment tools INDIVIDUALLY appropriate to inform funding decisions for NDIS plans?

The individual assessment tools included within the pilot IA, are robust tools when used for their intended purposes. These are summarised in Table 1. None, however, was developed for the purpose of determining NDIS participant plan budgets. Each tool has evidence-based validity when used with the particular cohorts they were designed for. However, they cease to be valid when they are used with other cohorts. Apart from the WHODAS-2, none was developed to be used in a ‘disability neutral’ manner.

The WHODAS-2 is a global measure of based on the conceptual framework of the International Classification of Functioning, Disability, and Health (ICF). While the WHODAS- 2 can be used across all disability cohorts, it is a functional screen only, not a functional capacity assessment (Ustun et al 2010). For example, it is entirely possible for a person to have substantially reduced capacity for self-care, and for this not to register on the WHODAS-2.

Attempting to use specific IA tools in a disability neutral manner, is problematic in practice. For example, the Vineland 2, a measure of adaptive functioning, has proven validity when used with people who live with developmental disability, such as autism, intellectual disability, or attention-deficit hyperactivity disorder. The Vineland 2 does not have proven validity to assess people with degenerative conditions, or those who live with psychosocial disability where capacity fluctuates over time. The NDIA intends the Vineland-2 to be used with all NDIS applicants and participants, including cohorts the tool has not been validated for. The current IA pilot espouses this practice. The company publishing the Vineland-2 is aware of the limits of its proven validity, and states that the ‘burden of proof’ for appropriate use of the tool sits with the NDIA and NDIA-contracted companies. So, we have a situation whereby participants are undergoing a lengthy, deficit-focused assessment, which is not a valid tool to use with a large proportion of those participants and which will result in flawed scores being used to inform funding decisions in NDIS plans.