VOCA

An aviation researcher, writer, aviation participant, pilot & agricultural researcher. Author of over 35 scientific publications world wide.

Categories

Good reads

Oil Prices

#ozaviation

#casa KPI’s have no means of measurement

I followed through onto the #casa web site to look at the recently [July, 2017] released corporate plan.

In reading the document, with it’s key section, the measurement of performance, come to the distinct conclusion that #casa must not continue as there is no measurable metric that is given.

From KlipFolio, a metric, in measurement is:

Business Metric is a quantifiable measure that is used to track and assess the status of a specific business process. It’s important to note that business metrics should be employed to address key audiences surrounding a business, such as investors, customers, and different types of employees, such as executives and middle managers.

Every area of business has specific performance metrics that should be monitored – marketers track marketing and social media metrics, such as campaign and program statistics, sales teams monitor sales performance metrics such as new opportunities and leads, and executives look at big picture financial metrics.

Business Metric or Key Performance Indicator? What’s the Difference?

To be effective, business metrics should be compared to established benchmarks or business objectives. This provides valuable context for the values used in the metric and allows business users to better act on the information they are viewing. For instance, $20M sales in Q4 sounds like an impressive figure; however, if you’re Boeing Aircraft, this figure would have you contemplating filing for bankruptcy.

Context allows business metrics to make an impact. In fact, this is where the line between key performance indicators and performance metrics becomes blurry. The difference between the two ultimately comes down this:

  • Business metrics are used to track all areas of business
  • KPIs target critical areas of performance

For example, a metric may monitor website traffic compared to a goal, whereas a KPI would monitor how website traffic contributed to incremental sales.

In the Governments programme, it is specified that:

“there is (sic)………..an appropriate benchmark for regulator performance……………….”

In the KPI’s that #casa specify below, there is no specified metric and the KPA’s and KPI’s cannot be measured. This is a serious deficiency in a pivotal requirement in spending taxpayer funds, or funds drawn fromthe public, in #casa’s case, via excise on AVTUR and AVGAS.

In fact the Colmar Brunton survey, [Part 2 – action plan] which is the first and an independent survey into #casa gave a very desultory view of the regulator.

 

Main results from Colmar Brunton survey

Reading:

  1. #casa Corporate Plan 2015;
  2. #casa 2017 released corporate plan;
  3. Colmar Brunton report on #casa “Health”;
  4. Main Colmar Brunton Action Plan

In the 2017 Corporate Plan #casa say:

CASA will be reporting against five Key Performance Areas (KPAs) in this plan using performance indicators and measures in accordance with the requirements detailed in the Regulator Performance Framework and the PGPA Rule.

These KPAs are:

Aviation safety regulation and service delivery

  1. Industry oversight
  2. Stakeholder engagement
  3. Governance and organisational effectiveness
  4. CASA’s capability and capacity

CASA will measure its success against these KPAs utilising the following 10 Key Performance Indicators (KPIs):

1. CASA is an effective aviation safety regulator by international standards
2. CASA does not unnecessarily impede the efficient operation of regulated entities *
3. CASA actively contributes to the continuous improvement of the aviation safety regulatory framework *
4. Actions undertaken by CASA are proportionate to the risk being managed *
5. Compliance and monitoring approaches are streamlined and coordinated *
6. CASA is open and transparent in its dealings with regulated entities and all of its stakeholders *
7. Communication with regulated entities and all of CASA’s stakeholders is clear, targeted and effective*
8. Education and promotion deliverables are relevant, timely, effective and appropriately targeted
9. CASA’s governance structures, finance and risk management align with Commonwealth better practice
10. CASA maintains the capability and capacity to effectively deliver aviation safety regulation.

* KPIs required under the Regulator Performance Framework.


Regulator Performance Framework requirements

USING THE KPIs – BETTER PRACTICE

The descriptions accompanying each KPI are intended to demonstrate ways that a regulator may be successfully achieving the KPI, to assist reviewers in formulating an appropriate benchmark for regulator performance. The description of the better practice principles will also help regulators as a guide to future better practice.

MEASURES

The measures of good regulatory performance used in the Framework outline the principles that all regulators should be using to guide the collection of evidence and for review to assess achievement of the KPIs. Whilst not necessarily an exhaustive list, the suggested measures are considered sufficient to enable assessment against the KPIs. It is expected that tailored measures of good regulatory performance, based on these high-level measures, would be adopted to enable comprehensive review of individual regulators and their specific tasks and role.

EXAMPLES OF EVIDENCE

The suggested examples of output/activity-based evidence are a starting point for reviewers to determine the evidence that will be used in assessing performance of a regulator. Reviewers should ensure that: all areas considered relevant by the stakeholders are included in the review (to prevent regulators adapting practices to meet indicators whilst neglecting areas that are more difficult to observe) and multiple sources of evidence are used to assess each measure on performance and areas for improvement.


From the document directing how #casa should measure these issues:

Measures of good regulatory performance:

1. Regulators provide guidance and information that is up to date, clear, accessible and concise through media appropriate to the target audience.
2. Regulators consider the impact on regulated entities and engage with industry groups and representatives of the affected stakeholders before changing policies, practices or service standards.
3. Regulators’ decisions and advice are provided in a timely manner, clearly articulating expectations and the underlying reasons for decisions.
4. Regulators’ advice is consistent and supports predictable outcomes.

#casa’s score on this basis:

Item 1: 2/10

Item 2: 1/10

Item 3: 0/10

Item 4: 0/10

Yes I don’t have much faith in #casa as readers know, but to release a “….Corporate Plan…” with no KPI’s just leads to a low score.

AND

Examples given to guide the #casa Board and #casa “Corporate”

Examples of output / activity-based evidence

  1. Percentage of guidance materials that complies with government accessibility guidelines.
  2. Maximum, minimum and average time for decision.
  3. Published timeframes for decision making.
  4. Percentage of decisions accompanied by statement of reasons and advice about relevant review or appeal mechanisms, where appropriate.
  5. Number of policy/standards changes which are preceded by comprehensive engagement with stakeholders.
  6. Approved procedures for communications (including issue-specific scripts if relevant) are available for staff use when interacting with regulated entities.
  7. Advice provided to regulated entities is consistent with communication policies
  8. Demonstrated feedback is sought from stakeholders on guidance and advice provided by the regulator via a wide range of mechanisms, including stakeholder surveys.
  9. Demonstrated mechanisms for responding to stakeholder engagement/complaint

Nothing from #casa like this in their listed KPI’s.