Assessing Quality Systems



Yüklə 469 Kb.
tarix31.08.2018
ölçüsü469 Kb.
#65611


Assessing Quality Systems

  • An Overview


Course Objectives

  • At the conclusion of this course, participants will be able to:

    • Discuss
      • the purpose of Quality Systems Assessments (QSA)
      • the benefits of conducting QSAs
      • how QSAs differ from other types of assessments
    • Demonstrate a process for planning, conducting, and reporting the results of QSAs


Quality Systems Assessment

  • WHAT

    • a systematic, independent, and documented examination that uses specified criteria to assess quality systems
  • WHY

    • EPA Order 5360.1
    • 48 CFR Part 46 and 40 CFR Parts 30, 31, and 35


Assessment vs. Audit

  • Assessment

    • the evaluation process used to measure the performance or effectiveness of a system and its elements
  • Audit

    • a systematic and independent examination to determine whether quality activities comply with specifications and whether these specifications are implemented effectively




QSA Perception Check

  • True or False

  • Planning and conducting a QSA requires no expertise in an organization's activities and how it operates

  • QSA results will tell management where to look to uncover problems (e.g., performance issues, fraud)

  • A QSA can serve as a key deterrent to organizational improprieties

  • New organizations don’t need an assessment of the quality system

  • A QSA can indicate the effectiveness of communicating quality policies in an organization



QSA Benefits

  • QSA results can be indicators of the:

  • Adequacy and implementation of programs developed to achieve an organization’s mission

  • Quality of products and services

  • Compliance with governing specifications



Practice What You Learned

  • For each scenario, determine:

  • What type of assessment is appropriate (system, project/program, process)?

  • Should the assessment be internal or external?

  • What benefits may be achieved from the assessment?

  • A recent grantee has water sampling scheduled through a QAPP that the funding organization has just approved. Their QMP had few details, and it is suspected that they have inadequate processes for overseeing the work.

  • A large organization, with many divisions and several QA staff have recently amended their QMP to change the way they delegate QAPP approval.

  • An organization had been subject to regular assessments. Each set of assessment results contains the same findings for insufficient documentation of the quality system. However, improvement in the project management QA responsibilities is noted.



Assessment Framework



The Assessment Cast

  • Authorizing Entity

  • Assessee

  • Assessment Team Leader

  • Assessors



Authorizing Entity

  • Roles and Responsibilities

  • Authorize and provide resources for the assessment

  • May set the scope of the assessment

  • Review assessment results and ensure corrective actions



The Assessee

  • Roles and Responsibilities

  • Acknowledge the QSA communications

  • Work with the assessor to accomplish the QSA



Assessment Team Leader

  • Roles and Responsibilities

  • Select team members

  • Prepare the assessment plan

  • Complete and submit the assessment report

  • Address comments to the report



The Assessors

  • Roles and Responsibilities

  • Identify and accurately document

    • non-compliance
    • noteworthy practices
  • Review and comment on draft reports to ensure accuracy



Assessment Planning



Initial Planning

  • The initial planning process involves:

    • gaining knowledge and understanding of the assessee organization
    • determining the lines of assessment inquiry
    • formalizing assessment objectives
    • considering available resources


Initial Planning Steps

  • Determine criteria for the assessment

  • Determine the assessment scope and available resources

  • Select the assessment team



Initial Planning Steps Determine Assessment Criteria

  • Assessment criteria can be set by policy or directives:

    • Quality System Specifications
      • Order 5360.1
      • ANSI/ASQ E-4
    • Assessee specifications/policy
      • the assessee’s Quality Management Plan
      • assessee reports
    • Regulatory requirements
      • contracts and assistance agreements
      • program specific requirements


Initial Planning Steps Determine the Assessment Scope

  • The scope is set by the:

    • directive of the authorizing entity
    • allocated resources
    • onsite considerations


Initial Planning Steps Select the Assessment Team

  • Select team members that:

    • have appropriate skills and knowledge
    • are competent and credible
    • have no conflict of interest


Documenting the Assessment Process

  • The Assessment File contains:

    • planning documents
    • relevant correspondence
    • working papers
    • assessment reports


Documenting the Assessment Process Tracking Tools

  • HOW

    • Tools for managing document flow and status from planning, observations and actions taken, and filing.
      • Paper notation
      • Electronic data base
  • WHY

    • Papers flow through many channels
    • A need to effectively manage assessment information


Assessment Communications

  • Initial contact with the assessee to begin scheduling

  • Approval of the assessment plan by the authorizing entity

  • Formally notify the assessee of the planned assessment

  • Reporting, tracking corrective actions, close-out



Case Study – Part I

  • Review the Case Study

  • Appoint a spokesperson

  • Complete the Initial Planning Worksheet



Detailed Planning

  • Whatever failures I have known,

  • Whatever errors I have committed,

  • Whatever follies I have witnessed in private and public life Have been the consequence of action without thought.

  • Bernard Baruch, stock broker, advisor to presidents Woodrow Wilson, Harry S. Truman,



Detailed Planning

  • Information Review

  • Issue Selection

  • Identifying Interviewees and Onsite Documentation



Information Review

  • Review the organization’s quality system

    • Quality Management Plan
    • Applicable regulations for environmental programs
    • Reports from previous assessments
    • Quality Assurance Annual Report and Work Plan/Fiscal reports
    • Published products
    • Information about personnel (chart, CVs)


Information Review cont’d

  • What can the documentation tell you?

    • QMP
      • Mission/quality policy
      • QA support/resources
      • Specifications of the quality system- roles (e.g., description of programs and products)
    • Regulations
      • QA context for work
    • Previous results
      • Prior problems
    • Annual reports
      • Resources, activities, plans
    • Fiscal reports
      • relative importance
    • Published products
      • conformance, status
    • Staff information
      • qualifications, experience


Issue Selection

  • Items that help to define/select assessment issues:

    • assessment issues issues pre-defined by the authorizing entity.
    • quality system processes affecting costly or important products
    • quality system specifications having a major effect on environmental data operations
    • processes appearing difficult or expensive to implement
    • recently implemented policies and procedures
    • processes having been problematic for similar organizations


For Example

  • Roles and responsibilities from the QMP

    • The Envirumdata Group, composed of 4 programmers and 2 data entry professionals, is responsible for ascertaining the quality of all analytical data entered into the organization’s database
  • Specification of the quality system

    • All data (collected by other organizations) in the database must be assessed prior to use
  • Relevance of specification to the program

    • Data are used in support of rule making


Resulting Questions/Issues

    • What is the process for data quality assessment?
    • How is the process documented?
    • How are the results documented?
    • Do they have adequate expertise to assess data quality?


Tips for Issue Selection

  • Look for:

  • incomplete and unclear quality system documentation

  • processes that seem too extensive for the organization’s size and mission

  • processes that are not clearly described either in full or by reference

  • end products that lack review and approval processes

  • issues that pass the “So What” test



Where do you draw the line??



QSA Data Gathering

  • Primary QSA data gathering tools

    • Interviews
      • Interaction with the interviewee provides the team with direct information about the quality system.
    • Document reviews
      • Supporting documentation can be consulted and questions explained and clarified as necessary.
  • Remember: The right people need to be asked the right questions.



QSA Data Gathering Select the Right Documents and the Right People

  • Define what documents contain information relevant to the selected issues. Include information that:

    • may be requested prior to the onsite
    • will be available during the onsite review
  • Identify the appropriate interviewees for the selected issues

    • usually those who dictate the specifications for, create, contribute to, review, approve or use the documentation.


QSA Data Gathering Get the Right Type and Number

  • Selection considerations:

    • size of organization
    • primary functions
    • personnel factors, including availability, experience, knowledge of issues selected, length of time in position, number in each job category
    • document factors, including importance of work products, size of document pool, appropriateness to the time period and process being assessed.


Think About It…

  • Your assessment plan instructs you to ask 26 people whether they are familiar with their organization’s QMP. What assessment finding could you make?

  • You review QA documentation processes and understand the specifications (what is required, roles and responsibilities for preparation, review, approval, and record keeping according to the QMP). You then examine 26 examples and ask interviewees about the process they followed. What assessment finding could you make?



LUNCH!



CASE STUDY PART II



AFTER the PLAN is Done……

  • Assessment plan approval

  • Confidentiality considerations

  • Assessment tools



Review and Approval of the Assessment Plan

  • Who?

    • Your management
    • Authorizing entity
  • Why?

    • Provides an opportunity to raise questions about the approach
    • Approval of the plan affirms the authority, credibility and scope of the assessment process
    • Encourages buy-in and ownership


Confidentiality Considerations

  • Dissemination of assessment findings “generally only to involved parties.”

  • Interview notes are discarded when the report is final

  • CBI – important in RCRA, CAA, FIFRA



Assessment Tools Checklists

  • Used to:

    • organize interviews and document review
    • assure completeness and consistency in reviews
    • promote objectivity among assessors
  • Design the checklist to:

    • fill information gaps in Quality System information
    • verify Quality System specifications
    • address specific observable practices


Assessment Tools Matrix

  • Suited for complex assessments

  • Serves as a visual check for the assessment team; shows where noncompliance may be clustered or how wide spread a practice may be



Ready, Set………………

  • Just as there are three R's there are also three A's of business life. They are: Ability, Ambition, and Attitude. Ability establishes what a worker does and will bring him a paycheck. Ambition determines how much he does and will get him a raise. Attitude guarantees how well he does.

  • Wilbert Sheer



Conducting the Assessment

  • Logistics

  • Onsite Activities

  • Reporting



Logistics

  • BEFORE YOU GET THERE, CONFIRM:

    • Building access
    • Assessment schedule/issues
      • Opening and closing Meetings
      • Interview schedules
      • Assessment team meeting times
      • Availability of meeting rooms/work space
      • File review access
  • TO GET THERE

    • Travel (arrival and departure) itinerary for all team members
    • Lodging (phone, address)
    • Local city transportation (maps, taxi rates)


The Onsite

  • Conduct:

  • The Opening Meeting

  • The Onsite Review

    • Documents and Records Review
    • Interviews


Opening Meeting

  • Typical Format

    • Introductions
    • Brief description of the scope of the assessment
    • Explain the assessment team’s schedule
    • Answer any questions regarding the assessment
    • Limit time to one hour or less


The Onsite Review

  • Document and Records Review

    • Review documents listed in the assessment plan
    • Focus on documents that verify policies and processes
    • Verify supplemental information (e.g., databases, archived information)
    • With permission, make copies of relevant information


Interviews

  • Elements of the interview

    • Planning
    • Opening
    • Conducting
    • Closing


Planning

  • Establish information needs

    • What information is needed
    • Who needs to be interviewed
  • Format assessment questions



Types of Questions

  • Open ended

  • Close ended

    • Questions that offer a set of alternative answers (e.g., yes or no, true or false, scaled answers)


Formulating Questions



Tips

  • Questions should be:

    • as neutral as possible
    • asked one at a time
    • worded clearly
  • Use "why" questions sparingly



Interview Scheduling

  • Scheduling considerations

    • Schedule in advance
    • Plan no more than one hour for each interview
    • One on one interviews (preferred)
    • Try to avoid the hour right before or after lunch


Opening the Interview

  • Sets tone for the interview

    • Make introductions
    • Keep small talk to a minimum
    • Re-state the purpose of the interview and how you will proceed


Opening the Interview cont’d

  • Tell them:

    • who will get access to their answers and how their answers will be analyzed
    • the format of the interview
    • how long the interview usually takes
    • how to get in touch with you after the interview
    • that you will take notes during the interview


Note Taking Tips

  • Use a notepad

  • Record main ideas

  • Periodically read back notes to check for accuracy

  • Notes are about PROCESSES, not PEOPLE



Conducting the Interview

  • Remain as neutral as possible

  • Encourage responses

  • Control your behaviors when note taking

  • Provide transition between major topics

  • Don't lose control of the interview



Closing the Interview

  • Briefly summarize your information

  • Thank the interviewee for his/her time

  • Ask if the interviewee would be willing to follow up with you if necessary



PRACTICE WHAT YOU LEARNED

  • The character of the questions you ask will influence the appropriateness of the answers.

  • You ask the interviewee about his interaction with the QA Manager and he immediately displays resentment. You ask….. "So I guess you don’t like him much, huh?"

    • You should ask……….
  • You ask a database operator ….. "Do you know how to work this thing?"

    • You should ask……………………..


PRACTICE WHAT YOU LEARNED

  • One More for Good Measure!

  • You ask a field sampler…. "Have you been following this SOP?“

    • You should ask……………………….


Think About It

  • Based on your draft assessment plan, formulate 2 questions for each of the following categories:

  • Are Quality System policies implemented by staff and management?

  • Are the Quality System policies effective?

  • Do personnel understand the Quality System requirements?



Working Papers

  • Assessment Working Papers

    • Notes from document review
    • Copies of documents (with permission!)
    • Interview notes
  • Cross reference to criteria

  • Date each document



Now What???

  • Meet with your team each day to review working papers

  • Develop preliminary findings



Finding



Other Assessment Results

  • Noteworthy practices

    • Positive practices implemented by the organization
  • Observations

    • Practices noted during the review. No corrective action required.


Example: Finding 1

  • The QA Coordinators need training in QA concepts and theories



Example: Revised Finding 1

  • The QA Coordinators are unable to perform the review and approval of QAPPs as specified in the QMP because they have not been trained in the process.



Draft Preliminary Findings

  • Presented to summarize what you have observed during the onsite

  • Observations should be factual and to the point

  • Findings can change during follow up review

    • (HINT - You’re not finished!)


Exercise!



Closing Meeting

  • Brief management on your preliminary findings

  • Explain your reporting process and timetable

  • Remember:

    • No Surprises!
    • Preliminary findings must not be frivolous
    • Keep the meeting to no more than 1 hour
    • Try to address immediate concerns but DON’T
      • Make promises you can’t keep
      • Usurp your management’s authority


The Black Hole Phenomenon



The Report

  • The assessment report communicates assessment findings to management. It should clearly identify:

    • the type of assessment,
    • the assessor,
    • the assessee,
    • what was assessed,
    • the findings,
    • if requested by the authorizing entity, recommendations for corrective actions


Effective Reporting

  • Clear and concise writing

  • Address significant deficiencies in a constructive manner

  • Report the organization’s actions, not the actions of specific individuals

  • Prioritize assessment findings and any need for corrective actions



Example Report Outline

  • Background and Purpose

  • Assessment Scope

  • Assessment Results and Rationale

  • References

  • Appendices

    • A. List of Personnel Interviewed
    • B. List of Documents and Records Reviewed (optional)
    • C. Corrective Action Plan


Close Out



Close Out

  • BEFORE you can close the assessment you need:

    • A response from the assessed organization with an acceptable corrective action plan (if necessary) and an indication that the corrective actions have been completed
    • A review of the assessment file to ensure that it is complete before it is archived.


SUMMARY

  • Now you can discuss:

    • the purpose and benefits of Quality Systems Assessments (QSAs)
    • how QSAs differ from other types of assessments
  • You can demonstrate a process for planning, conducting, and reporting the results of QSAs

    • I can give you a six-word formula for success: "Think things through - then follow through."
    • Edward Rickenbacker


Yüklə 469 Kb.

Dostları ilə paylaş:




Verilənlər bazası müəlliflik hüququ ilə müdafiə olunur ©genderi.org 2024
rəhbərliyinə müraciət

    Ana səhifə