Course Objectives At the conclusion of this course, participants will be able to: - Discuss
- the purpose of Quality Systems Assessments (QSA)
- the benefits of conducting QSAs
- how QSAs differ from other types of assessments
- Demonstrate a process for planning, conducting, and reporting the results of QSAs
Quality Systems Assessment WHAT - a systematic, independent, and documented examination that uses specified criteria to assess quality systems
WHY - EPA Order 5360.1
- 48 CFR Part 46 and 40 CFR Parts 30, 31, and 35
Assessment vs. Audit Assessment - the evaluation process used to measure the performance or effectiveness of a system and its elements
Audit - a systematic and independent examination to determine whether quality activities comply with specifications and whether these specifications are implemented effectively
QSA Perception Check True or False Planning and conducting a QSA requires no expertise in an organization's activities and how it operates QSA results will tell management where to look to uncover problems (e.g., performance issues, fraud) A QSA can serve as a key deterrent to organizational improprieties New organizations don’t need an assessment of the quality system A QSA can indicate the effectiveness of communicating quality policies in an organization
QSA Benefits QSA results can be indicators of the: Adequacy and implementation of programs developed to achieve an organization’s mission Quality of products and services Compliance with governing specifications
Practice What You Learned For each scenario, determine: What type of assessment is appropriate (system, project/program, process)? Should the assessment be internal or external? What benefits may be achieved from the assessment? A recent grantee has water sampling scheduled through a QAPP that the funding organization has just approved. Their QMP had few details, and it is suspected that they have inadequate processes for overseeing the work. A large organization, with many divisions and several QA staff have recently amended their QMP to change the way they delegate QAPP approval. An organization had been subject to regular assessments. Each set of assessment results contains the same findings for insufficient documentation of the quality system. However, improvement in the project management QA responsibilities is noted.
Assessment Framework
The Assessment Cast Authorizing Entity Assessee Assessors
Authorizing Entity Roles and Responsibilities Authorize and provide resources for the assessment May set the scope of the assessment Review assessment results and ensure corrective actions
The Assessee Roles and Responsibilities Acknowledge the QSA communications Work with the assessor to accomplish the QSA
Assessment Team Leader Roles and Responsibilities Select team members Prepare the assessment plan Complete and submit the assessment report Address comments to the report
The Assessors Roles and Responsibilities Identify and accurately document - non-compliance
- noteworthy practices
Review and comment on draft reports to ensure accuracy
Assessment Planning
Initial Planning The initial planning process involves: - gaining knowledge and understanding of the assessee organization
- determining the lines of assessment inquiry
- formalizing assessment objectives
- considering available resources
Initial Planning Steps Determine criteria for the assessment Determine the assessment scope and available resources Select the assessment team
Initial Planning Steps Determine Assessment Criteria Assessment criteria can be set by policy or directives: - Quality System Specifications
- Order 5360.1
- ANSI/ASQ E-4
- Assessee specifications/policy
- the assessee’s Quality Management Plan
- assessee reports
- Regulatory requirements
- contracts and assistance agreements
- program specific requirements
Initial Planning Steps Determine the Assessment Scope The scope is set by the: - directive of the authorizing entity
- allocated resources
- onsite considerations
Initial Planning Steps Select the Assessment Team Select team members that: - have appropriate skills and knowledge
- are competent and credible
- have no conflict of interest
Documenting the Assessment Process The Assessment File contains: - planning documents
- relevant correspondence
- working papers
- assessment reports
Documenting the Assessment Process Tracking Tools HOW - Tools for managing document flow and status from planning, observations and actions taken, and filing.
- Paper notation
- Electronic data base
WHY - Papers flow through many channels
- A need to effectively manage assessment information
Assessment Communications Initial contact with the assessee to begin scheduling Approval of the assessment plan by the authorizing entity Formally notify the assessee of the planned assessment
Case Study – Part I Review the Case Study Appoint a spokesperson Complete the Initial Planning Worksheet
Detailed Planning Whatever failures I have known, Whatever errors I have committed, Whatever follies I have witnessed in private and public life Have been the consequence of action without thought. Bernard Baruch, stock broker, advisor to presidents Woodrow Wilson, Harry S. Truman,
Detailed Planning Information Review Issue Selection Identifying Interviewees and Onsite Documentation
Information Review Review the organization’s quality system - Quality Management Plan
- Applicable regulations for environmental programs
- Reports from previous assessments
- Quality Assurance Annual Report and Work Plan/Fiscal reports
- Published products
- Information about personnel (chart, CVs)
Information Review cont’d What can the documentation tell you? - QMP
- Mission/quality policy
- QA support/resources
- Specifications of the quality system- roles (e.g., description of programs and products)
- Regulations
- Previous results
- Annual reports
- Resources, activities, plans
- Fiscal reports
- Published products
- Staff information
- qualifications, experience
Issue Selection Items that help to define/select assessment issues: - assessment issues issues pre-defined by the authorizing entity.
- quality system processes affecting costly or important products
- quality system specifications having a major effect on environmental data operations
- processes appearing difficult or expensive to implement
- recently implemented policies and procedures
- processes having been problematic for similar organizations
For Example Roles and responsibilities from the QMP - The Envirumdata Group, composed of 4 programmers and 2 data entry professionals, is responsible for ascertaining the quality of all analytical data entered into the organization’s database
Specification of the quality system - All data (collected by other organizations) in the database must be assessed prior to use
Relevance of specification to the program - Data are used in support of rule making
Resulting Questions/Issues - What is the process for data quality assessment?
- How is the process documented?
- How are the results documented?
- Do they have adequate expertise to assess data quality?
Tips for Issue Selection Look for: incomplete and unclear quality system documentation processes that seem too extensive for the organization’s size and mission processes that are not clearly described either in full or by reference end products that lack review and approval processes issues that pass the “So What” test
Where do you draw the line??
QSA Data Gathering Primary QSA data gathering tools - Interviews
- Interaction with the interviewee provides the team with direct information about the quality system.
- Document reviews
- Supporting documentation can be consulted and questions explained and clarified as necessary.
Remember: The right people need to be asked the right questions.
QSA Data Gathering Select the Right Documents and the Right People Define what documents contain information relevant to the selected issues. Include information that: - may be requested prior to the onsite
- will be available during the onsite review
Identify the appropriate interviewees for the selected issues - usually those who dictate the specifications for, create, contribute to, review, approve or use the documentation.
-
QSA Data Gathering Get the Right Type and Number Selection considerations: - size of organization
- primary functions
- personnel factors, including availability, experience, knowledge of issues selected, length of time in position, number in each job category
- document factors, including importance of work products, size of document pool, appropriateness to the time period and process being assessed.
Think About It… Your assessment plan instructs you to ask 26 people whether they are familiar with their organization’s QMP. What assessment finding could you make? You review QA documentation processes and understand the specifications (what is required, roles and responsibilities for preparation, review, approval, and record keeping according to the QMP). You then examine 26 examples and ask interviewees about the process they followed. What assessment finding could you make?
LUNCH!
CASE STUDY PART II
AFTER the PLAN is Done…… Assessment plan approval Confidentiality considerations Assessment tools
Review and Approval of the Assessment Plan Who? - Your management
- Authorizing entity
Why? - Provides an opportunity to raise questions about the approach
- Approval of the plan affirms the authority, credibility and scope of the assessment process
- Encourages buy-in and ownership
Confidentiality Considerations Dissemination of assessment findings “generally only to involved parties.” Interview notes are discarded when the report is final CBI – important in RCRA, CAA, FIFRA
Assessment Tools Checklists Used to: - organize interviews and document review
- assure completeness and consistency in reviews
- promote objectivity among assessors
Design the checklist to: - fill information gaps in Quality System information
- verify Quality System specifications
- address specific observable practices
Assessment Tools Matrix Suited for complex assessments Serves as a visual check for the assessment team; shows where noncompliance may be clustered or how wide spread a practice may be
Ready, Set……………… Just as there are three R's there are also three A's of business life. They are: Ability, Ambition, and Attitude. Ability establishes what a worker does and will bring him a paycheck. Ambition determines how much he does and will get him a raise. Attitude guarantees how well he does. Wilbert Sheer
Conducting the Assessment Logistics Onsite Activities Reporting
Logistics BEFORE YOU GET THERE, CONFIRM: - Building access
- Assessment schedule/issues
- Opening and closing Meetings
- Interview schedules
- Assessment team meeting times
- Availability of meeting rooms/work space
- File review access
TO GET THERE - Travel (arrival and departure) itinerary for all team members
- Lodging (phone, address)
- Local city transportation (maps, taxi rates)
The Onsite Conduct: The Opening Meeting The Onsite Review - Documents and Records Review
- Interviews
Opening Meeting Typical Format - Introductions
- Brief description of the scope of the assessment
- Explain the assessment team’s schedule
- Answer any questions regarding the assessment
- Limit time to one hour or less
The Onsite Review Document and Records Review - Review documents listed in the assessment plan
- Focus on documents that verify policies and processes
- Verify supplemental information (e.g., databases, archived information)
- With permission, make copies of relevant information
Interviews Elements of the interview - Planning
- Opening
- Conducting
- Closing
Planning Establish information needs - What information is needed
- Who needs to be interviewed
Format assessment questions
Types of Questions Open ended Close ended - Questions that offer a set of alternative answers (e.g., yes or no, true or false, scaled answers)
Formulating Questions
Tips Questions should be: - as neutral as possible
- asked one at a time
- worded clearly
Use "why" questions sparingly
Interview Scheduling Scheduling considerations - Schedule in advance
- Plan no more than one hour for each interview
- One on one interviews (preferred)
- Try to avoid the hour right before or after lunch
Opening the Interview Sets tone for the interview - Make introductions
- Keep small talk to a minimum
- Re-state the purpose of the interview and how you will proceed
Opening the Interview cont’d Tell them: - who will get access to their answers and how their answers will be analyzed
- the format of the interview
- how long the interview usually takes
- how to get in touch with you after the interview
- that you will take notes during the interview
Note Taking Tips Use a notepad Record main ideas Periodically read back notes to check for accuracy Notes are about PROCESSES, not PEOPLE
Conducting the Interview Remain as neutral as possible Encourage responses Provide transition between major topics Don't lose control of the interview
Closing the Interview Briefly summarize your information Thank the interviewee for his/her time Ask if the interviewee would be willing to follow up with you if necessary
PRACTICE WHAT YOU LEARNED The character of the questions you ask will influence the appropriateness of the answers. You ask the interviewee about his interaction with the QA Manager and he immediately displays resentment. You ask….. "So I guess you don’t like him much, huh?" You ask a database operator ….. "Do you know how to work this thing?"
PRACTICE WHAT YOU LEARNED One More for Good Measure! You ask a field sampler…. "Have you been following this SOP?“
Think About It Based on your draft assessment plan, formulate 2 questions for each of the following categories: Are Quality System policies implemented by staff and management? Are the Quality System policies effective? Do personnel understand the Quality System requirements?
Working Papers Assessment Working Papers - Notes from document review
- Copies of documents (with permission!)
- Interview notes
Cross reference to criteria Date each document
Now What??? Meet with your team each day to review working papers Develop preliminary findings
Finding
Other Assessment Results Noteworthy practices - Positive practices implemented by the organization
Observations - Practices noted during the review. No corrective action required.
Example: Finding 1 The QA Coordinators need training in QA concepts and theories
Example: Revised Finding 1 The QA Coordinators are unable to perform the review and approval of QAPPs as specified in the QMP because they have not been trained in the process.
Draft Preliminary Findings Presented to summarize what you have observed during the onsite Observations should be factual and to the point Findings can change during follow up review - (HINT - You’re not finished!)
Exercise!
Closing Meeting Brief management on your preliminary findings Explain your reporting process and timetable Remember: - No Surprises!
- Preliminary findings must not be frivolous
- Keep the meeting to no more than 1 hour
- Try to address immediate concerns but DON’T
- Make promises you can’t keep
- Usurp your management’s authority
The Black Hole Phenomenon
The Report The assessment report communicates assessment findings to management. It should clearly identify: - the type of assessment,
- the assessor,
- the assessee,
- what was assessed,
- the findings,
- if requested by the authorizing entity, recommendations for corrective actions
Effective Reporting Clear and concise writing Address significant deficiencies in a constructive manner Report the organization’s actions, not the actions of specific individuals Prioritize assessment findings and any need for corrective actions
Example Report Outline Background and Purpose Assessment Scope Assessment Results and Rationale References Appendices - A. List of Personnel Interviewed
- B. List of Documents and Records Reviewed (optional)
- C. Corrective Action Plan
Close Out
Close Out BEFORE you can close the assessment you need: - A response from the assessed organization with an acceptable corrective action plan (if necessary) and an indication that the corrective actions have been completed
- A review of the assessment file to ensure that it is complete before it is archived.
SUMMARY Now you can discuss: - the purpose and benefits of Quality Systems Assessments (QSAs)
- how QSAs differ from other types of assessments
You can demonstrate a process for planning, conducting, and reporting the results of QSAs - I can give you a six-word formula for success: "Think things through - then follow through."
- Edward Rickenbacker
Dostları ilə paylaş: |