U -- DEVELOP AND VALIDATE A COMPREHENSIVE TEST FOR SAFETY AND SOUNDNESS EXAMINER ACCREDITATION

Notice Date

October 24, 2000

Contracting Office

Department of the Treasury, Office of Thrift Supervision (OTS), Procurement and Administrative Services, 1700 G Street, N.W., 3rd Floor, Washington, DC, 20552

ZIP Code

20552

Solicitation Number

RFP-TOTC00019

Response Due

October 27, 2000

Point of Contact

Douglas Mason, Sr. Contract Specialist, Phone (202) 906-7624, Fax (202) 906-5648, Email douglas.mason@ots.treas.gov

E-Mail Address

Douglas Mason (douglas.mason@ots.treas.gov)

Description

This is Amendment #1 to RFP-TOTC00019, the purpose of which is to provided to all potential offerors, OTS's response to several questions concerning the solicitation. We are also extending the due date for proposals at this time. Question #1. Section B: Schedule of Items/Prices: The RFP describes steps 1-3 (Job Analysis, Test Item Writing, and Test Item Linking) as optional. Should cost estimates for these steps be included in the project cost estimate? If so, can a range for Steps 1-3 be provided? RESPONSE: The RFP lists these steps as being completed on an _as needed_ basis. However, our perference is for the offeror to provide specific pricing for each of the steps. Question #2. Section C: Statement of Work, Section II _ The Comprehensive Test: The RFP mentions that the training tests will become _developmental_. Does this mean that if an individual fails the test that they can still proceed in the course sequence? RESPONSE: An FTR candidate who fails a test in the accredited course sequence will be placed under a developmental plan to acquire the minimum level of knowledge. However, they will proceed to the next course in the sequence. Qestion #3. Section C: Statement of Work, Section II _ The Comprehensive Test: Regarding test item development, the RFP notes that OTS will be _responsible for developing the initial test question database and answer keys_. What is the status or stage of item development? Will the contractor selected for validation have an opportunity to participate in or influence the methods used in test item development? Are test items currently under development by another vendor or by subject matter experts? What type of item development training have these individuals received? RESPONSE: OTS Safety and Soundness examiners will serve as subject matter experts for the creation and technical review of test items. Test item selection and development has not been started although a small database of older test items is available to use which may reduce the overall number of items that must be created. The contractor will be expected to guide the test item development process to insure that critical job functions are tested with suitably written questions. SME_s have not yet received any training on item development. Question #4. Section C: Statement of Work, Section III _ Content Validation: The RFP details that the contractor will be responsible for establishing linkages between items and job activity/knowledge areas. It is recommended that this task would best be completed by job experts during item writing or during meetings of subject matter experts facilitated by the contractor. Is there flexibility in this contractor requirement? RESPONSE: OTS_s preference is to minimize the time commitment of its SMEs_ on this project. As we understand the question, the option exists to concurrently develop the links between job performance objectives and the test and the creation of test items. This option may be the most efficient way to proceed. However, OTS would be open to discussion on this point as long as the final test becomes a valid and reliable measure of job skills and knowledge. Question #5. Section C: Statement of Work, Section III -- Implementation: The RFP notes that the contractor is responsible for developing the tests in a format consistent with electronic test administration. What is the required platform for test delivery (e.g., Microsoft Word, Perception)? In what format will the test questions be provided? With regard to scoring, what type of scoring is desired (i.e., electronic scoring, handscoring)? Can OTS provide further detail surrounding the scoring manual requirement? RESPONSE: Test items will be created using Microsoft Word. OTS_s intent is to administer the test using ProMetrics (formerly, Sylvan) centers. We will be using some form of electronic scoring to facilitate test results analysis. Question #6. Section C: Statement of Work, Section III -- Implementation: With regard to the review/coaching system requirement, there is some concern surrounding the development of a study guide that keys each question to a specific knowledge/skill/ability area and reference site. Specifically, our concerns center on test security and the possibility of the test items being compromised if study guides in the aforementioned format are widely distributed. Is there flexibility in the format of the study guide? RESPONSE: OTS is likewise concerned about maintaining test integrity and security. The intent of the study guide is to assist those FTR candidates who must retake the test. It is envisioned that each test item will be keyed to a specific examination procedure or program or, a section of the Thrift Activities Handbook. The guide would show only those exam procedures, programs or Handbook sections that relate to an incorrectly answered test question. The guide would provide only general guidance on those topics needing additional study emphasis but would not be specific enough to impair the security or integrity of the test. Question #7. Section C: Statement of Work, Section III _ Follow-up Analysis: In this section, the RFP mentions that follow-up analyses should provide _evidence that each test accurately and reliably assesses the relevant knowledge, skills, and abilities of the Safety and Soundness Examiner with 36-48 months of experience._ Does this mean that OTS desires a criterion-related validity study to assess the relationship between performance on the test and performance in the job? Does this requirement imply that the test items will need to be content validated post-implementation? Can OTS provide more detail on this analysis requirement? RESPOSNE: OTS_s Comprehensive Test is to be criterion-referenced to the extent that it measures critical job knowledge areas necessary for an examiner to possess at the 36-48 months level of experience. The Test will be one of several assessment tools that will be used to measure FTR eligibility. It is not intended as a total performance assessment tool. Question #8. Section C: Statement of Work, Section V _ Contractor Experience and Evaluation Criteria: This section of the RFP requests _samples of similar work performed for banks or bank regulatory agencies._ The contractor will be able to include example materials from content validation data collection sessions (e.g., presentation materials, rating forms and scales), but due to confidentiality concerns will be unable to share specific test items validated at another bank regulatory agency. We will provide another bank regulatory agency as a reference who will be able to speak about our performance and test validation experience. Please let us know if this is acceptable. RESPOSNE: OTS would like the opportunity to examine similar work products during the proposal review process. However, to preserve confidentiality, we are willing to conduct this review onsite at the contractor_s office at another location or organization. If at another location or organization, we would like the contractor to make the necessary introductions and arrangements. Question #9. I understand that two separate tests will be developed with similar content and difficulty. Will each test consist of approximately 150 multiple-choice questions? RESPONSE: OTS wishes to develop a criterion-referenced test that effectively and reliably measures the job knowledge of a FTR candidate. In order to insure that all critical knowledge areas are adequately tested and the test has a high level of reliability, it is roughly estimated that each test will contain at least 150 questions. Question #10. It is stated in the RFP that OTS will be responsible for developing the initial test question database and answer keys. Will OTS be doing their own item writing and supplying the contractor with the test items and keys for the contractor to review and build two forms according to content specifications? Or will the contractor be assisting with the item writing and development as well? Please see RFP, under section III. Contractor Requirements and Deliverables, there are 7 items listed. Items 1 thru 3 are on _as needed_ bases. Do you anticipate on-site meetings for job analysis and test item writing and development (items 1 -- 3), or is this something that may be done through the mail? For example, it is common for the contractor to have item writing workshops (meetings), usually about 2 12 days, with subject matter experts, which OTS would be responsible for recruiting, to write and review test items. RESPONSE: This response is similar to one given to a previous question. OTS Safety and Soundness examiners will serve as subject matter experts for the creation and technical review of test items. Test item selection and development has not been started although a small database of current and older test items is available to use potentially reducing the overall number of items that must be created. The contractor will be expected to guide the test item writing process to insure all critical job knowledge areas are tested with suitably written questions. It is expected that the initial developmental meetings will be held on-site, most likely at our headquarters office in Washington, DC. However, once the development process is underway, mail, e-mail, etc. can be used. Question #11. Also, it is stated in the RFP that OTS will be responsible for scoring each test after they are administered. Normally, with computer-based testing, the exams are scored automatically (determined by a raw cutscore) and the candidate will know if he or she has passed or failed the exam before leaving the testing center. Do you not want any test results given to the candidates when they have completed the exam at the test site? We will be more than happy to arrange for OTS to receive the test results for scoring from the Prometric Testing Centers through an FTP site. RESPONSE: OTS prefers to not have the candidate know the final score while at the test center. Question #12. We need some clarification under Section C, #5. Normally, we determine a pass/fail cut by doing a cut score study. Can you explain what is meant by _develop preliminary pass points and establish minimum/maximum scores for each test within the following categories: Highly Recommended, Recommended, and Not Recommended?_ Also, normally we pilot new test forms by having eligible candidates take the test. No scores are given at the testing center, but candidates receive scores after a proper analysis has been performed on the usability of each test item. Then the tests or test items are modified to insure the appropriate level of difficulty, relevance and clarity. It is noted in the RFP that the tests will be administered to a sample of OTS examiners holding an FTR designation. Then the contractor will conduct a statistical analysis of the test results. Can you tell me how many OTS Examiners will be participating in the pilot test? And..... any idea on what your yearly volumes will be? RESPONSE: We would like to change the RFP to include only two categories of test achievement: Recommended or Not Recommended. The process for developing the cutoff score can be done using, for example, the Angoff-Nedelsky method. The test difficulty, relevance and clarity will need to be (at least preliminarily) established before it is administered to FTR candidates. The only possible available sample group to pilot the test are examiners already holding the FTR designation. We are estimating that approximately 20-25 examiners will participate in the pilot test. Yearly volume for the Comprehensive is estimated to be about 15 to 20 people. -- -- - NOTE: The due date for proposals is hereby extended from 3:00PM local time October 30, 2000 TO 3:00PM local time on November 03, 2000. YOU MUST ALSO ACKNOWLEDGE RECEIPT OF THIS AMENDMENT WITH YOUR PROPOSAL.

Web Link

Visit this URL for the latest information about this (http://www2.eps.gov/cgi-bin/WebObjects/EPS?ACode=M&ProjID=RFP-TOTC00019&LocID=1767)

Record

Loren Data Corp. 20001026/USOL002.HTM (D-298 SN5050P1)