[Editor's note: This is Part 2 of the "Handbook for the Assurance of Radar Cross Section Measurements." The original document was published by NIST in DRAFT Form using Scientific Word v2.5. This rendition was converted first to Rich Text Format (RTF), then the document was converted to HTML, with formatting clean-up applied manually. Hopefully, the formatting that resulted does not detract from the readability of the document. Comments regarding content should be directed to Dr. Lorant Muth, NIST, (303) 497-3603, email@example.com -- Roger Davis, 25 Feb 98]
[Part 1: Handbook for the Assurance of Radar Cross Section Measurements (26k)]
[Back to RCSweb Home Page]
This Standard applies to calibration laboratories in the development and implementation of their quality system. [NCSL Z540-1]
Organization and Management
The laboratory shall:
Interpretive Guidance [NCSL Z540-1 Handbook]: Laboratory management needs the authority to plan, organize, direct and control to assure quality and protect integrity of results. Laboratory management needs the support of senior management in terms of budget, equipment, facilities, and people.
b) have arrangements to ensure that its personnel are free from any undue pressures which might adversely affect the quality of their work [NCSL Z540-1];
Interpretive Guidance [NCSL Z540-1 Handbook]: Laboratory personnel should be insulated from any work related pressures which would compromise the quality of work. The source of undue pressure may be internal (management pressure, deadlines, etc.) or external (customer complaints, priority requests etc.)
c) be organized in such a way that confidence in its independence of judgement and integrity is maintained at all times [NCSL Z540-1];
Interpretive Guidance [NCSL Z540-1 Handbook]: However a laboratory operates, as part of a larger organization/government agency, etc. or as a stand-alone business, it should be operated so that quantity does not influence quality.
d) specify and document the responsibility, authority and interrelation of all personnel who manage, perform or verify work affecting the quality of calibrations [NCSL Z540-1];
Interpretive Guidance [NCSL Z540-1 Handbook]: The laboratory should have an organization chart and job descriptions for the entire calibration laboratory. Responsibilities and authorities should be clearly defined.
e) provide supervision by persons familiar with the calibration methods and procedures, the objective of the calibration and the assessment of the results. Management practices shall be such as to ensure adequate supervision [NCSL Z540-1];
Interpretive Guidance [NCSL Z540-1 Handbook]: Individuals assigned direct supervisory responsibilities over laboratory personnel should be knowledgeable in the methods, practices and procedures of the calibrations being performed and of assessing calibration results. The responsibilities of these supervisory personnel should not be so extensive as to limit their effectiveness.
f) have a technical manager (however named) who has overall responsibility for the technical operations [NCSL Z540-1];
Interpretive Guidance [NCSL Z540-1 Handbook]: There should be an individual designated whose job description includes the authority to make decisions regarding the technical (standard selection, calibration method selection, technical qualifications of personnel, etc.) aspects of laboratory operations.
g) have a quality manager (however named) who has responsibility for the quality system and its implementation. The quality manager shall have direct access to the highest level of management at which decisions are taken on laboratory policy or resources, and to the technical manager. In some laboratories, the quality manager may also be the technical manager or deputy technical manager [NCSL Z540-1];
Interpretive Guidance [NCSL Z540-1 Handbook]: Responsibility for the quality system of the laboratory should be assigned to an individual who is free of pressures which may conflict with the quality of calibrations.
Interpretive Guidance [NCSL Z540-1 Handbook]: Individuals who have the required knowledge and expertise to assume the duties of the technical or quality manager should be assigned as alternates to provide continuity of operation during absences of the technical or quality manager.
i) where relevant, have documented policy and procedures to ensure the protection of customer's confidential information and proprietary rights [NCSL Z540-1];
Interpretive Guidance [NCSL Z540-1 Handbook]: A laboratory should identify any customer information or material that is considered confidential or proprietary and have appropriate policies and procedures in place to protect the customer's interest.
j) where appropriate, participate in interlaboratory comparisons and proficiency testing programs [NCSL Z540-1].
Quality system, audit and review
Interpretive Guidance [NCSL Z540-1 Handbook]: The laboratory should have a quality system, documented in a quality manual, that defines the policies and practices which must be followed in the conduct of its business. Laboratory management is responsible for ensuring that all laboratory personnel are aware of, understand, and implement the elements of the quality system. The quality manager is responsible for maintaining the quality manual.
The quality manual, and related documentation, shall state the laboratory's policies and operational procedures established in order to meet the requirements of this Standard [NCSL Z540-1].
Interpretive Guidance [NCSL Z540-1 Handbook]: The phrase "quality manual, and related documentation" is a direct reference to the quality documentation defined in paragraph 5.1 as being required to "be available for use by the laboratory personnel". The quality manual should contain "policies and objectives for, and commitment to, good laboratory practice and quality calibration services". These policies and objectives should reference the quality documentation containing the procedures and practices to be followed to ensure the policies and objectives are met.
The quality manual and related documentation shall also contain [NCSL Z540-1]:
a) a quality policy statement, including objectives and commitments, by top management [NCSL Z540-1];
b) the organization and management structure of the laboratory, its place in any parent organization and related organization charts [NCSL Z540-1];
c) the relations between management, technical operations, support services and the quality system [NCSL Z540-1];
d) procedures for control and maintenance of documentation [NCSL Z540-1];
e) job descriptions of key staff and reference to descriptions of other staff [NCSL Z540-1];
f) identification of the laboratory's approved signatories (where this concept is appropriate) [NCSL Z540-1];
g) the laboratory's procedures for achieving traceability of measurements [NCSL Z540-1];
Interpretive Guidance [NCSL Z540-1 Handbook]: Reference to the methods and procedures used to establish the traceability path, to a national laboratory, an intrinsic standard or other acceptable source, should be included. Sections 9.2 and 9.3 of the Standard define the requirements for achieving traceability.
h) the laboratory's scope of calibrations and/or verifications [NCSL Z540-1];
i) arrangements for ensuring that the laboratory reviews all new work to ensure that it has the appropriate facilities and resources before commencing such work [NCSL Z540-1];
j) reference to the calibration and verification procedures used [NCSL Z540-1];
k) procedures for handling calibration and verification items [NCSL Z540-1];
l) reference to the major equipment and reference measurement standards used [NCSL Z540-1];
m) reference to procedures for calibration, verification and maintenance of equipment used [NCSL Z540-1];
n) reference to quality assurance practices including interlaboratory comparisons, proficiency testing programs, use of reference materials and internal quality control schemes [NCSL Z540-1];RCS NOTES -The results of interlaboratory comparisons should be documented and made generally available to relevant parties.o) procedures to be followed for feedback and corrective action whenever measurement discrepancies are detected, or departures from documented policies and procedures occur [NCSL Z540-1];
Interpretive Guidance [NCSL Z540-1 Handbook]: Policies and procedures should describe the actions to be taken when it is discovered that an error has occurred or procedures were not followed. An assessment of the impact (of the discrepancy or departure) on the results of measurements and a determination of appropriate corrective action should be conducted. This requirement pertains to discrepancies or unplanned departures from normal operation.
p) the laboratory management arrangements for exceptionally permitting departures from documented policies and procedures or from standard specifications [NCSL Z540-1];
Interpretive Guidance [NCSL Z540-1 Handbook]: Laboratory management is responsible for ensuring that laboratory policies and procedures are adhered to. The quality documentation should include the required process to be followed when departing from these policies and procedures.
q) procedures for dealing with complaints [NCSL Z540-1];
r) procedures for protecting confidentiality and proprietary rights;
s) procedures for audit and review [NCSL Z540-1];
Interpretive Guidance [NCSL Z540-1 Handbook]: An audit is a detailed evaluation of the compliance with the quality system. It attempts to find out if the system is in fact working the way it is documented. A review is a comparison of the system with the general standard, other quality needs, and the general business of the laboratory to determine if something needs to be changed in the system which will end up with new or revised documentation. These are separate functions and must be treated separately by the laboratory. Requirements for audit and review procedures are defined in paragraphs 5.3 to 5.5 of the Standard and should be included in the quality documentation.
u) a statement of the laboratory's policy concerning the technique(s) to be used for determining measurement uncertainty and calibration/verification adequacy [NCSL Z540-1].
Interpretive Guidance [NCSL Z540-1 Handbook]: Section 10 of the Standard covers the requirements for determining measurement uncertainty and calibration/verification adequacy. A policy statement referencing these procedures should be included in the quality documentation.
The laboratory shall arrange for audits of its activities at appropriate intervals to verify that its operations continue to comply with the requirements of the quality system. Such audits shall be carried out by trained and qualified staff who are, wherever possible, independent of the activity to be audited. Where the audit findings cast doubt on the correctness of validity of the laboratory's calibrations results, the laboratory shall take immediate corrective action and shall immediately notify, in writing, any client whose work may have been affected [NCSL Z540-1] .
Interpretive Guidance [NCSL Z540-1 Handbook]: A quality audit is a periodic check to ensure that the laboratory is operating in accordance with the policies and procedures set out in the quality manual and related documentation. Quality audits may include audits of results. Audit intervals should be based on an analysis of factors such as risk, criticality of the measurement system, complexity of the system, and failure rates, etc. and may be different for each measurement area. Audits should be conducted by persons possessing the knowledge and skills necessary to understand the process being audited as well as the process of auditing. Auditors are preferably persons from outside the area being audited. Immediate corrective action refers to the speed and breadth of response necessary to prevent the taking of bad data and/or the generation of inaccurate results. Each aspect of the quality system should be periodically audited at an appropriate interval to provide sufficient data for an effective management review. Many organizations find that an interval of six or twelve months to be appropriate.
The quality system adopted to satisfy the requirements of this Standard shall be reviewed at least once a year by the management to ensure its continuing suitability and effectiveness and to introduce any necessary changes or improvements [NCSL Z540-1].
All audit and review findings and any corrective actions that arise from them shall be documented. The person responsible for quality shall ensure that these actions are discharged within the agreed timeframe [NCSL Z540-1] .
In addition to periodic audits the laboratory shall ensure the quality of results provided to customers by implementing checks. These checks shall be reviewed and shall include, as appropriate, but not be limited to [NCSL Z540-1] :
1- internal quality control using, wherever possible, statistical techniques; NOTE-Measurement assurance techniques are acceptable means to control the measurement process and consistently produce the highest quality measurements;
2- participation in proficiency testing or other interlaboratory comparisons;
3- regular use of certified reference materials and/or in-house quality control using reference materials;
4- replicate measurements using the same or different methods;
5- correlation of results for different characteristics of an item [NCSL Z540-1] .
The calibration laboratory shall have sufficient personnel, having the necessary education, training, technical knowledge and experience for their assigned functions [NCSL Z540-1].
The calibration laboratory shall ensure that the training of its personnel is kept up-to-date consistent with employee assignments and development [NCSL Z540-1] .
Records on the relevant qualifications, training, skills and experience of the technical personnel shall be maintained and be available to the laboratory [NCSL Z540-1].
Accommodation and environment
The laboratory shall effectively monitor, control and record environmental conditions as appropriate. Due attention shall be paid, for example, to biological sterility, dust, electromagnetic interference, humidity, line voltage, temperature, and sound and vibration levels, as appropriate to the calibrations concerned [NCSL Z540-1].
Equipment and reference materials
The laboratory shall be furnished with all items of equipment ... required for the correct performance of calibrations/verifications. In those cases where the laboratory needs to use equipment outside its permanent control, it shall ensure that the relevant requirements of this Standard are met [NCSL Z540-1] .See Handbook.
All equipment shall be properly maintained. Maintenance procedures shall be documented. Any item of the equipment which has been subjected to overloading or mishandling, or which gives suspect results, or has been shown by verification or otherwise to be defective, shall be taken out of service, clearly identified and wherever possible stored at a specified place until it has been repaired and shown by calibration or verification to perform satisfactorily. The laboratory shall examine the effect of this defect on previous calibrations [NCSL Z540-1].See Handbook.
Each item of equipment including reference materials shall, when appropriate, be labeled, marked or otherwise identified to indicate its calibration status [NCSL Z540-1].See Handbook.
Records shall be maintained of each item of equipment and all reference materials significant to the calibrations/verifications performed [NCSL Z540-1] .See Handbook.
Measurement traceability and calibration
Interpretive Guidance [NCSL Z540-1 Handbook]: A calibration laboratory should have a calibration system for all equipment utilized in the calibration laboratory. This system should allow for:
1- identification of equipment requiring calibration;
2- identification of overdue items in the recall system and removal of such items until corrected; and
3- items that are known to be broken, out of tolerance, unstable so that its parameters cannot be predicted, or have intermittent problems, etc., or are judged unreliable, are identified and segregated until appropriate corrective action has been taken.
Corrective action may be in the form of limited use, repair, or correction factors, etc. Example(s): There are usually two distinct groups of instruments involved in any calibration system:
1- Those that belong to or are assigned to the calibration laboratory itself. The calibration laboratory should keep tight control of all its equipment used for calibration. This includes periodic recalibration of its instruments/tools in most cases. Exceptions should be listed and identified as such, e.g., DC power supplies, height transfer standards, etc.; i.e., instruments/tools which require initial calibration and only functional checks thereafter.
2- Those that belong to or are assigned to customers of the calibration laboratory or that are maintained through an instrument pool or lease/rental program. Control of these instruments/tools is usually left to the user or its assigned delegate. It is up to the user to identify items requiring periodic recalibration, to provide adequate identification, and maintain a tracking program for his equipment.
The calibration laboratory should have complete control of its inventory of instruments and tools used for the calibration and verification of its own items. These instruments/tools, utilized by the calibration laboratory, could either be owned, borrowed, leased, or rented from other facilities or signed out from loan pools. In all cases, those items requiring calibration or verification should be fully accounted for at all times and should be in a calibrated/verified status through the use of an official calibration recall system. [ANSI/NCSL Z540-1 Handbook] It is important to note that not ALL instruments/tools used by the calibration laboratory for calibration or verification do require such actions. The calibration laboratory decides which items are used as sources or indicators, thereby requiring no calibration, and which items have an effect on the accuracy for the units under test. Examples of items not usually requiring calibration could be: DC power supplies and sine plates. [ANSI/NCSL Z540-1 Handbook] A recall system will mean different things for different organizations, facilities, or agencies. In general, it can be said, however, that a recall system should include: [ANSI/NCSL Z540-1 Handbook]
1- identification of items requiring calibration or verification (which is discussed in detail in section 11);
2- a detailed data base [see 3) below] for items requiring calibration or verification that indicates past and future calibration dates, etc.; and
3- a description of how the instruments/tools requiring calibration and verification are moved to and from different calibration laboratories, how they are calibrated or verified including what procedures are used, and how calibration records are kept.
The recall system, through its database, should provide a means for identifying calibration laboratory items that are overdue for calibration. The calibration laboratory should ensure that such items are taken out of service and segregated until recalibrated. [ANSI/NCSL Z540-1 Handbook] Problem items such as repeated out-of-tolerance conditions of the same parameter of the same unit should be investigated for cause and possible problems downstream. Appropriate action should be taken by the laboratory to either repair, derate, or eliminate such units from the inventory. A mechanism should be available to identify these items. Items that show abnormal behavior should be temporarily removed from the active inventory until the problems have been clearly identified and can be controlled. Items that are broken should be removed from the active inventory and identified as such. Items for which trend and control charts are kept and which indicate that certain parameters are falling outside the predicted control limits, should be removed from the active inventory list, identified as such, and investigated for cause. Once understood and corrected they can be entered into the active inventory once again after adequate calibration. [ANSI/NCSL Z540-1 Handbook]
The overall program of calibration and/or verification of equipment shall be designed and operated so as to ensure that, wherever applicable, measurements made by the laboratory are traceable to national, international, or intrinsic standards of measurements where available. Calibration certificates and/or reports shall, wherever applicable, state the traceability to national, international, or intrinsic standards of measurements and shall provide the measurement results and associated uncertainty of measurement and/or a statement of compliance with an identified metrological specification [NCSL Z540-1].
Where traceability to international, national, or intrinsic standards of measurement is not available, traceability requirements may be satisfied by:
1- participation in a suitable program of interlaboratory comparisons or proficiency testing;
2- internationally accepted standards in the field concerned;
3- suitable reference materials;
4- ratio or reciprocity-type measurements; or
5- mutual consent standards which are clearly specified and mutually agreed upon by all parties concerned [NCSL Z540-1].
Interpretive Guidance [NCSL Z540-1 Handbook]: In addition to traceability to national or international standards laboratories, items 1) through 5) can be used to fulfill traceability arguments.
Reference standards of measurements held by the laboratory shall be used for calibration or verification only and for no other purpose, unless it can be demonstrated that their performance as reference standards has not been invalidated [NCSL Z540-1].
Reference standards of measurement shall be calibrated by a competent body that can provide traceability as described in 9.2 or 9.3. There shall be a program of calibration and verification for reference standards [NCSL Z540-1] .
Interpretive Guidance [NCSL Z540-1 Handbook]: Any reference standards employed should be traceable to a competent source. It is to be understood that competent implies ability to prove adequacy of the calibration process and standards employed through detailed procedures and documentation, trend and control charts, and full uncertainty analysis. The calibration laboratory should have a program in place that provides periodic calibration of the reference standards through a recall system. A laboratory in compliance with this Standard would be considered competent. Example(s) [NCSL Z540-1 Handbook]:
1- If the reference standard is calibrated by a national or international standards laboratory, then it is assumed that traceability is fulfilled and that adequate uncertainty is provided. It is up to the user of these standards to maintain the integrity of such standards through adequate handling, trend and control charts, or other intermediate checks as necessary.
2- If the reference standard is calibrated with an intrinsic standard by the same laboratory then full adequacy of calibration has to be maintained in terms of calibration processes, procedures, documentation, trend and control charts, and uncertainty analysis.
3- If the reference standard is calibrated by a laboratory not identified as a national or international standards laboratory, then that laboratory has to be prepared to show adequacy of the calibration process through detailed documentation of the procedures, trend and control charts, and uncertainty analysis. This laboratory also has to be able to prove full traceability for that standard.
Where relevant, reference standards and measuring and test equipment shall be subject to in-service checks between calibrations and verifications [NCSL Z540-1] .
Interpretive Guidance [NCSL Z540-1 Handbook]: In some applications certain standards and equipment might require additional checks and verifications in addition to the normal calibrations performed based on the official recall system. Such checks help in providing trend and control charts, improve the maintenance of drift rates, and increase confidence in measurement capability. Irrespective of the recall system, adequate trend and control charts should be maintained of the traceable reference standards. In some cases the recall interval is short enough. In other cases, the laboratory should implement additional measures to ensure the integrity of the standards. This applies especially to those standards that exhibit large (compared to the quoted accuracies) drift rates. In most cases, though, a well monitored recall system should suffice to guarantee the integrity of the standards and other test and measuring equipment maintained by the laboratory. Items not needed for long periods or only required if and when a special test arises, could be put into a temporary storage category with the understanding that they will be calibrated prior to the next use. [ANSI/NCSL Z540-1 Handbook]
Reference materials shall, where possible, be traceable to national or international standards of measurements, or to national or international standard reference materials [NCSL Z540-1].
Interpretive Guidance [NCSL Z540-1 Handbook]: Reference materials used in the calibration laboratory should be traceable to some ultimate source. Such sources could be national or international standards laboratories, or vendors whose products are recognized by such laboratories.
Interpretive Guidance [NCSL Z540-1 Handbook]: Documentation of the calibration/verification process is usually necessary to maintain repeatability of measurement(s). This includes documentation of the process and the necessary instructions to properly operate the standards, the device being tested and any ancillary equipment that directly affects the calibration/verification. Some items being calibrated require special handling and preparation. The documents should be current to support the calibration/verification process. Technicians and engineers performing calibrations should be able to access the needed documentation within the laboratory, offices in close proximity to the laboratory or other suitable location near the workplace. The person responsible for performing the calibration/verification should not only be able to obtain the needed documents, but needs to know when and what changes have been made that may affect the process. Documented instructions do not necessarily mean hard copy documents. Documentation can be in the form of software, CD ROM, microfilm, or other suitable media. Example(s): [ANSI/NCSL Z540-1 Handbook] The detail of the instructions necessary to accomplish a measurement varies according to the skill and experience of the metrology engineer or technician, the complexity of the calibration being performed and how often that the calibration is performed. Entry level personnel use detailed instructions and well written procedures to master each new measurement they perform. A very senior engineer performing the same procedure on a daily or weekly basis may not need to refer to the documentation as often. However, should a manual or procedure change be made and not properly annotated, a problem could arise that could make the calibration /verification invalid. Documentation can include: [ANSI/NCSL Z540-1 Handbook]
1- calibration procedures or procedure checklists;
2- equipment installation, operator and service manuals; and
3- reference books, such as CRC Engineering Tables, ISO International Vocabulary, technical textbooks, and equipment manufacturer's application notes.
The laboratory shall use appropriate methods and procedures for all calibrations/verifications and related activities within its responsibility (including, but not limited to, sampling, handling, transport and storage, preparation of items, estimation of uncertainty of measurement and analysis of calibration data) [NCSL Z540-1].
a) Calibration procedures shall contain the required range and tolerance or uncertainty of each item or unit parameter being calibrated or verified. In addition, the procedures shall contain the generic description of the measurement standards and equipment needed with the required parameter, range, tolerances or uncertainties, and specifications for performing the measurement of the calibration or verification, and/or representative types (manufacturer, model, option) that are capable of meeting the generic description for the measurement standards. The procedures shall be consistent with the accuracy required, and with any standard specifications relevant to the calibrations/verifications concerned [NCSL Z540-1].
b) The laboratory shall ensure that the calibration uncertainties are sufficiently small so that the adequacy of the measurement is not affected. Well defined and documented measurement assurance techniques or uncertainty analyses may be used to verify the adequacy of a measurement process. If such techniques or analyses are not used, then the collective uncertainty of the measurement standards shall not exceed 25% of the acceptable tolerance (e.g., manufacturer's specification) for each characteristic of the measuring and test equipment being calibrated or verified [NCSL Z540-1] .
Interpretive Guidance [NCSL Z540-1 Handbook]: a) The method of calibration employed should be appropriate for the parameter being calibrated/verified. A detailed calibration procedure should be available and followed to the degree necessary for the calibration/verification being performed. The use of well documented calibration procedures is desired in order to maintain consistency of the measurement process when performed at different times and by different operators. b) The standards selected and their associated uncertainties or tolerances should be adequate for the calibration/verification being performed. A detailed explanation of the complete process of expressing measurement uncertainty can be found in the "ISO Guide to the Expression of Uncertainty in Measurement" (Listed in the Reference Section of this Handbook). The NCSL Recommended Practice RP-3; Calibration Procedures Content and Format, provides good guidance on how a calibration procedure may be written.
Where methods are not specified, the laboratory shall, wherever practical, select methods that have been published in international or national standards, those published by reputable technical organizations or in relevant scientific texts or journals [NCSL Z540-1].
Interpretive Guidance [NCSL Z540-1 Handbook]: The laboratory performing the calibration/verification should be using an acceptable calibration method or procedure. If the equipment manufacturer or the customer does not prescribe a method or procedure, then one must be developed or obtained from another source.
Example(s) [NCSL Z540-1 Handbook]: References to measurement techniques can be found in textbooks, scientific papers, professional journals, and from the results of measurement research. Examples include:
1- Textbooks, such as "Microwave Measurements" edited by A. E. Bailey, or "Technology of Electrical Measurements" by L. Schnel;
2- International Standards;
3- National Standards, Mil-Std-461D & 462D, American Society for Testing and Materials (ASTM), American National Standards Institute (ANSI), American Society of Mechanical Engineers (ASME), Instrument Society of America (ISA);
4- IEEE Proceedings;
5- NCSL Workshop & Symposium Proceedings;
Some research relevant to RCS technology has been published in
1- NIST/NBS Technical Notes; and
2- NIST/NBS Special Publications.
Where it is necessary to employ methods that have not been well established, these shall be subject to agreement with the customer, be fully documented and validated, and be available to the customer or other recipients of the relevant reports [NCSL Z540-1].
Example(s) [NCSL Z540-1 Handbook]:
Some techniques for validating measurement methods are:
1- technical review by others in same industry or discipline; (These are sometimes utilized in disciplines such as super high frequency microwave, gloss, hardness, etc. which often have to rely upon "consensus methods")
2- review by engineers with specialty experience in the discipline;
3- evaluation and/or review by scientific investigation;
4- obtaining acceptance by the measurement community, or "Benchmarking";
5- customer verification that the calibration data or acceptance criteria provided satisfies the needs of the application; and
6- repeating the measurement by a second method and comparing the results.
Where sampling is carried out as part of the calibration method, the laboratory shall use documented procedures and appropriate statistical techniques to select the samples [NCSL Z540-1].
Calculations and data transfers shall be subject to appropriate checks [NCSL Z540-1] .
Example(s) [NCSL Z540-1 Handbook]: Some appropriate checks may include: a) review of calculations; b) comparison to previous results; c) standard data sets; d) review of abnormal and/or unexpected results; e) comparison of original data sheet against the final report; and f) investigation of out-of-tolerance results.
Where computers or automated equipment are used for the capture, processing, manipulation, recording, reporting, storage or retrieval of calibration data, the laboratory shall ensure that [NCSL Z540-1]:
1 - the requirements of this Standard are complied with;
2 - computer software is documented and adequate for use;
3 - procedures are established and implemented for protecting the integrity of data; such procedures shall include, but are not limited to, integrity of data entry or capture, data storage, data transmission and data processing;
4 - computer and automated equipment is maintained to ensure proper functioning and provided with the environmental and operating conditions necessary to maintain the integrity of calibration data;
5 - it establishes and implements appropriate procedures for the maintenance of security of data including the prevention of unauthorized access to, and the unauthorized amendment of, computer records [NCSL Z540-1] .
Interpretive Guidance [NCSL Z540-1 Handbook]: This section relates strictly to calibration data. When automated procedures are used the laboratory should ensure that the measurement is made correctly and the calibration is not invalidated. The user of the calibration results expects that instruments calibrated, data taken, and calculations accomplished are correct, private (secure) and have not been tampered with or corrupted.
1 - The use of automated calibration processes does not relieve the laboratory of its responsibility to meet all of the requirements of the Standard [NCSL Z540-1 Handbook].
2 - This applies to both purchased and internally developed software. Since the Standard does not identify specific requirements, each laboratory should define its own needs for software documentation and adequacy [NCSL Z540-1 Handbook].
3 - Integrity means that the data has not been corrupted nor tampered with [NCSL Z540-1 Handbook].
4 - Computers and peripheral equipment used for calibration processes have recommended environmental operating needs and routine maintenance procedures that ensure their continued operation [NCSL Z540-1 Handbook] .
5 - This section of the Standard addresses the issue of security against corruption of data as well as maintaining its confidentiality. Each laboratory should determine the level of security needed to adequately protect its data. Laboratories with many different operators and a more complex system may require more elaborate security measures than a laboratory with a single operator and limited access [NCSL Z540-1 Handbook].
1 - None offered
2 - Documentation elements to consider:
Inventory of utilized software
Documentation of software is a requirement of the Standard. This documentation may be within the program or in accompanying documents. Written instructions on how to load, execute and process a particular program are recommended. Screen prompts to assist the operator with the steps of the procedures, connections, and any precautions are desirable. Some calibrations can be accomplished with a single connection and the equipment can then be left unattended until the program is complete and the report printed. Therefore, the level of operator intervention is an indicator of how extensive the operator instructions in the documentation should be. Testing of software provides objective evidence of its adequacy. All software should be tested prior to use and periodically thereafter. One method of testing or verifying the software would be to compare measurement values taken by the automated process with those taken manually. The periodic use of a check standard could also provide a reliability check of the software.
3 - Care in the processing of calibration data can be demonstrated by documenting periodic tests of the software and computer system. Tests can be developed for algorithms to ensure that calculations are being accomplished correctly. Frequent backups of the system discs can ensure that data is recoverable. Clearly identifying the latest revision of software being used can ensure that the latest changes or corrections have been accomplished. Maintaining calibration programs on a local area network (LAN) or shared resource manager (SRM) with a designated system administrator can be effective in reducing errors in handling software and saving files. Calibration reports can sometimes have faults which could indicate possible problems with the calibration itself or with data reduction. Reading the calibration report and reviewing the data can help eliminate problems which may have occurred during the calibration.
4 - None offered.
5 - Security to protect the computer system, the software and the calibration data can be demonstrated. Access to program code should be limited to reduce the possibility of changes to test points, limits, and data. System administrators and programmers are usually authorized to change program code. Prohibition of unauthorized operators can be accomplished with a system of log-on protocols and passwords that are changed frequently. A positive system backup process, repeated frequently, is a good plan to ensure data is not lost and is recoverable. Routine screening for computer viruses is a good plan to protect programs and archived data [NCSL Z540-1 Handbook].
Documented procedures shall exist for the purchase, reception and storage of consumable materials used for the technical operations of the laboratory that can affect the results of calibrations [NCSL Z540-1].
Handling of calibration items
All records (including those listed in 8.4 pertaining to calibration equipment), certificates and reports shall be safely stored and held secure and in confidence to the customer for the period specified in the quality manual [NCSL Z540-1].See Handbook.RCS NOTES -1) An RCS measurement range shall have a documented configuration control procedure. Accurate records of calibration and measurement system configuration will be available to range personnel and other interested parties. See Section 8.2) A range will follow a well established calibration schedule. The results of each calibration will be documented, including a record of personnel performing the calibrations and the full configuration of the measurement system.3) Control charts will be available to monitor system performance over time.
4) Records of interlaboratory comparison studies will be available to range personnel, customers and other interested parties.
Certificates and Reports
See Handbook. Each certificate or report shall include at least the following information:
a) a title, e.g. "Calibration Report" or "Calibration Certificate";
b) name and address of laboratory, and location where the calibration was carried out if different from the address of the laboratory;
c) unique identification of the certificate or report (such as serial number) and of each page, and the total number of pages;
d) name and address of customer, where appropriate;
e) description and unambiguous identification of the item calibrated;
f) characterization and condition of the calibration item;
g) date(s) of performance of calibration, where appropriate;
h) identification of the calibration procedure used, or unambiguous description of any non-standard method used;
i) reference to sampling procedure, where relevant;
j) any deviation from, additions to, or exclusions from the e calibration method, and any other information relevant to a specific calibration, such as environmental conditions;
k) measurements, examinations and derived results, supported by tables, graphs, sketches, and photographs, as appropriate, and any failures identified;
l) a statement of the estimated uncertainty of the calibration result (where relevant);
m) a signature and title or an equivalent identification of the person(s) accepting responsibility for the content of the certificate or report (however produced), and date of issue;
n) where relevant, a statement to the effect that the results relate only to the items calibrated;
o) a statement that the certificate or report shall not be reproduced except in full, without the written approval of the laboratory;
p) special limitations of use; and
q) traceability statement [NCSL Z540-1].
Where the certificate or report contains results of calibrations performed by sub-contractors, these results shall be clearly identified [NCSL Z540-1] .
Interpretive Guidance [NCSL Z540-1 Handbook]: If all or part of the calibration is performed by a sub-contractor, this fact must also appear on the report or certificate as well as appropriate other related information per 13.2, e.g. identification, location, etc. of the sub-contractor.
Particular care and attention shall be paid to the arrangement of the certificate or report, especially with regard to presentation of the calibration data and ease of assimilation by the reader. The format shall be carefully and specifically designed for each type of calibration carried out, but the headings shall be standardized as far as possible [NCSL Z540-1] .
Material amendments to a calibration report or calibration certificate after issue shall be made only in the form of a further document, or data transfer including the statement "supplement to calibration report [or calibration certificate], serial number...[or otherwise identified]", or equivalent form of wording. Such amendments shall meet all the relevant requirements of 13.2 of this Standard [NCSL Z540-1].
Example(s) [NCSL Z540-1 Handbook]: Material amendments may include: revision of the actual data; correction of erroneous calculations, data transcription errors, erroneous calibration dates; insertion of omitted data.
The laboratory shall notify customers promptly, in writing, of [NCSL Z540-1]:
1 - any event such as the identification of defective calibration equipment that casts doubt on the validity of results given in any calibration report or certificate, or amendment to a report or certificate. Such notification shall quantify the magnitude of error created in the calibration results.
2 - any customer's measuring and test equipment found significantly out-of-tolerance during the calibration/verification process. Measurement data shall be reported so that appropriate action can be taken [NCSL Z540-1] .
The laboratory shall ensure that, where customers require transmission of calibration results by telephone, telex, facsimile or other electronic or electromagnetic means, staff will follow documented procedures that ensure that the requirements of this Standard are met and that confidentiality is preserved [NCSL Z540-1].
Sub-contracting of Calibration
Where a laboratory sub-contracts any part of the calibration, this work shall be placed with a laboratory complying with the requirements of this Standard. The laboratory shall ensure and be able to demonstrate that its sub-contractor is competent to perform the activities in question and complies with the same criteria of competence as the laboratory with respect of the work being sub-contracted. The laboratory shall advise the customer of its intent to sub-contract any portion of the calibration to another party [NCSL Z540-1].
Accreditation of the sub-contractor against the requirements of the Standard is generally the preferred means by which the laboratory can be assured that the sub-contractor is in compliance. However, some of the other approaches listed above, or combinations of them, may prove to be adequate. For example, a laboratory may request that a sub-contractor complete a questionnaire survey, examine the sub-contractor's quality manual, and require that the sub-contractor participate in a measurement assurance program. The laboratory might then determine the sub-contractor's compliance to the Standard through the combination of all three elements [NCSL Z540-1 Handbook].
The laboratory shall record and retain details of its investigation of the competence and compliance of its sub-contractors and maintain a register of all sub-contracting [NCSL Z540-1].
Outside support services and supplies that affect calibration results
Interpretive Guidance [NCSL Z540-1 Handbook]: The laboratory should have documented policies and procedures for addressing complaints against its activities. The laboratory's activities may be considered as any activity that relates to the quality of the service provided, including, but not limited to, calibrations, consultations, advertisement of its services, or advertisement of its accredited status (if applicable). Example(s): The laboratory's system may be considered adequate when the policies and procedures require some type of record be maintained to document all complaints received and subsequent actions taken to resolve those complaints. The system used to address and resolve complaints can be as simple or as elaborate as needed. Simple log books may suffice in some operations and a complex computer data base may be needed in others. The complaint system may be tailored to the size of the calibration operation, type of customers serviced, contract requirements, or other criteria.
Where a complaint, or any other circumstance, raises a concern regarding the laboratory's compliance with the laboratory's policies or procedures, or with the requirements of this Standard or otherwise concerning the quality of the laboratory's calibrations, the laboratory shall ensure that complaints in those areas of activity and responsibility involved are promptly resolved [NCSL Z540-1].
Interpretive Guidance [NCSL Z540-1 Handbook]: The laboratory's documented policy and procedures should provide for the prompt resolution of all complaints regarding the quality of the laboratory's activities.
Appendix A for the RCS Range Book
RCS ranges may consider developing their range books as electronic documents rather than as paper documents. While a conventional paper copy certainly meets all requirements of ANSI/NCSL Z-540-1, an electronic version, available on a network server or CD ROM, may be easier to develop and maintain.
The RCS range book should serve as an important general reference for range personnel and customers, and will play an important part in the certification review. We believe that organizing range documentation and making it easy to access will produce better communications and more consistent practice within each range. Hence, great care should be exercised in developing and maintaining it.
General Range Book Outline
Section -- Title
1-Introduction and Endorsement (5.2a): This section should contain a brief statement of compliance with the ANSI/NCSL Z-540-1 standard, as well as a policy statement regarding the organization's commitment to continuous quality improvement. The "range book" should be signed as "approved" by an appropriate manager or director, who should be in the reporting chain of the range quality manager or technical lead engineer. The exact format and wording can be customized by the organization.
2-References: This section should list documents cited in the range book. In particular, it may be convenient to complete stand-alone reports covering various aspects of range operations (operational security, measurement procedures, uncertainty analysis procedures, equipment documentation, for example) which could be included in the range book by reference only.
3-Glossary: This section should list the relevant definitions from NCSL Z540-1 Section 3, in addition to specific range-related definitions and acronyms. For instance, if your range uses an SPC MKIII radar, it might be helpful to identify this acronym in advance. Other specialized terms related to the acquisition and processing of RCS data should also be defined.
4-Organization and Management Structure (5.2b, c, f): This section should fully define the organization which is chartered to operate the range. The management hierarchy from the range-line technicians and engineers through the highest levels of the range management should be defined. Lead personnel and approved signatories should be identified. Standard company organization charts are acceptable.
5-Quality System, Audit Trail, and Review (5.2d, i, p, r, s, 5.4, 5.5): All items of section 5 need to be addressed at some point in the range book. Because of their importance, some of these items are treated in separate sections of the range book. Here we include all items that are not treated separately, as listed after the section heading. The purpose of this section is to ensure that a complete overview of the company's quality system and its self-auditing processes is given. Important miscellaneous items are to be included here. We recommend closely following the appropriate parts of section 5 of the main text.
6-Range Personnel Information (5.2e): This section should contain information and qualifications for all personnel assigned to the range. Specialized training, training certificates, or other range-related qualifications should appear in this section. Company or resume format may be used.
7-Accommodation and Environment: This section should document the range environment as well as any and all parameters of the range environment which may effect the outcome of measurements, and thereby should be monitored. (Depending on the complexity of the range, this section may or may not be a significant section of the range book.)
8-Equipment, Reference Material(s), Configuration Control (5.2l): Configuration control is essential for maintaining a repeatable, quality RCS system. Therefore, this section of the range book may be one of the most significant. Here the range should identify all electro-mechanical equipment that makes up the RCS range. References to appropriate equipment manuals should be given in sufficient detail so that a qualified technician can trace the major subsystems of the range. Maintenance and calibration histories should be kept for critical components and complete system configuration should be tracked. The procedures for identifying equipment, maintenance and configuration control should be documented here. Personnel responsible for following these procedures should be clearly identified.
9-Measurement Traceability and Calibration (5.2g, m, t): This section should complement Section 8 as it specifically relates to primary RCS calibration. The range will document its calibration program that ensures that all equipment critical to RCS measurement calibrations operate within certified performance limits. Equipment and system calibration intervals should be clearly established. Whenever possible traceability of system components and of total system calibration to a national standard should be maintained. Calibration certificates and reports should be referenced. System configuration used in RCS calibrations should be tracked using the method documented in section 8. The system configuration used in interlaboratory comparisons should be clearly documented together with the results of calibration comparisons. System components tracked under section 8 need only be referenced here.
10-Measurement and Calibration Procedures (5.2h, j, t, u): Both measurement and calibration procedures should be fully documented. Separate documentation should be available for wideband calibration and measurement procedures, monostatic and bistatic (multistatic) procedures, and polarimetric (reciprocal or nonreciprocal) calibration procedures. Procedures for deriving or assigning an RCS value to a primary target through the use of "transfer" calibration standards should be documented in this section. Calibration and measurement results to support the validity of procedures used should be displayed or referenced. The validity of such results should be supported by stated uncertainty bounds obtained through well-defined uncertainty procedures appropriately referenced. Calibration intervals should be clearly established.
11-Handling Procedures for RCS Standard Calibration Items (5.2k): This section should summarize how primary calibration items are protected during storage, handling, and use. Mechanical tolerance certificates or techniques for verifying tolerances of primary standards is appropriately placed this section.
12-Record Keeping System (5.2d): This section describes the range's procedures for creating and maintaining records of all aspects of range operations. Procedures for maintaining calibration records, measurement records, system configuration, etc. are especially important. Procedures to record interlaboratory comparison results should be clearly defined.
13-Certificates and Reports: This section should define the standard reporting formats used by the ranges when reporting RCS measurements to customers. While any report can be specialized to the requirements of the customer, this section should provide what baseline information will always be available in a test report. A range is allowed to use its own format, provided that the minimum information required by NCSL Z540-1 is included.
14-Subcontracting Records: This section details any aspects of the RCS calibration process that is subcontracted. Any subcontracted work must satisfy the NCSL Z-540-1 standard. (This section may not be applicable for most ranges.)
15-Outside Services and Supplier Records: This section should document any services or suppliers that are used to produce calibrated RCS data. This section complements section 14. For instance, if a range purchases an 8 in calibration sphere from a vendor for primary calibration, this section would document the needed tolerances of such a standard, as well as the delivered tolerance of the calibration item. As an additional example, if radar absorbing material is used on a calibrated target measurement and absolute RCS data is reported, it would be appropriate to create records of where the RAM was purchased, what characteristics were expected and delivered, and what, if any, repeatability would result from reusing the same material samples on the same target location.
16-Complaint Response System (5.2o, q): This section should document a range's formal complaint procedure used to mitigate and resolve disputes between the range and range customer. Appropriate complaint record keeping system should be also defined or referenced here.
17-Interlaboratory Comparison Programs (5.2n): This section should include the companies policy concerning interlaboratory comparisons. Records of participation in interlaboratory comparisons, the results obtained, conclusions drawn from such studies are to be presented here. It is expected that interlaboratory comparisons be used regularly to improve measurement and calibration quality at the range. An intercomparison schedule should be stated. (Note: Participation in a national intercomparison study is essential for certification.) Publication of intercomparison results is strongly encouraged; such publications should be referenced here.
18-Data Processing Procedures: Important details of processing measurement and calibration data should be documented here. Sample results of processing should be included here. Standard data processing procedures should be referenced, and the validity of innovative processing should be documented. This section should strongly support and complement the material in 17 above.
19-Range Specific Uncertainty Analysis (5.2u): Policies and procedures for establishing range uncertainties need to be completely documented here or referenced. A sample uncertainty table together with system parameters should be displayed here. For a NIST recommended practice, see .
20-Ongoing Research, Planned Improvements: Other sections of this "range book" describe elements of the overall range measurement assurance program (MAP). The purpose of this section is to briefly identify and summarize any on-going range research and other activities designed to improve range data quality, efficiency, repeatability, or traceability. A list of desirable research areas to be conducted in the future to improve specific and known deficiencies on the range or in the RCS industry should also be constructed. Such research information would allow customers to quickly identify and evaluate on-going range improvements in the context of their current or planned use of the range. It also may offer the opportunity for a customer to cost share or jointly sponsor range research of interest to the customer.
Appendix B: Standard Calibration Procedures
Dual Cylinder Calibration
Polarimetric Calibration with a Rotating Dihedral
Appendix C: Artifacts and Intercomparisons
Presently, we are exploring calibration standards which include cylinders dihedrals, trihedrals, and Bruderhedrals. We are circulating several sets of such targets as part of a national intercomparison program see Tables 2-4. If your organization would like to participate in this study, please contact the authors.
In the future we would like to consider more complicated targets, and artifacts optimized for polarimetric and bistatic calibrations. Intercomparison procedures must be improved to provide maximum benefit to the RCS community. In particular, as data accumulates, statistical analysis methods will become more important.