Development and Validation of Work Permit System Performance Assessment Questionnaire , a Case Study in an Iranian Oil Refinery

a Department of Occupational Health Engineering, School of Public Health, Tehran University of Medical Sciences, Tehran, Iran b Associate Professor, Department of Occupational Health Engineering, School of Public Health, Tehran University of Medical Sciences, Tehran, Iran c Office of Traffic Safety and Control, Abadan oil Refinery, Abadan, Iran d Department of Health Information Management and Medical Informatics, Faculty of Paramedical Sciences, Tehran University of Medical Sciences, Tehran, Iran

Accidents in the process industries are the leading causes of high costs imposed on chemical industries and refineries in different regions of the world, including our country (1). Process industries are defined as industries in which raw materials are transformed into intermediate or final products through physical or chemical processes or other ways. In these industries the production, storage, transportation, usage and disposal of chemicals are inherently hazardous and the potential for catastrophic events, such as toxic leaks, fires, and explosions is high (2).Some examples of these catastrophic events around the world include the explosion at Texas American Oil Refinery in 2005, explosion at Venezuelan Amuey Oil Refinery in 2012, explosion and fire at Piper Alfa oil platform, explosion at French Feyzine Oil Refinery in 1966.
Moreover, among the process accidents occurred in Iran we can refer to the explosion of chemical tank in Shazand, Arak in 2008, widespread fire in the surface water channel below pipe racks of side services in Mahshahr petrochemical area in December 2009, as well as petrochemical blast in Mahshahr port in 2012, and petrochemical blaze broke out in Booali Petrochemical Company in 2016.These accidents claimed the lives of many people and cost economy billions of dollars for the country. (3)(4)(5)(6)(7)(8). Studies performed on the investigation of reasons fueling such accidents have revealed that one of the leading causes of these accidents is the failure or lack of a work permit system (9,10).
permit-to-work system is a documented system to control the activities which are potentially hazardous and likely to cause accidents. In other words, this system shows the method of safe working, control of potential risks or the elimination and reduction of the risks identified before the commencement of work (11). In addition, this system is a part of the process safety management program to identify, evaluate, and control the dangers of potentially hazardous unusual tasks (12).
The implementation of permit to work system is of paramount importance in the control and coordination of hazardous operations, Overhaul, installations, isolation, and responsibility allocation. Past investigations signified that accidents in process industries, such as refineries, petrochemicals, and chemical companies are pertinent to maintenance activities and their leading cause is the absence of permit-to-work system (11,13,14).
The results of the studies indicated that about 30% of permit-to-work systems in the process industries suffer from some deficiencies and problems regarding the form of work permits, training of people, identification of hazards, segregation methods, personal protective equipment, time limits, method of shift change, and the operationalization of repaired equipment. In addition, they did not use the existing licensing method properly and had deficiencies in the approval of the work license, with 60% of the defects being related to system monitoring (15).
The petroleum industry is regarded as the mother industry and the most important chemical industry in the country due to the excessive reliance of the national economy on expert revenues of oil. It is evident that various hazardous chemicals used and produced under different operating conditions in these industrial processes can have the potential for such accidents as fires and explosions that will have irreversible consequences on the safety elements and even the economy of the country. Therefore, the prevention of accidents in this industry should be one of the most important priorities of the aforementioned industry and national security professionals. Accordingly, more attention must be devoted to Permit to work system due to significant potential for accidents at oil refineries. Monitoring and reviewing the systems are needed to gain knowledge of their performance and work permit system is no exception in this regard. Given the importance of the work permit system in the prevention of accidents in the process industries, the assessment of its performance is of utmost importance. This assessment can identify strengths and weaknesses of this system in order to improve its performance. The audit checklists recommended by organizations, such as HSE-UK and SHELL are currently used to monitor the work permit system, and there are no local instruments to monitor the performance of work permit system in refineries and process industries, such as petrochemicals. With this background in mind, the present study aimed to develop and validate a performance and effectiveness assessment questionnaire to identify the strengths and weaknesses of the performance of a work permit system in an oil refinery located in southern Iran.
This descriptive-analytical study was designed in two phases and validity and reliability were assessed in six stages in 2019 (Table 1).
In the first stage of questionnaire development which was designing phase, a number of six criteria were identified by reviewing the literature and comparing the programs and guidelines for issuing work permits in different subsidiaries of the Ministry of Oil (NIOC, NIOC, petrochemical companies) and abroad ( HSE, TOTAL, OGP, SHELL (Oman Oil Company), as well as conducting interviews with academic and industrial experts familiar with work-permit system. The criteria include risk assessment, risk prevention and control, sense of responsibility, coordination and information, clarity and transparency, and documentation. Three groups of experts participated in this study, including the group of question designing experts, experts on face validity assessment, and experts on content validity. Initial questions were designed by the members of question designing experts, who were health professionals with a background of related safety research. Thereafter, a group was formed at the oil refinery involving 15 experts familiar with the requirements of the work permit system and issuing a work permit. They assessed sentences in terms of face validity, word choice, and comprehensibility and expressed their perspectives to shed more light Table 1) Identified Criteria and some of their most important roles in the performance of the work permit system Row Criterion The importance of criterion in the work permit system 1 Risk assessment A critical element in the work permit system which is the key to identification and assessment of the potential risks that may occur while performing the task.

Prevention and control of risks
A series of suggested controlling measures which are implemented before issuing a work permit, defined based on the performed risk assessment, and recorded in work permit. 3

Sense of responsibility
The performance and effectiveness of a system depend on the level of responsibility of the system members.

Coordination and information
Due to the large number of people involved in the system or other people who are interacting with the system, information and coordination play an important role in this system.

5
Clarity and transparency In the work permit system, work permit forms must be clear and comprehensible to all.

6
Documentation Due to critical guidelines in the work permit system, the importance of documenting it cannot be ignored. on the sentences and options. Thereafter, the questions which needed revision based on the opinion of the members of the content validity assessment group were revised ( Figure 1). Content validity assessment group was formed to perform content validity upon the completion of face validity. In this stage, it was agreed upon to use more participants to overcome future limitations, such as participants' withdrawal of the study and non-return of the questionnaire and increased the reliability of the study. Therefore, a number of 25 occupational health and safety experts, including academic staff and PhD students with at least 5 years of work experience, research and instruction, and safety experts and supervisors with over 10 years experience in oil refinery and familiar with the work permit system, were selected by convenient method.

Materials & Methods
Content validity was assessed using the content validity ratio (CVR) and content validity index (CVI). In addition, Lawshe's method was used for the calculation of CVR index (16,17). At the commencement of the study, the members of expert group were provided with the goals of the study and operational explanation of the questions. Thereafter, they were asked to classify each question based on the 3-point Likert scale, including "the question is necessary", "the question is useful but not necessary" and the question is not necessary". Content validity ratio was then calculated based on Equation1 : Where represents the number of members who found the question necessary, and n is the total number of members. The minimum acceptable value of CVR for each item was determined based on the number of experts evaluating the questions and the questions with a CVR value less than the number specified in Table 2 were excluded.
To examine CVI index, the members of the expert group were asked to identify the items of "relevance", "clarity" and "simplicity" of each question on a 4-point Likert scale. Experts rated the question as 1 (unrelated), 2 (relatively relevant), 3 (relevant), and 4 (quite relevant). In addition, The simplicity of the questions was rated as 1 (complicated), 2 (somewhat complicated), 3 (simple), and 4 (quite simple).  , represents the number of experts who rated the question as 3 and 4 and n signifies the total number of experts. The minimum acceptable value of CVI is measured at 0.79 and questions with a CVR less than 0.37 were excluded from the questionnaire and the final version of the questionnaire was prepared.
The final questionnaire is the assessment of the performance and efficiency of the work permit system consisting of 50 questions on the implementation of six specified criteria. In this questionnaire, experts and assessors of the work permit system were asked to comment on the process of work permit system administration using a 5-point Likert scale(1 very low, 2 low, 3 medium, 4 high, 5 very high).
In this questionnaire, the total score of the work permit system is obtained using the following equation: Permit to work system score= Ʃ risk assessment score+Ʃ control and prevention of hazards score+Ʃ sense of responsibility score+Ʃcoordination and information score+Ʃ documentation score.
Finally, based on the obtained score, the performance status of the system falls in one of the groups of excellent (76-100), good (51-75), moderate (26-50), and poor (25 and below). The final questionnaire was distributed among 120 oil refinery personnel who were in charge of issuing a work permit. The obtained results were analyzed in SPSS software(version 25) and a reliability coefficient of six criteria and the total questionnaire was estimated using Cronbach's alpha coefficient.
In the present study, the initial questionnaire was delivered to the members of the face validity assessment panel after being designed by the members of the expert panel. The face validity assessment panel consisted of 15 individuals, 3 of them had occupational health engineering degree, 4 industrial safety, 3 with technical inspection, 2 with industrial engineering, and 3 with chemical engineering. The mean age of the panel members was 32±2.08 and their mean work experience was 10±1.16 years. All questionnaires were returned and the mean return time was 36 days.The verdicts of members of face validity assessment panel on the initial questionnaire including writing corrections, word corrections, or addition of new questions were reviewed and necessary amendments were made.
Upon the completion of face validity assessment of the questionnaire and making the scientific and textual changes, the modified questionnaire was delivered to the panel of content validity assessment experts for content validity. The panel consisted of 25 content validity experts, including 5 faculty members with occupational safety and health, 3 undergraduate PhD students in occupational   health engineering with a focus on safety, and 17 industrial experts based at the Oil Refinery. Mean of age and work experience was reported as 37 (2.61%) and 12 (2.34%), respectively. In this study, in accordance with the values suggested by the Lawshe method, the minimum acceptable CVR value was measured at 37% when the panel of content validity assessment experts consisted of 25 members. The questions with CVR< 0.37 were deleted from the questionnaire. Finally, the questionnaire entailed 50 questions since 15 questions were deleted from 65 developed questions. The number of 100 questionnaires were returned out of the 120 questionnaires distributed among those involved in the work permit system and licensed to issue work permit at the refinery. The mean age of the participants was 41±6.24 and their mean work experience was 11±4.1. Other demographic characteristics of the participants are demonstrated in Table 3.
Cronbach's alpha coefficient was calculated to be 0.81 using SPSS software(version 25) for the whole questionnaire, as well as the six criteria. According to the results demonstrated in Table 4, the risk assessment dimension has the highest Cronbach's alpha (0.86) and the lowest Cronbach's alpha (0.78) was pertinent to the documentation dimension.
In the present study, a 50-item questionnaire with a 5-point Likert scale was designed and validated to evaluate the performance of the work permit system in order to evaluate the performance of the work permit system. The present questionnaire examines the performance of the work permit system in six criteria, including risk assessment, risk prevention, control, accountability, information and coordination, clarity and transparency, and documentation (Table 1). Finally, the overall system score is obtained based on the sum of the total criteria scores and its performance level (excellent, good, average, poor) is specified. Among the different criteria of the questionnaire, the highest number of questions was specified to the criterion of risk control and prevention which may be due to the importance of control and prevention of human errors.
The results of a study conducted by Jahangiri on the role of human errors in issuing work permit at a petrochemical site reported that the highest and lowest odds of human errors among job responsibilities of issuing work permit was related to the job duty of flammable and toxic gas tests and oxygen test, respectively. This study emphasizes the existence of an engineering approach to the assessment of human errors that may occur in the process of issuing a work permit and reduce the efficiency of the work system (10). In addition, Atabi et al. performed a study entitled "Classification of Human Errors in the Process of Licensing in a Petrochemical Industry Using the SHERPA Method". The results of this study signified that despite the existence of a work permit system for controlling non-routine activities, high accident statistics indicate the prominent role of human error in the failure of the work permit system (13).
HSE-UK agency calculates the performance of the work permit system based on 5-point criteria, including system, training and qualification, work permits, coordination, and monitoring. These criteria were rated as 1 "weak system", 2 "The system is not being used effectively", 3 "partial and functional system", 4 "applicable and acceptable system", and 5" applicable and satisfactory system". Finally, the whole status of the work permit system is categorized into four staus of very poor, poor, moderate, and good based on mean of every criterion (18). Level 4/3/2 Audits checklist which is held by SHELL refinery company is among the other existing tools to audit and evaluate the compliance of the work permit system with the licensing regulations and guidelines. This checklist consists of questions concerning the system, documentation, training and qualifications, work permit form, coordination, monitoring and isolation.
In this checklist, a score of 5 asserts good system performance (19). The introduced checklists for the supervision of work permit system aim to monitor the work permits issued in the workplace on a daily basis, and the answers are not designed in Likert-scale form, rather they expect a simple yes/no answer. The respondents cannot express their opinions since some questions are associated with uncertainty, and the yes/ no answer that indicates certainty in a particular subject cannot always reflect the current status of a subject; accordingly, a need exists to develop Likert-scale to improve the quality of responding. Therefore, the Likert-scale was used in designing response options of performance assessment questionnaire.
The answers to the questions raised by issuing of work permit at home and abroad and interviewing academic experts in different fields can reveal the weaknesses of the system. Ghahremani in his study investigated the compliance of the current state of the work permit system with the auditing criteria.SHELL checklists were used in this study, and the reasons for non-compliance were reported to be non-anticipation of emergencies, lack of documentation system, unspecified responsibilities (18). In addition, in another study performed by Mirdarikvand, failure to perform risk assessment, non-registration of necessary precautionary measures, and absence of a method for assessing the competence of work permits signatories were identified as the three major reasons of non-compliance in the implementation of work permit system based on HSE-UK audit checklists in an oil platform company (19). In the designed questionnaire, documentation, responsibility, prevention and control of risks, and risk assessment are considered as a separate criterion with several questions. Therefore, it can be suggested that the present questionnaire is also able to identify the discrepancies in the studies conducted by Ghahremani and Mirdarikvand.
In addition, Cronbach's alpha test indicates the reliability of the suggested tool to measure the performance of the work permit system at the oil refinery.
The examination of validity and reliability indices of the designed questionnaire indicated that the questionnaire enjoys satisfactory reliability and validity to evaluate the performance and effectiveness of the work permit system. This questionnaire can pinpoint the strengths and weaknesses of the work permit system by raising some questions regarding the conditions and factors affecting the performance and effectiveness of work permit system. Accordingly, it can be used as an effective tool to assess the performance and efficacy of the work permit system in process industries, such as oil refineries and petrochemicals.