Marit Boeker

BACKGROUND The importance of ontologies in the biomedical domain is generally recognized. However, their quality is often too poor for large-scale use in critical applications, at least partially due to insufficient training of ontology developers. OBJECTIVE To show the efficacy of guideline-based ontology development training on the performance of ontology developers. The hypothesis was that students who received training on top-level ontologies and design patterns perform better than those who only received training in the basic principles of formal ontology engineering. METHODS A curriculum was implemented based on a guideline for ontology design. A randomized controlled trial on the efficacy of this curriculum was performed with 24 students from bioinformatics and related fields. After joint training on the fundamentals of ontology development the students were randomly allocated to two groups. During the intervention, each group received training on different topics in ontology development. In the assessment phase, all students were asked to solve modeling problems on topics taught differentially in the intervention phase. Primary outcome was the similarity of the students’ ontology artefacts compared with gold standard ontologies developed by the authors before the experiment; secondary outcome was the intra-group similarity of group members’ ontologies. RESULTS The experiment showed no significant effect of the guideline-based training on the performance of ontology developers (a) the ontologies developed after specific training were only slightly but not significantly closer to the gold standard ontologies than the ontologies developed without prior specific training; (b) although significant differences for certain ontologies were detected, the intra-group similarity was not consistently influenced in one direction by the differential training. CONCLUSION Methodologically limited, this study cannot be interpreted as a general failure of a guideline-based approach to ontology development. Further research is needed to increase insight into whether specific development guidelines and practices in ontology design are effective.

OBJECTIVE: (a) To measure the effect of a guideline-based training on the performance of ontology developers compared with the performance after unspecific training by a competency question based evaluation; and (b) to provide empirical evidence for the applicability of competency questions in formal ontology evaluation in general. BACKGROUND: A close connection between ontology development and ontology evaluation as quality management procedure can been attained with the use of competency questions. Competency questions are often used as a semi-formal specification of requirements for an ontology under development. Hence they can also be used as evaluation instruments, in order to check how far an ontology fulfills these requirements. METHODS: A randomized controlled trial was conducted with two groups of 12 students each. The intervention consisted in a differential guideline-based training on ontology development vs. unspecific training. After a group-specific training focusing on three topics per group, performance of students was assessed with 12 exercises (2 exercises per topic), in which the students had to apply their skills. Different types of competency questions were elaborated for the analysis of the students' ontologies. We used the proportion of correct answers as a measure of ontology quality. RESULTS: On single topic level, the performance of ontology developers increased after guideline-based training for two out of the six topics: it increased from a proportion of 0.46 to 0.63 for the topic Process \& Participation and from 0.44 to 0.53 for the topic Collective Material Entity. In regression analysis, a positive correlation was shown between the performance of students on untrained topics and the performance after specific guideline-based training. Moreover, in multiple regression analysis an overall effect of specific training of 0.09 was calculated (p < 0.1). CONCLUSION: The results show an effect of a specific guideline-based training on the performance of ontology developers compared to the performance after unspecific training by an increase of about 10 \% on the rate of correct competency questions. In addition, this study has shown the general applicability of competency questions in a formal ontology evaluation scenario. However, the study also shows that the training of ontology developers and their performance evaluation is a tedious task. The resulting performance of ontology developers is more dependent on the a priori individual competencies than on the specific acquired skills after training.

Background: In order to improve ontology quality, tool- and language-related tutorials are not sufficient. Care must be taken to provide optimized curricula for teaching the representational language in the context of a semantically rich upper level ontology. The constraints provided by rigid top and upper level models assure that the ontologies built are not only logically consistent but also adequately represent the domain of discourse and align to explicitly outlined ontological principles. Finally such a curriculum must take into account the pre-existing skills and knowledge of the target audience. Objective: To develop a well-structured curriculum aligned to the particular requirements of life science professionals, in order to enable them to create logically sound, domain adequate and predicable ontologies using the Web Ontology Language (OWL) in Protégè. Methods: Content selection for the curriculum was based on the literature, pre-existing tutorials, and a guideline for good ontology development (i.e ontology design enhancing domain adequacy, sustainability and interoperability) that drew on the authors previous experiences with large ontology development projects. Learning objectives were formulated according to a needs assessment of the targeted learners, who were students trained in life sciences with basic knowledge and practical skills in computer science. As instructional format we choose an approach with a high amount of practical exercises. The curriculum was first implemented with 24 Students and 7 lecturers/ tutors over 5 full days. The curriculum was evaluated by gathering the participants feedback via a questionnaire. Results: Curricular development produced 16 modules of approximately 2 hours each, which covered basic principles of Applied Ontology, description logic syntax and semantics, as well as best design practices outlined in ontology design patterns and variants of the BioTop upper ontology. An opinion survey based on questionnaires indicated that the participants took advantage from the teaching strategies applied, as they indicated good knowledge gain and acknowledged the relevance of the modules. The difficulty was rated slightly lower. Conclusion: The development of teaching material for principled ontology design and best practices is of crucial importance in order to enhance the quality of biomedical ontologies. Here, we present a curriculum for a week long workshop, leveraging on current educational principles, focusing on interactive hands-on exercises, group interactions, and problem-oriented learning. Whereas evaluation clearly showed the success of this approach, in particular regarding student’s satisfaction, the objective measurement of traceable effects on the quality of the generated ontology, although of much higher interest, has just started.

Background Ontology Design Patterns (ODPs) are representational artifacts devised to offer solutions for recurring ontology design problems. They promise to enhance the ontology building process in terms of flexibility, re-usability and expansion, and to make the result of ontology engineering more predictable. In this paper, we analyze ODP repositories and investigate their relation with upper-level ontologies. In particular, we compare the BioTop upper ontology to the Action ODP from the NeOn an ODP repository. In view of the differences in the respective approaches, we investigate whether the Action ODP can be embedded into BioTop. We demonstrate that this requires re-interpreting the meaning of classes of the NeOn Action ODP in the light of the precepts of realist ontologies. Results As a result, the re-design required clarifying the ontological commitment of the ODP classes by assigning them to top-level categories. Thus, ambiguous definitions are avoided. Classes of real entities are clearly distinguished from classes of information artifacts. The proposed approach avoids the commitment to the existence of unclear future entities which underlies the NeOn Action ODP. Our re-design is parsimonious in the sense that existing BioTop content proved to be largely sufficient to define the different types of actions and plans. Conclusions The proposed model demonstrates that an expressive upper-level ontology provides enough resources and expressivity to represent even complex ODPs, here shown with the different flavors of Action as proposed in the NeOn ODP. The advantage of ODP inclusion into a top-level ontology is the given predetermined dependency of each class, an existing backbone structure and well-defined relations. Our comparison shows that the use of some ODPs is more likely to cause problems for ontology developers, rather than to guide them. Besides the structural properties, the explanation of classification results were particularly hard to grasp for 'self-sufficient' ODPs as compared with implemented and 'embedded' upper-level structures which, for example in the case of BioTop, offer a detailed description of classes and relations in an axiomatic network. This ensures unambiguous interpretation and provides more concise constraints to leverage on in the ontology engineering process.