This deliverable, D3.4 Validation Report v2, is the second in a series of three Validation Reports produced under WP3 Task T3.3 (Evaluation of Methodology (NIST) and Policy Recommendations). It reports on the first full application of the validation plan defined in D3.3, covering both the FAITH Trustworthiness Assessment Framework version 2 (TAF v2) together with its supporting Knowledge Base and ontologies, and the suite of in isolation trustworthiness assessment tools used across the FAITH Large Scale Pilots (LSPs). The report first presents the development and validation of the FAITH AI Trustworthiness Ontology (AITO) and the TrustGuard Ontology, which together operationalise the extended FAITH Knowledge Base and enable the in context capabilities introduced in TAF v2. AITO was assessed through automated logical and structural validation, representational adequacy exercises, and ongoing domain coverage review, and was found to satisfy the formal requirements of the production pipeline while remaining subject to continued extension as new resources and use cases are incorporated. The TrustGuard Ontology was then validated through a set of stakeholder competency questions and an accompanying query tool, confirming that the ontology supports the practical information retrieval needs of the FAITH tool suite while also surfacing refinements that feed back into AITO itself. The report then presents the validation of TAF v2 as a method, conducted through two structured expert workshops involving developers, risk assessors, and end users across the FAITH LSPs. Both workshops converged on a consistent rating of validated with minor refinements,
with participants confirming that the framework is conceptually clear, complete, and internally coherent. The refinements identified, including clearer phase to phase narrative, simplified terminology, and additional worked examples, are operational in nature rather than indicating any fundamental weakness in the framework. Validation of the framework’s in context, whole system capabilities remains provisional, as the underlying System Security Modeller (SSM) tool was not yet available to participants at the time of the workshops. In parallel, the three in isolation trustworthiness assessment tools, namely the AI Model Hub and Passport, TrustSense, and TrustGuard, were validated through standardised formal feedback surveys administered across all seven LSPs, supplemented by informal feedback gathered through training sessions and partner exchanges. All three tools achieved
a consistent expert assessment of validated with minor refinements, supported by strong scores for perceived usefulness, ease of use, and user trust. Three cross cutting priorities emerged across the tools and the framework alike: improving accessibility for non specialist users, reducing friction in integrating the tools into existing organisational workflows, and deepening cross tool integration through shared authentication infrastructure and alignment with the FAITH ontology. Taken together, the validation activities reported in this deliverable confirm that the FAITH methodology and toolset provide a credible, usable, and sufficiently useful basis for AI trustworthiness assessment in real world contexts. The identified refinements are already reflected in updates implemented or planned for Version 2 of the tools, and a further round of validation is planned once the SSM and updated TrustGuard are released, to directly assess the in context capabilities that could not be evaluated in
this round.