Black-box Reliability Challenge 2019 – announcement

Hi everyone,

For the impatient:

As an offshoot of the discussion of this year’s workshop in Delft we would like to share with you the draft version of a website dedicated to a reliability benchmarking framework and reliability problem repository:

It will be followed by a reliability benchmarking challenge starting in next year February. We are still working to meet that deadline but we believe that we have sufficient material to be shared. We are curious to hear your opinion and incorporate your suggestions!

For the more thorough:

In this email we would like to give you an update on the progress and status of the Black-box Reliability Challenge 2019 and we kindly invite you to give your feedback on the provided material and benchmark framework tutorial.

First, a brief recap to remind you what is this all about. In January this year, we organized the workshop on the “Computational Challenges in the Reliability Assessment of Existing Structures” ( ). As a response, the workshop attendees expressed their strong desire to come up with an open, systematic, representative and reproducible benchmarking framework to assess the accuracy and efficiency of currently available reliability algorithms and those to come in the future. A first proposal for such benchmarking framework was made in April this year (see ), on which we received valuable feedback from multiple researchers. From then, you haven’t heard anything from us, since other projects required most of our attention. However, especially in the last few weeks we picked up the gauntlet again, and we made a lot of progress.

Our activities focuses on several aspects:

  • The set-up of a black-box challenge server on which the performance functions will be executed, and an interface to allow for input-output communication between participants and server;
  • The development of a repository with details of the challenge (short term goal) and, after closure of the challenge, the description, function files and visualization of performance functions (long term goal), see;
  • The development of a dedicated website for Black-box Reliability Challenge 2019, with detailed information of the challenge (will be made live early next year);
  • Selection of benchmark problems based on symbolic expressions and finite element models (related to problems in structural, geo-, hydraulic, and earthquake engineering);
  • The development of an approach to create representative (overfitted) surrogate models for the engineering related reliability problems.

It is our intention to complete all the tasks early next year, so that we can open the challenge in February 2019.

If your time permits, we would like to ask you to check the repository and to execute the tutorial on how you can participate to the challenge. Please, provide us your feedback by writing us an email or using a GitLab account. Your feedback will be used in the further development of the challenge framework. Finally, we would like to encourage you to forward this email into your network, so that more people can join the Black-box Reliability Challenge 2019.

Arthur Slobbe <>

Arpad Rozsas <>