Enhance external link validation

[Kostov6]

What would you like to be added:
Right now just a single HTTP HEAD request is send to validate an external link. In some cases this could not be enough and a more complex procedure would be required.

Also the whole link validation logic should be reviewed. Right now it seems we have a best effort approach - validation code was added to solve some specific problem at a given time in the past without having a test that defined the case. This leaves some logic that seems to be random like the Retry-After response header with the comment // support only value in seconds <= 5 min .
Because of this questions arise:

1. How do we handle request taking unexpectedly long to finish? By using expiring contexts?
2. How do we handle context cancelation or receiving connection error in general?
3. It is observed that Retrying request! could happen but does it provide any value and if so when?
4. What about the Retry-After header?

Do we search for a standard link validation procedure and implement it or do we stick with a best effort covering questionable scenarios? Without deciding on a direction enhancing link validation could not proceed.

How to categorize this?:

/kind enhancement
/priority 3