Due to the lacking information from the support network team to provide the information about how the LB monitoring the application health we are hardly to troubleshoot during incident happen.
For the easy way to verify the health monitoring rule set for ICMP and HTTP we can do the following test to verify.
ICMP : simply shut down all the node and leave one server node up , try to access then you will know the helath monitoring is working or not. ( this method is try to ping the server node , once detected unable to ping then the traffic will stop route to that server ~ around 1 to 2 minute to detect it)
TCP (by server name with port 443 / 80 ): this is something new i learnt which is we can stop the IIS [listener for port 443 and 80 will stop ]service at the server node then the traffic will stop route to this server node.
Back to the incident i have faced last week , which is we have delete one of the old SSL certificate and during reboot server causing the LB unable communicate to the server node.
root cause : one of the unused web application (stop) have blank hostname with 443 port enable and the SSL cert is blank . Causing LB unable to communicate with the server node.
solution : binding the SSL cert to the blank host name with 443 port
error message can see from system log file :
Event ID 15021 Source HttpEvent
An error occurred while using SSL configuration for endpoint 0.0.0.0:443. The error status code is contained within the returned data.