Load balancing with a dedicated edit server

We have three load balanced front end web server and one back end end edit server.  All content editing is done from this edit server.  No one logs into or does any edits on the front end server.  I have the load balancing setup per the instructions (at least I think I do), but I'm getting lots of errors in the logs when the front end sites start up like the following.  This seems to make the sites take a long time to start up.  We are also using the multisite option and with each site we add, the startup time seems to increase, I'm assuming because all of these failed communications between the front end servers.  Why do these front ends even need to attempt to contact each other if they are essentially read only.  I would think only the back end edit server would need to contact the front ends when content was updated.  Can this front end communication be turned off?

Here is one of the log entries from one front end server.

Timestamp: 1/27/2017 8:39:19 AM

Message: HandlingInstanceID: 411b7961-ce89-4a7b-959d-3e389c8c51e7
An exception of type 'System.Exception' occurred and was caught.
----------------------------------------------------------------
01/27/2017 02:39:19
Type : System.Exception, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
Message : Error sending system message to URL: sitefinity1/.../HandleMessages; HTTP method: PUT; Host: 'sitefinity1'
Source : 
Help link : 
Data : System.Collections.ListDictionaryInternal
TargetSite : 
HResult : -2146233088
Stack Trace : The stack trace is unavailable.
Additional Info:

TimeStamp : 1/27/2017 8:39:19 AM
FullName : Telerik.Sitefinity.Utilities, Version=9.1.6110.0, Culture=neutral, PublicKeyToken=b28c218413bdf563
AppDomainName : /LM/W3SVC/42/ROOT-1-131299799203347576
ThreadIdentity : 
WindowsIdentity : IIS APPPOOL\Sitefinity
Inner Exception
---------------
Type : System.Net.WebException, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
Message : The remote server returned an error: (401) Unauthorized.
Source : System
Help link : 
Status : ProtocolError
Response : System.Net.HttpWebResponse
Data : System.Collections.ListDictionaryInternal
TargetSite : System.Net.WebResponse EndGetResponse(System.IAsyncResult)
HResult : -2146233079
Stack Trace :    at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
  at Telerik.Sitefinity.LoadBalancing.WebServiceSystemMessageSender.WithExceptionHandling(String url, String httpMethod, String hostHeader, Action action)

Each server in the group needs to be able to talk to each other to communicate events especially around cache items. For example when to remove a page cached page when it has been updated.

(Sitefinity has a local memory cache)

Have you disabled the front end servers backend access?

I am assuming your front end servers are unable to access your backend server and that is where these errors are coming from.

But in my case, there is only one edit server, so the only system that should be contacting the front end servers to invalidate their cache is the one edit server.

Yes, you can not login to the backend from the front end server.  This is blocked by a rewrite rule.

The front end servers seem to be unable to access any other server.  Most of the errors are from one front end trying to talk to another front end.

There is a tick box you can select to disable the backend in the configuration. You don't have to use a rewrite rule.

All the front end servers need to communicate with each other.  I suspect you are using Web Services as the Load Balancing communication method.

Sitefinity has to assume that each server can act as an admin server and so it needs to communicate with each other.

I would suggest you ensure each can communicate (no network blocking issues) but detail your setup and log a ticket to support as they can then focus on your particular environment and set up.

I don't remember exactly what it was, but I had problems using the tick box to disable the back end, hence why I am using the rewrite rule.

I think our network setup has problems with the servers in the front end farm talking to each other.  I'll check with our network guys to see if that can be fixed.  I was hoping for a solution that would just turn that off since the cross talk between read only servers seems unnecessary overhead.

I believe there is more than just managing the cache between servers. I don't know all the hidden details.

Looking back at the error, it may not be that the front end servers can't see each other but more that they can't authenticate to each other. The error that it gets is a 401 Unauthorized which seems to imply sufficient rights are not available.  Could this be caused by the securityTokentIssuers/relyingParties settings being wrong?