Repository: dotnet/orleans

Issue #1069: Orleans clients using PersistentStreamProviders must have configured gateways for every silo in the cluster.
Created: 2015-11-25T01:19:09Z
Orleans clients communicate with the cluster through configured gateways.
http://dotnet.github.io/orleans/Orleans-Configuration-Guide/Client-Configuration.
If a grain activation is on a silo that is not on one of the configured gateways, Orleans will route communication with that grain through one of the configured gateways.  This is possible because each gateway can obtain the address of the silo responsible for any activation.
System targets, however, are not routed, as their address cannot be obtained.  Any call to a system target must be made directly to the silo responsible for that system target's activation.  This means that an Orleans client that needs to send a message to a system target must have the silo responsible for the system target's activation in it's list of configured gateways.
Since the Persistent Stream Providers used system targets to read from the queue, for Orleans Clients to use Persistent Stream Providers they must have gateway connections to every silo in the cluster.
Orleans clients should require only a single gateway.  Since this breaks that expectation, this is a bug.

Duplicate issue #1031: Client fails to connect to Secondary Silo when using a persistent stream provider
Created: 2015-11-17T17:36:27Z
[Question] [Help Needed]
I have all of the following on the same machine:

Primary Silo
Secondary Silo
A producer app setting some data on a Grain

This grain then writes that to a persistent stream which uses Redis as a backend


Client reading that stream
Redis

This configuration sometimes works and sometimes fails regardless of which server client/producer connect to. It seems to be dependent on the order of when each component starts.
When it fails I get the following exception:
    [2015-11-17 17:17:54.419 GMT     6  WARNING 100011  OutsideRuntimeClient    169.183.186.58:0]   No callback for response message: Unrecoverable Rejection (info: SystemTarget PullingAgentSystemTarget/FF/240b61ae not active on this silo. Msg=Response S169.183.186.58:11112:185475886*cli/80669cbe@084dfaec->S169.183.186.58:11112:185475886PullingAgentSystemTarget/FF/240b61ae@S240b61ae #2617: Orleans.Streams.IStreamConsumerExtension:GetSequenceToken()) Response S169.183.186.58:30001:0PullingAgentSystemTarget/FF/240b61ae@S240b61ae->*cli/80669cbe@084dfaec #2617: Orleans.Streams.IStreamConsumerExtension:GetSequenceToken() 

Configurations:
Notice that "169.183.186.58" refers to my machine's IP address
Primary listening on 30000 gateway and 11111 inter-silo:
<?xml version="1.0" encoding="utf-8"?>
<OrleansConfiguration xmlns="urn:orleans">
  <Globals>
    <Liveness LivenessType ="MembershipTableGrain" />
    <StorageProviders>
      <Provider Type="Orleans.Storage.MemoryStorage" Name="MemoryStore" />
      <Provider Type="Orleans.Storage.MemoryStorage" Name="PubSubStore" />
    </StorageProviders>
    <StreamProviders>
      <Provider Type="Orleans.Providers.Streams.SimpleMessageStream.SimpleMessageStreamProvider" Name="SMSProvider"/>
      <Provider Type="PipeStreamProvider.PipeStreamProvider" Name="PSProvider" NumQueues="2" Server="169.183.186.58:6379"/>
    </StreamProviders>
    <SeedNode Address="169.183.186.58" Port="11111" />
  </Globals>
  <Defaults>
    <Networking Address="169.183.186.58" Port="11111" />
    <ProxyingGateway Address="169.183.186.58" Port="30000" />
    <Tracing DefaultTraceLevel="Warning" TraceToConsole="true" TraceToFile="{0}-{1}.log">
      <TraceLevelOverride LogPrefix="Application" TraceLevel="Verbose" />
    </Tracing>
    <Statistics MetricsTableWriteInterval="30s" PerfCounterWriteInterval="30s" LogWriteInterval="300s" WriteLogStatisticsToTable="true"/>
  </Defaults>
  <Override Node="Primary">
    <Networking Address="169.183.186.58" Port="11111" />
    <ProxyingGateway Address="169.183.186.58" Port="30000" />
  </Override>
</OrleansConfiguration>

Secondary listening on 30001 gateway and 11112 inter-silo:
<?xml version="1.0" encoding="utf-8"?>
<OrleansConfiguration xmlns="urn:orleans">
  <Globals>
    <Liveness LivenessType ="MembershipTableGrain" />
    <StorageProviders>
      <Provider Type="Orleans.Storage.MemoryStorage" Name="MemoryStore" />
      <Provider Type="Orleans.Storage.MemoryStorage" Name="PubSubStore" />
    </StorageProviders>
    <StreamProviders>
      <Provider Type="Orleans.Providers.Streams.SimpleMessageStream.SimpleMessageStreamProvider" Name="SMSProvider"/>
      <Provider Type="PipeStreamProvider.PipeStreamProvider" Name="PSProvider" NumQueues="2" Server="169.183.186.58:6379"/>
    </StreamProviders>
    <SeedNode Address="169.183.186.58" Port="11111" />
  </Globals>
  <Defaults>
    <Networking Address="169.183.186.58" Port="11112" />
    <ProxyingGateway Address="169.183.186.58" Port="30001" />
    <Tracing DefaultTraceLevel="Warning" TraceToConsole="true" TraceToFile="{0}-{1}.log">
      <TraceLevelOverride LogPrefix="Application" TraceLevel="Verbose" />
    </Tracing>
    <Statistics MetricsTableWriteInterval="30s" PerfCounterWriteInterval="30s" LogWriteInterval="300s" WriteLogStatisticsToTable="true"/>
  </Defaults>
  <Override Node="Secondary">
    <Networking Address="169.183.186.58" Port="11112" />
    <ProxyingGateway Address="169.183.186.58" Port="30001" />
  </Override>
</OrleansConfiguration>

Producer connected to Primary on 30000:
<?xml version="1.0" encoding="utf-8" ?>
<ClientConfiguration xmlns="urn:orleans">
  <GatewayProvider ProviderType="Config"/>
    <Gateway Address="169.183.186.58" Port="30000"/>
  <Statistics MetricsTableWriteInterval="30s" PerfCounterWriteInterval="30s" LogWriteInterval="300s" WriteLogStatisticsToTable="true"/>
  <Tracing DefaultTraceLevel="Warning" TraceToConsole="true" TraceToFile="{0}-{1}.log">
    <!--<TraceLevelOverride LogPrefix="Application" TraceLevel="Info" />-->
  </Tracing>
</ClientConfiguration>

Client connected to Secondary on 30001:
<?xml version="1.0" encoding="utf-8" ?>
<ClientConfiguration xmlns="urn:orleans">
  <GatewayProvider ProviderType="Config"/>
  <Gateway Address="169.183.186.58" Port="30001"/>
  <Statistics MetricsTableWriteInterval="30s" PerfCounterWriteInterval="30s" LogWriteInterval="300s" WriteLogStatisticsToTable="true"/>
  <Tracing DefaultTraceLevel="Warning" TraceToConsole="true" TraceToFile="{0}-{1}.log">
  </Tracing>
  <StreamProviders>
    <Provider Type="Orleans.Providers.Streams.SimpleMessageStream.SimpleMessageStreamProvider" Name="SMSProvider"/>
      <Provider Type="PipeStreamProvider.PipeStreamProvider" Name="PSProvider" NumQueues="2" Server="169.183.186.58:6379"/>
  </StreamProviders>
</ClientConfiguration>

Any ideas why this could happen?
Thanks everyone and specially Jason Bragg for his time before,
Space Monkey

Closing comment (2015-11-30T18:55:00Z): @amamh As you pointed out, this looks related to #1069.  Noting this here for other readers.
If you explicitly specify all the gateways in your client configurations, does the intermittent failure go away?  If it does, that would confirm that this is being caused by #1069.