Kafka properties for CCADK
Published: 2022-09-11 12:37:52

message.timestamp.type -->  create time(default) and LogAppendTime( when the message is appended to Kafka partition)



**log.roll.hours** --> Time after which the segment would be rolled over (default 168 hours)
**log.msg.size** --> Size after which the segment would be rolled over (default 1GB)

**Default cleanup policy**  --> delete

**compact policy** --> Applicable only for keyed messages (Only the latest message with the key would be kept)
 
**retention.ms** --> delete segments older than this - default 7 days
**retention.bytes** -->  Delete oldest segments when partition gets too large - default -1 (unlimited)

## Producer properties
 -->


**compression.type** How data should be compressed. Values are none, snappy, gzip, lz4, zstd. Compression is performed on batches of records. Default: none

**acks **Used to determine when a write request is successful. Can be 0, 1, or all (-1). If acks is not zero, then the producer will retry failed requests. Default: 1

**delivery.timeout.ms** Puts an upper bound on the time to report success or failure after a call to send() returns. Use this to control producer retries. Default: 120000

**batch.size** Max number of bytes for a batch of messages sent to a partition. Default: 16384

**linger.ms** Time a batch will wait to accumulate before sending. Default: 0

 **group.id** - Consumers are grouped together using group id

**partition.assignment.strategy** ->  Strategy to assign partitions ->
						 org.apache.kafka.clients.consumer.RangeAssignor (Use RangeAssignor when joining data from multiple topics (this is the default))
						org.apache.kafka.clients.consumer.RoundRobinAssignor (Use RoundRobin when performing stateless operations on records from many topics)
						org.apache.kafka.clients.consumer.StickyAssignor (Sticky is RoundRobin with a best effort to maintain assignments across rebalances)
						org.apache.kafka.clients.consumer.CooperativeStickyAssignor 
							(CooperativeSticky is Sticky but it uses consecutive rebalances rather than the single stop the-world used by Sticky)


**heartbeat.interval.ms** (default: 3s) -> s the interval between successive heartbeat signals

**session.timeout.ms** -> If no liveness signal is received for more than session.timeout.ms milliseconds (default:10s) 
then the consumer is considered to be dead and the group coordinator triggers a partition reassignment

** max.poll.interval.ms** -> Data polling happens in the main thread of the consumer. It is possible for the liveness
thread of a consumer to work but its main thread to "hang." In this case, the liveness thread will notice that polling has timed out and proactively leave the consumer group so that another consumer can take over its partitions. 
This timeout is configured by this property. Default : 5 min

**state shuffling** -> the problem of managing state through failure

** group.instance.id ** ->  Enable static group membership by
configuring a unique group.instance.id to each Consumer in the Group

**The session.timeout.ms property must be between the
group.min.session.timeout.ms (Default: 6 seconds) and
group.max.session.timeout.ms (Default: 5 minutes) properties on the Broker.**

## Client properties

**bootstrap.servers** -> List of Broker host/port pairs used to establish the initial connection to the cluster

**key.deserializer** -> Class used to deserialize the key. Must implement the Deserializer interface

**value.deserializer** -> Class used to deserialize the value. Must implement the Deserializer interface

**client.id** -> String to identify this consumer uniquely; used in monitoring and logs

**group.id** -> A unique string that identifies the Consumer Group this Consumer belongs to.

**enable.auto.commit** -> When set to true (the default), the Consumer will trigger offset commits based on the value of auto.commit.interval.ms (default 5000ms)

**enable.auto.commit=false** -> disable the automatic offset commit feature. The default interval for offset auto commit is 5 seconds

**KafkaConsumer is not thread-safe**

**__consumer_offsets** -> This topic tracks group.id, topic and partition

**max.partition.fetch.bytes** -> Max. amount of data fetched per partition (default : 1 MB)

**max.poll.records** -> Max number of records per poll across all partitions (default 500)

**fetch.min.bytes** -> (Default: 1)
The minimum amount of data the poll() method should wait for, before returning it to the consumer.

**fetch.max.wait.ms** -> (Default: 500)
The maximum amount of time the server will block before
answering the fetch request if there isn’t sufficient data to
immediately satisfy the requirement given by fetch.min.bytes.