public void closeConnection() {
    LOGGER.debug("Closing connection");
    Connection connectionToRemove = threadLocalConnection.get();
    jdbcUtil.closeQuietly(connectionToRemove);
    if (connectionToRemove != null) {
      synchronized (this) {
        connectionsToCloseDuringDestroy.remove(connectionToRemove);
      }
    }
    threadLocalConnection.set(null);
  }

public
void
closeConnection
(
)
{
LOGGER
.
debug
(
"Closing connection"
)
;
Connection
connectionToRemove
=
threadLocalConnection
.
get
(
)
;
jdbcUtil
.
closeQuietly
(
connectionToRemove
)
;
if
(
connectionToRemove
!=
)
{
synchronized
(
this
)
{
connectionsToCloseDuringDestroy
.
remove
(
connectionToRemove
)
;
}
}
threadLocalConnection
.
set
(
)
;
}


private static String escapeQuotedSubstring(String input) {
    String[] parts = input.split("'");
    StringBuilder output = new StringBuilder(input.length() * 2);
    for (int i = 0; i < parts.length; i++) {
      if ((i % 2) == 1) {
        output.append("'").append(parts[i].replace("|", "\\|")).append("'");
      } else {
        output.append(parts[i]);
      }
    }
    return output.toString();
  }

private static String escapeQuotedSubstring ( String input ) { String [ ] parts = input . split ( "'" ) ; StringBuilder output = new StringBuilder ( input . length ( ) * 2 ) ; for ( int i = 0 ; i < parts . length ; i ++ ) { if ( ( i % 2 ) == 1 ) { output . append ( "'" ) . append ( parts [ i ] . replace ( "|" , "\\|" ) ) . append ( "'" ) ; } else { output . append ( parts [ i ] ) ; } } return output . toString ( ) ; }

Add backslash to escape the | character within quoted sections of the
input string.  This prevents the | from being processed as part of a regex.
@param input Contains the field name to process.
@return New string

Add backslash to escape the | character within quoted sections of the input string . This prevents the | from being processed as part of a regex .


private FlowControlSettings getFlowControlSettings() {
    return FlowControlSettings.newBuilder()
        .setLimitExceededBehavior(FlowController.LimitExceededBehavior.Block)
        .setMaxOutstandingElementCount((long) conf.basic.maxBatchSize * conf.maxThreads / conf.advanced.numSubscribers)
        .build();
  }

private FlowControlSettings getFlowControlSettings ( ) { return FlowControlSettings . newBuilder ( ) . setLimitExceededBehavior ( FlowController . LimitExceededBehavior . Block ) . setMaxOutstandingElementCount ( ( long ) conf . basic . maxBatchSize * conf . maxThreads / conf . advanced . numSubscribers ) . build ( ) ; }

Returns a flow control setting such that a subscriber will block if it has buffered more messages than can be
processed in a single batch times the number of record processors. Since the flow control settings are per
subscriber, we should divide by the number of subscribers to avoid buffering too much data in each subscriber.

@return settings based on the stage configuration.

Returns a flow control setting such that a subscriber will block if it has buffered more messages than can be processed in a single batch times the number of record processors . Since the flow control settings are per subscriber we should divide by the number of subscribers to avoid buffering too much data in each subscriber .


private InstantiatingGrpcChannelProvider getChannelProvider() {
    return SubscriptionAdminSettings
        .defaultGrpcTransportProviderBuilder()
        .setMaxInboundMessageSize(MAX_INBOUND_MESSAGE_SIZE)
        .setEndpoint(Strings.isNullOrEmpty(conf.advanced.customEndpoint) ? SubscriptionAdminSettings
            .getDefaultEndpoint() : conf.advanced.customEndpoint)
        .build();
  }

private InstantiatingGrpcChannelProvider getChannelProvider ( ) { return SubscriptionAdminSettings . defaultGrpcTransportProviderBuilder ( ) . setMaxInboundMessageSize ( MAX_INBOUND_MESSAGE_SIZE ) . setEndpoint ( Strings . isNullOrEmpty ( conf . advanced . customEndpoint ) ? SubscriptionAdminSettings . getDefaultEndpoint ( ) : conf . advanced . customEndpoint ) . build ( ) ; }

Creates a channel provider shared by each subscriber. It is basically the default ChannelProvider with the
exception that it can be configured with a custom endpoint, for example when running against the PubSub Emulator.

@return channel provider based on the stage configuration.

Creates a channel provider shared by each subscriber . It is basically the default ChannelProvider with the exception that it can be configured with a custom endpoint for example when running against the PubSub Emulator .


public Object put(OffsetAndResult<Map.Entry> batch) {
    if (consumerError != null) {
      throw new RuntimeException(Utils.format("Consumer encountered error: {}", consumerError), consumerError);
    }
    if (producerError != null) {
      throw new RuntimeException(Utils.format("Producer encountered error: {}", producerError), producerError);
    }
    try {
      Object expectedOffset = "EMPTY_BATCH";
      if (!batch.getResult().isEmpty()) {
        expectedOffset = batch.getResult().get(batch.getResult().size() - 1).getKey(); // get the last one
      }
      while (!dataChannel.offer(batch, 10, TimeUnit.MILLISECONDS)) {
        for (ControlChannel.Message controlMessage : controlChannel.getProducerMessages()) {
          switch (controlMessage.getType()) {
            case CONSUMER_ERROR:
              Throwable throwable = (Throwable) controlMessage.getPayload();
              consumerError = throwable;
              throw new ConsumerRuntimeException(Utils.format("Consumer encountered error: {}", throwable), throwable);
            default:
              String msg = Utils.format("Illegal control message type: '{}'", controlMessage.getType());
              throw new IllegalStateException(msg);
          }
        }
      }
      return expectedOffset;
    } catch (Throwable throwable) {
      controlChannel.producerComplete();
      if (!(throwable instanceof ConsumerRuntimeException)) {
        String msg = "Error caught in producer: " + throwable;
        LOG.error(msg, throwable);
        controlChannel.producerError(throwable);
        if (producerError == null) {
          producerError = throwable;
        }
      }
      throw Throwables.propagate(throwable);
    }
  }

public
Object
put
(
OffsetAndResult
<
Map
.
Entry
>
batch
)
{
if
(
consumerError
!=
)
{
throw
new
RuntimeException
(
Utils
.
format
(
"Consumer encountered error: {}"
,
consumerError
)
,
consumerError
)
;
}
if
(
producerError
!=
)
{
throw
new
RuntimeException
(
Utils
.
format
(
"Producer encountered error: {}"
,
producerError
)
,
producerError
)
;
}
try
{
Object
expectedOffset
=
"EMPTY_BATCH"
;
if
(
!
batch
.
getResult
(
)
.
isEmpty
(
)
)
{
expectedOffset
=
batch
.
getResult
(
)
.
get
(
batch
.
getResult
(
)
.
size
(
)
1
)
.
getKey
(
)
;
// get the last one
}
while
(
!
dataChannel
.
offer
(
batch
,
10
,
TimeUnit
.
MILLISECONDS
)
)
{
for
(
ControlChannel
.
Message
controlMessage
:
controlChannel
.
getProducerMessages
(
)
)
{
switch
(
controlMessage
.
getType
(
)
)
{
case
CONSUMER_ERROR
:
Throwable
throwable
=
(
Throwable
)
controlMessage
.
getPayload
(
)
;
consumerError
=
throwable
;
throw
new
ConsumerRuntimeException
(
Utils
.
format
(
"Consumer encountered error: {}"
,
throwable
)
,
throwable
)
;
default
:
String
msg
=
Utils
.
format
(
"Illegal control message type: '{}'"
,
controlMessage
.
getType
(
)
)
;
throw
new
IllegalStateException
(
msg
)
;
}
}
}
return
expectedOffset
;
}
catch
(
Throwable
throwable
)
{
controlChannel
.
producerComplete
(
)
;
if
(
!
(
throwable
instanceof
ConsumerRuntimeException
)
)
{
String
msg
=
"Error caught in producer: "
+
throwable
;
LOG
.
error
(
msg
,
throwable
)
;
controlChannel
.
producerError
(
throwable
)
;
if
(
producerError
==
)
{
producerError
=
throwable
;
}
}
throw
Throwables
.
propagate
(
throwable
)
;
}
}

Queues the batch for the consumer and waits until the consumer
successfully commits the batch. While waiting processes any
control messages from the consumer. Throws an exception
when the consumer has indicated it encountered an error.

Queues the batch for the consumer and waits until the consumer successfully commits the batch . While waiting processes any control messages from the consumer . Throws an exception when the consumer has indicated it encountered an error .


public Grok compileExpression(final String expression) {

    throwErrorIfDictionaryIsNotReady();

    final String digestedExpression = digestExpressionAux(expression);

    logger.debug("Digested [" + expression + "] into [" + digestedExpression + "] before compilation");
    return new Grok(Pattern.compile(digestedExpression));
  }

public
Grok
compileExpression
(
final
String
expression
)
{
throwErrorIfDictionaryIsNotReady
(
)
;
final
String
digestedExpression
=
digestExpressionAux
(
expression
)
;
logger
.
debug
(
"Digested ["
+
expression
+
"] into ["
+
digestedExpression
+
"] before compilation"
)
;
return
new
Grok
(
Pattern
.
compile
(
digestedExpression
)
)
;
}

Compiles the expression into a pattern

This uses the internal dictionary of named regular expressions

@param expression
@return The compiled expression


private String digestExpressionAux(String originalExpression) {

    final String PATTERN_START = "%{";
    final String PATTERN_STOP = "}";
    final char PATTERN_DELIMITER = ':';

    while(true) {

      int PATTERN_START_INDEX = originalExpression.indexOf(PATTERN_START);
      int PATTERN_STOP_INDEX = originalExpression.indexOf(PATTERN_STOP, PATTERN_START_INDEX + PATTERN_START.length());

      // End the loop is %{ or } is not in the current line
      if (PATTERN_START_INDEX < 0 || PATTERN_STOP_INDEX < 0) {
        break;
      }

      // Grab what's inside %{ }
      String grokPattern = originalExpression.substring(PATTERN_START_INDEX + PATTERN_START.length(), PATTERN_STOP_INDEX);

      // Where is the : character
      int PATTERN_DELIMITER_INDEX = grokPattern.indexOf(PATTERN_DELIMITER);

      String regexName = grokPattern;
      String groupName = null;

      if (PATTERN_DELIMITER_INDEX >= 0) {
        regexName = grokPattern.substring(0, PATTERN_DELIMITER_INDEX);
        groupName = grokPattern.substring(PATTERN_DELIMITER_INDEX + 1, grokPattern.length());
      }

      final String dictionaryValue = regexDictionary.get(regexName);

      if (dictionaryValue == null) {
        throw new GrokCompilationException("Missing value for regex name : " + regexName);
      }

      // Defer till next iteration
      if (dictionaryValue.contains(PATTERN_START)) {
        break;
      }

      String replacement = dictionaryValue;

      // Named capture group
      if (null != groupName) {
        replacement = "(?<" + groupName + ">" + dictionaryValue + ")";
      }

      originalExpression = new StringBuilder(originalExpression).replace(PATTERN_START_INDEX, PATTERN_STOP_INDEX + PATTERN_STOP.length(), replacement).toString();
    }

    return originalExpression;
  }

Digests the original expression into a pure named regex

@param originalExpression


public void addDictionary(final InputStream inputStream) {

    try {
      addDictionaryAux(new InputStreamReader(inputStream, "UTF-8"));
    } catch (IOException e) {
      throw new GrokCompilationException(e);
    }
  }

public void addDictionary ( final InputStream inputStream ) { try { addDictionaryAux ( new InputStreamReader ( inputStream , "UTF-8" ) ) ; } catch ( IOException e ) { throw new GrokCompilationException ( e ) ; } }

Loads dictionary from an input stream

This can be used to load dictionaries available in the class path <p>

@param inputStream


public void addDictionary(Reader reader) {

    try {
      addDictionaryAux(reader);
    } catch (IOException e) {
      throw new GrokCompilationException(e);
    } finally {
      IOUtils.closeQuietly(reader);
    }
  }

public void addDictionary ( Reader reader ) { try { addDictionaryAux ( reader ) ; } catch ( IOException e ) { throw new GrokCompilationException ( e ) ; } finally { IOUtils . closeQuietly ( reader ) ; } }

Adds a dictionary entry via a Reader object

@param reader


public Map<Aggregator, AggregatorData> stop() {
    Utils.checkState(started, "Not started");
    Utils.checkState(!stopped, "Already stopped");
    stopped = true;
    long currentTimeMillis = System.currentTimeMillis();
    for(Map.Entry<Aggregator, AggregatorData> e : data.entrySet()) {
      e.getValue().setTime(currentTimeMillis);
    }
    Map<Aggregator, AggregatorData> result = data;
    result = aggregateDataWindows(result);
    return result;
  }

public
Map
<
Aggregator
,
AggregatorData
>
stop
(
)
{
Utils
.
checkState
(
started
,
"Not started"
)
;
Utils
.
checkState
(
!
stopped
,
"Already stopped"
)
;
stopped
=
true
;
long
currentTimeMillis
=
System
.
currentTimeMillis
(
)
;
for
(
Map
.
Entry
<
Aggregator
,
AggregatorData
>
e
:
data
.
entrySet
(
)
)
{
e
.
getValue
(
)
.
setTime
(
currentTimeMillis
)
;
}
Map
<
Aggregator
,
AggregatorData
>
result
=
data
;
result
=
aggregateDataWindows
(
result
)
;
return
result
;
}

Stops the AggregatorDataProvider instance.

Stops the AggregatorDataProvider instance .


public Map<Aggregator, AggregatorData> roll(long newDataWindowEndTimeMillis) {
    Utils.checkState(started, "Not started");
    Utils.checkState(!stopped, "Already stopped");

    Map<Aggregator, AggregatorData> result = data;
    Map<Aggregator, AggregatorData> newData = new ConcurrentHashMap<>();
    for (Aggregator aggregator : aggregators) {
      newData.put(aggregator, aggregator.createAggregatorData(newDataWindowEndTimeMillis));
    }
    data = newData;

    Map<Aggregator, AggregatorData> oldData = result;
    // In case of sliding window, aggregate the data windows to get the result
    result = aggregateDataWindows(result);

    if (currentDataWindow != null) {
      currentDataWindow.setDataAndClose(oldData);
    }
    DataWindow newDataWindow = createDataWindow(newDataWindowEndTimeMillis);
    synchronized (dataWindowQueue) {
      dataWindowQueue.add(newDataWindow);
      dataWindowList = new ArrayList<>(dataWindowQueue);
    }
    currentDataWindow = newDataWindow;
    return result;
  }

Atomically rolls the DataWindow of all aggregators associated with the AggregatorDataProvider.

@param newDataWindowEndTimeMillis ending time of the new DataWindow.
@return a Map with all the Aggregators data of the DataWindow that closed.

Atomically rolls the DataWindow of all aggregators associated with the AggregatorDataProvider .


public String serialize(Object obj) throws ApiException {
    try {
      if (obj != null)
        return mapper.writeValueAsString(obj);
      else
        return null;
    } catch (Exception e) {
      throw new ApiException(400, e.getMessage());
    }
  }

public String serialize ( Object obj ) throws ApiException { try { if ( obj != null ) return mapper . writeValueAsString ( obj ) ; else return null ; } catch ( Exception e ) { throw new ApiException ( 400 , e . getMessage ( ) ) ; } }

Serialize the given Java object into JSON string.

Serialize the given Java object into JSON string .


public <T> T deserialize(String body, TypeRef returnType) throws ApiException {
    JavaType javaType = mapper.constructType(returnType.getType());
    try {
      return mapper.readValue(body, javaType);
    } catch (IOException e) {
      if (returnType.getType().equals(String.class))
        return (T) body;
      else
        throw new ApiException(500, e.getMessage(), null, body);
    }
  }

public < T > T deserialize ( String body , TypeRef returnType ) throws ApiException { JavaType javaType = mapper . constructType ( returnType . getType ( ) ) ; try { return mapper . readValue ( body , javaType ) ; } catch ( IOException e ) { if ( returnType . getType ( ) . equals ( String . class ) ) return ( T ) body ; else throw new ApiException ( 500 , e . getMessage ( ) , null , body ) ; } }

Deserialize the given JSON string to Java object.

@param body The JSON string
@param returnType The type to deserialize inot
@return The deserialized Java object

Deserialize the given JSON string to Java object .


public <T> T deserialize(File file, TypeRef returnType) throws ApiException {
    JavaType javaType = mapper.constructType(returnType.getType());
    try {
      return mapper.readValue(file, javaType);
    } catch (IOException e) {
      throw new ApiException(500, e.getMessage(), null, "File to read file");
    }
  }

public
<
T
>
T
deserialize
(
File
file
,
TypeRef
returnType
)
throws
ApiException
{
JavaType
javaType
=
mapper
.
constructType
(
returnType
.
getType
(
)
)
;
try
{
return
mapper
.
readValue
(
file
,
javaType
)
;
}
catch
(
IOException
e
)
{
throw
new
ApiException
(
500
,
e
.
getMessage
(
)
,
,
"File to read file"
)
;
}
}

Deserialize the given File to Java object.

@param file The File path
@param returnType The type to deserialize inot
@return The deserialized Java object

Deserialize the given File to Java object .


private void write(Record record) throws OnRecordErrorException {
    // This is a contrived example, normally you may be performing an operation that could throw
    // an exception or produce an error condition. In that case you can throw an OnRecordErrorException
    // to send this record to the error pipeline with some details.
    if (!record.has("/someField")) {
      throw new OnRecordErrorException(Errors.SAMPLE_01, record, "exception detail message.");
    }

    // TODO: write the records to your final destination
  }

Writes a single record to the destination.

@param record the record to write to the destination.
@throws OnRecordErrorException when a record cannot be written.

Writes a single record to the destination .


public static String resolveEL(ELEval elEval, ELVars variables, String val) throws ELEvalException {
    return elEval.eval(variables, val, String.class);
  }

public static String resolveEL ( ELEval elEval , ELVars variables , String val ) throws ELEvalException { return elEval . eval ( variables , val , String . class ) ; }


private static <T> Field generateInnerFieldFromTheList(
      LinkedHashMap<String, T> original,
      String innerPairFirstFieldName,
      String innerPairSecondFieldName,
      boolean isSecondFieldHiveType
  ) throws HiveStageCheckedException {
    List<Field> columnList = new LinkedList<>();
    for(Map.Entry<String,T> pair:  original.entrySet()) {
      LinkedHashMap<String, Field> entry = new LinkedHashMap<>();
      entry.put(innerPairFirstFieldName, Field.create(pair.getKey()));
      if (isSecondFieldHiveType){
        HiveTypeInfo hiveTypeInfo = (HiveTypeInfo) pair.getValue();
        entry.put(
            innerPairSecondFieldName,
            hiveTypeInfo.getHiveType().getSupport().generateHiveTypeInfoFieldForMetadataRecord(hiveTypeInfo)
        );
      } else {
        entry.put(innerPairSecondFieldName, Field.create(pair.getValue().toString())); //stored value is "INT". need to fix this
      }
      columnList.add(Field.createListMap(entry));
    }
    return !columnList.isEmpty() ? Field.create(columnList) : null;
  }

Opposite operation of extractInnerMapFromTheList.
It takes LinkedHashMap and generate a Field that contains the list.
This is to send metadata record to HMS target.
This function is called to for partition type list and partition value list.

Opposite operation of extractInnerMapFromTheList . It takes LinkedHashMap and generate a Field that contains the list . This is to send metadata record to HMS target . This function is called to for partition type list and partition value list .


public static String getTableName(Record metadataRecord) throws HiveStageCheckedException {
    if (metadataRecord.has(SEP + TABLE_FIELD)) {
      return metadataRecord.get(SEP + TABLE_FIELD).getValueAsString();
    }
    throw new HiveStageCheckedException(Errors.HIVE_17, TABLE_FIELD, metadataRecord);
  }

public static String getTableName ( Record metadataRecord ) throws HiveStageCheckedException { if ( metadataRecord . has ( SEP + TABLE_FIELD ) ) { return metadataRecord . get ( SEP + TABLE_FIELD ) . getValueAsString ( ) ; } throw new HiveStageCheckedException ( Errors . HIVE_17 , TABLE_FIELD , metadataRecord ) ; }

Get Table Name from the metadata record.
@param metadataRecord the metadata record
@return Table Name
@throws HiveStageCheckedException if the table field does not exist in the metadata record.

Get Table Name from the metadata record .


public static String getDatabaseName(Record metadataRecord) throws HiveStageCheckedException {
    if (metadataRecord.has(SEP + DATABASE_FIELD)) {
      String dbName = metadataRecord.get(SEP + DATABASE_FIELD).getValueAsString();
      return dbName.isEmpty()? DEFAULT_DBNAME : dbName;
    }
    throw new HiveStageCheckedException(Errors.HIVE_17, DATABASE_FIELD, metadataRecord);
  }

public static String getDatabaseName ( Record metadataRecord ) throws HiveStageCheckedException { if ( metadataRecord . has ( SEP + DATABASE_FIELD ) ) { String dbName = metadataRecord . get ( SEP + DATABASE_FIELD ) . getValueAsString ( ) ; return dbName . isEmpty ( ) ? DEFAULT_DBNAME : dbName ; } throw new HiveStageCheckedException ( Errors . HIVE_17 , DATABASE_FIELD , metadataRecord ) ; }

Get Database Name from the metadata record.
@param metadataRecord the metadata record
@return Database Name
@throws HiveStageCheckedException if the database name does not exist in the metadata record.

Get Database Name from the metadata record .


public static boolean getInternalField(Record metadataRecord) throws HiveStageCheckedException{
    if (metadataRecord.has(SEP + INTERNAL_FIELD)) {
      return metadataRecord.get(SEP + INTERNAL_FIELD).getValueAsBoolean();
    }
    throw new HiveStageCheckedException(Errors.HIVE_17, INTERNAL_FIELD, metadataRecord);
  }

public static boolean getInternalField ( Record metadataRecord ) throws HiveStageCheckedException { if ( metadataRecord . has ( SEP + INTERNAL_FIELD ) ) { return metadataRecord . get ( SEP + INTERNAL_FIELD ) . getValueAsBoolean ( ) ; } throw new HiveStageCheckedException ( Errors . HIVE_17 , INTERNAL_FIELD , metadataRecord ) ; }

Get internal field from the metadata record.
@param metadataRecord the metadata record
@return internal field value
@throws HiveStageCheckedException if the internal field does not exist in the metadata record.

Get internal field from the metadata record .


public static String getLocation(Record metadataRecord) throws HiveStageCheckedException{
    if (metadataRecord.has(SEP + LOCATION_FIELD)) {
      return metadataRecord.get(SEP + LOCATION_FIELD).getValueAsString();
    }
    throw new HiveStageCheckedException(Errors.HIVE_17, LOCATION_FIELD, metadataRecord);
  }

public static String getLocation ( Record metadataRecord ) throws HiveStageCheckedException { if ( metadataRecord . has ( SEP + LOCATION_FIELD ) ) { return metadataRecord . get ( SEP + LOCATION_FIELD ) . getValueAsString ( ) ; } throw new HiveStageCheckedException ( Errors . HIVE_17 , LOCATION_FIELD , metadataRecord ) ; }

Get Location from the metadata record.
@param metadataRecord the metadata record
@return location
@throws HiveStageCheckedException if the location field does not exist in the metadata record.


public static boolean getCustomLocation(Record metadataRecord) throws HiveStageCheckedException{
    if (metadataRecord.get(SEP + VERSION).getValueAsInteger() < 3) {
      return DEFAULT_CUSTOM_LOCATION;
    }

    if (metadataRecord.has(SEP + CUSTOM_LOCATION)) {
      return metadataRecord.get(SEP + CUSTOM_LOCATION).getValueAsBoolean();
    }
    throw new HiveStageCheckedException(Errors.HIVE_17, CUSTOM_LOCATION, metadataRecord);
  }

public static boolean getCustomLocation ( Record metadataRecord ) throws HiveStageCheckedException { if ( metadataRecord . get ( SEP + VERSION ) . getValueAsInteger ( ) < 3 ) { return DEFAULT_CUSTOM_LOCATION ; } if ( metadataRecord . has ( SEP + CUSTOM_LOCATION ) ) { return metadataRecord . get ( SEP + CUSTOM_LOCATION ) . getValueAsBoolean ( ) ; } throw new HiveStageCheckedException ( Errors . HIVE_17 , CUSTOM_LOCATION , metadataRecord ) ; }

Get the customLocation flag from the metadata record. This flag marks whether or not the Hive database object
is stored into a custom path on the Hadoop filesystem. In both cases, the path is stored in the 'location'
field of the metadata record.

The customLocation flag is included in partition metadata records with version > 2. For previous versions this
function returns {@code DEFAULT_CUSTOM_LOCATION} to keep the previous behavior.

@param metadataRecord the metadata record.
@return true if the location has been customized, false otherwise.
@throws HiveStageCheckedException

Get the customLocation flag from the metadata record . This flag marks whether or not the Hive database object is stored into a custom path on the Hadoop filesystem . In both cases the path is stored in the location field of the metadata record .


public static String getAvroSchema(Record metadataRecord) throws HiveStageCheckedException{
    if (metadataRecord.has(SEP + AVRO_SCHEMA)) {
      return metadataRecord.get(SEP + AVRO_SCHEMA).getValueAsString();
    }
    throw new HiveStageCheckedException(Errors.HIVE_17, AVRO_SCHEMA, metadataRecord);
  }

public static String getAvroSchema ( Record metadataRecord ) throws HiveStageCheckedException { if ( metadataRecord . has ( SEP + AVRO_SCHEMA ) ) { return metadataRecord . get ( SEP + AVRO_SCHEMA ) . getValueAsString ( ) ; } throw new HiveStageCheckedException ( Errors . HIVE_17 , AVRO_SCHEMA , metadataRecord ) ; }

Get Avro Schema from Metadata Record.
@param metadataRecord the metadata record.
@return Avro Schema
@throws HiveStageCheckedException if the avro schema field does not exist in the metadata record.


public static String getDataFormat(Record metadataRecord) throws HiveStageCheckedException {
    if (metadataRecord.get(SEP + VERSION).getValueAsInteger() == 1) {
      return DEFAULT_DATA_FORMAT;
    }

    if (metadataRecord.has(SEP + DATA_FORMAT)) {
      return metadataRecord.get(SEP + DATA_FORMAT).getValueAsString();
    }
    throw new HiveStageCheckedException(Errors.HIVE_17, DATA_FORMAT, metadataRecord);
  }

public static String getDataFormat ( Record metadataRecord ) throws HiveStageCheckedException { if ( metadataRecord . get ( SEP + VERSION ) . getValueAsInteger ( ) == 1 ) { return DEFAULT_DATA_FORMAT ; } if ( metadataRecord . has ( SEP + DATA_FORMAT ) ) { return metadataRecord . get ( SEP + DATA_FORMAT ) . getValueAsString ( ) ; } throw new HiveStageCheckedException ( Errors . HIVE_17 , DATA_FORMAT , metadataRecord ) ; }

Get DataFormat from Metadata Record.
@param metadataRecord the metadata record
@return the label of dataFormat


public static Field newSchemaMetadataFieldBuilder  (
      String database,
      String tableName,
      LinkedHashMap<String, HiveTypeInfo> columnList,
      LinkedHashMap<String, HiveTypeInfo> partitionTypeList,
      boolean internal,
      String location,
      String avroSchema,
      HMPDataFormat dataFormat
  ) throws HiveStageCheckedException  {
    LinkedHashMap<String, Field> metadata = new LinkedHashMap<>();
    metadata.put(VERSION, Field.create(SCHEMA_CHANGE_METADATA_RECORD_VERSION));
    metadata.put(METADATA_RECORD_TYPE, Field.create(MetadataRecordType.TABLE.name()));
    metadata.put(DATABASE_FIELD, Field.create(database));
    metadata.put(TABLE_FIELD, Field.create(tableName));
    metadata.put(LOCATION_FIELD, Field.create(location));
    metadata.put(DATA_FORMAT, Field.create(dataFormat.name()));

    //fill in column type list here
    metadata.put(
        COLUMNS_FIELD,
        generateInnerFieldFromTheList(
            columnList,
            COLUMN_NAME,
            TYPE_INFO,
            true
        )
    );
    //fill in partition type list here
    if (partitionTypeList != null && !partitionTypeList.isEmpty()) {
      metadata.put(
          PARTITION_FIELD,
          generateInnerFieldFromTheList(
              partitionTypeList,
              PARTITION_NAME,
              TYPE_INFO,
              true
          )
      );
    }
    metadata.put(INTERNAL_FIELD, Field.create(internal));
    metadata.put(AVRO_SCHEMA, Field.create(avroSchema));
    return Field.createListMap(metadata);
  }

public
static
Field
newSchemaMetadataFieldBuilder
(
String
database
,
String
tableName
,
LinkedHashMap
<
String
,
HiveTypeInfo
>
columnList
,
LinkedHashMap
<
String
,
HiveTypeInfo
>
partitionTypeList
,
boolean
internal
,
String
location
,
String
avroSchema
,
HMPDataFormat
dataFormat
)
throws
HiveStageCheckedException
{
LinkedHashMap
<
String
,
Field
>
metadata
=
new
LinkedHashMap
<>
(
)
;
metadata
.
put
(
VERSION
,
Field
.
create
(
SCHEMA_CHANGE_METADATA_RECORD_VERSION
)
)
;
metadata
.
put
(
METADATA_RECORD_TYPE
,
Field
.
create
(
MetadataRecordType
.
TABLE
.
name
(
)
)
)
;
metadata
.
put
(
DATABASE_FIELD
,
Field
.
create
(
database
)
)
;
metadata
.
put
(
TABLE_FIELD
,
Field
.
create
(
tableName
)
)
;
metadata
.
put
(
LOCATION_FIELD
,
Field
.
create
(
location
)
)
;
metadata
.
put
(
DATA_FORMAT
,
Field
.
create
(
dataFormat
.
name
(
)
)
)
;
//fill in column type list here
metadata
.
put
(
COLUMNS_FIELD
,
generateInnerFieldFromTheList
(
columnList
,
COLUMN_NAME
,
TYPE_INFO
,
true
)
)
;
//fill in partition type list here
if
(
partitionTypeList
!=
&&
!
partitionTypeList
.
isEmpty
(
)
)
{
metadata
.
put
(
PARTITION_FIELD
,
generateInnerFieldFromTheList
(
partitionTypeList
,
PARTITION_NAME
,
TYPE_INFO
,
true
)
)
;
}
metadata
.
put
(
INTERNAL_FIELD
,
Field
.
create
(
internal
)
)
;
metadata
.
put
(
AVRO_SCHEMA
,
Field
.
create
(
avroSchema
)
)
;
return
Field
.
createListMap
(
metadata
)
;
}

Fill in metadata to Record. This is for new schema creation.

Fill in metadata to Record . This is for new schema creation .


public static void validatePartitionInformation(
      TypeInfoCacheSupport.TypeInfo typeInfo,
      LinkedHashMap<String, String> partitionValMap,
      String qualifiedTableName
  ) throws HiveStageCheckedException {
    Set<String> partitionNamesInHive = typeInfo.getPartitionTypeInfo().keySet();
    Set<String> partitionNames = partitionValMap.keySet();
    if (!(partitionNamesInHive.size() == partitionNames.size()
        && partitionNamesInHive.containsAll(partitionNames))) {
      LOG.error(Utils.format(
          "Partition mismatch. In Hive: {}, In Record : {}",
          partitionNamesInHive.size(),
          partitionNames.size())
      );
      throw new HiveStageCheckedException(Errors.HIVE_27, qualifiedTableName);
    }
  }

Checks whether the number of partition columns and names match w.r.t hive.
@param typeInfo {@link com.streamsets.pipeline.stage.lib.hive.cache.TypeInfoCacheSupport.TypeInfo}
@param partitionValMap Map of partition name to values
@param qualifiedTableName Qualified table name.
@throws HiveStageCheckedException if there is a mismatch w.r.t hive

Checks whether the number of partition columns and names match w . r . t hive .


public static String generatePartitionPath(LinkedHashMap<String, String> partitions) {
    StringBuilder builder = new StringBuilder();
    for(Map.Entry<String, String> pair:  partitions.entrySet()) {
      builder.append(String.format(PARTITION_PATH, pair.getKey(), pair.getValue()));
    }
    return builder.toString();
  }

public static String generatePartitionPath ( LinkedHashMap < String , String > partitions ) { StringBuilder builder = new StringBuilder ( ) ; for ( Map . Entry < String , String > pair : partitions . entrySet ( ) ) { builder . append ( String . format ( PARTITION_PATH , pair . getKey ( ) , pair . getValue ( ) ) ) ; } return builder . toString ( ) ; }

Build a partition path for the external table.
@param partitions A list of key-value pair to build a partition path
@return String that represents partition path

Build a partition path for the external table .


public static String serializeSchemaToHDFS(
      UserGroupInformation loginUGI,
      final FileSystem fs,
      final String location,
      final String schemaFolder,
      final String databaseName,
      final String tableName,
      final String schemaJson
  ) throws StageException {
    String folderLocation;
    if (schemaFolder.startsWith(SEP)) {
      folderLocation = schemaFolder;
    } else {
      folderLocation = location + SEP + schemaFolder;
    }
    final Path schemasFolderPath = new Path(folderLocation);
    final String path =  folderLocation + SEP + String.format(
        AVRO_SCHEMA_FILE_FORMAT,
        databaseName,
        tableName,
        UUID.randomUUID().toString()
    );
    try {
      loginUGI.doAs(new PrivilegedExceptionAction<Void>() {
        @Override
        public Void run() throws Exception{
          if (!fs.exists(schemasFolderPath)) {
            fs.mkdirs(schemasFolderPath);
          }
          Path schemaFilePath = new Path(path);
          //This will never happen unless two HMS targets are writing, we will error out for this
          //and let user handle this via error record handling.
          if (!fs.exists(schemaFilePath)) {
            try (FSDataOutputStream os = fs.create(schemaFilePath)) {
              byte []schemaBytes = schemaJson.getBytes("UTF-8");
              os.write(schemaBytes, 0, schemaBytes.length);
            }
          } else {
            LOG.error(Utils.format("Already schema file {} exists in HDFS", path));
            throw new IOException("Already schema file exists");
          }
          return null;
        }
      });
    } catch (Exception e) {
      LOG.error("Error in Writing Schema to HDFS: " + e.toString(), e);
      throw new StageException(Errors.HIVE_18, path, e.getMessage());
    }
    return path;
  }

Returns the hdfs paths where the avro schema is stored after serializing.
Path is appended with current time so as to have an ordering.
@param schemaFolder Schema Folder (If this starts with '/' it is considered absolute)
@return Hdfs Path String.

Returns the hdfs paths where the avro schema is stored after serializing . Path is appended with current time so as to have an ordering .


@VisibleForTesting
  @SuppressWarnings("unchecked")
  int setParameters(
      int opCode,
      SortedMap<String, String> columnsToParameters,
      final Record record,
      final Connection connection,
      PreparedStatement statement
  ) throws OnRecordErrorException {
    int paramIdx = 1;

    // Set columns and their value in query. No need to perform this for delete operation.
    if(opCode != OperationType.DELETE_CODE) {
      paramIdx = setParamsToStatement(paramIdx, statement, columnsToParameters, record, connection, opCode);
    }
    // Set primary keys in WHERE clause for update and delete operations
    if(opCode != OperationType.INSERT_CODE){
      paramIdx = setPrimaryKeys(paramIdx, record, statement, opCode);
    }
    return paramIdx;
  }

Set parameters and primary keys in query.
@param opCode
@param columnsToParameters
@param record
@param connection
@param statement
@return

Set parameters and primary keys in query .


public Map<String, Object> getConfiguration () throws ApiException {
    Object postBody = null;
    byte[] postBinaryBody = null;

    // create path and map variables
    String path = "/v1/system/configuration".replaceAll("\\{format\\}","json");

    // query params
    List<Pair> queryParams = new ArrayList<Pair>();
    Map<String, String> headerParams = new HashMap<String, String>();
    Map<String, Object> formParams = new HashMap<String, Object>();

    final String[] accepts = {
      "application/json"
    };
    final String accept = apiClient.selectHeaderAccept(accepts);

    final String[] contentTypes = {

    };
    final String contentType = apiClient.selectHeaderContentType(contentTypes);

    String[] authNames = new String[] { "basic" };

    TypeRef returnType = new TypeRef<Map<String, Object>>() {};
    return apiClient.invokeAPI(path, "GET", queryParams, postBody, postBinaryBody, headerParams, formParams, accept,
        contentType, authNames, returnType);
  }

public
Map
<
String
,
Object
>
getConfiguration
(
)
throws
ApiException
{
Object
postBody
=
;
byte
[
]
postBinaryBody
=
;
// create path and map variables
String
path
=
"/v1/system/configuration"
.
replaceAll
(
"\\{format\\}"
,
"json"
)
;
// query params
List
<
Pair
>
queryParams
=
new
ArrayList
<
Pair
>
(
)
;
Map
<
String
,
String
>
headerParams
=
new
HashMap
<
String
,
String
>
(
)
;
Map
<
String
,
Object
>
formParams
=
new
HashMap
<
String
,
Object
>
(
)
;
final
String
[
]
accepts
=
{
"application/json"
}
;
final
String
accept
=
apiClient
.
selectHeaderAccept
(
accepts
)
;
final
String
[
]
contentTypes
=
{
}
;
final
String
contentType
=
apiClient
.
selectHeaderContentType
(
contentTypes
)
;
String
[
]
authNames
=
new
String
[
]
{
"basic"
}
;
TypeRef
returnType
=
new
TypeRef
<
Map
<
String
,
Object
>
>
(
)
{
}
;
return
apiClient
.
invokeAPI
(
path
,
"GET"
,
queryParams
,
postBody
,
postBinaryBody
,
headerParams
,
formParams
,
accept
,
contentType
,
authNames
,
returnType
)
;
}

Returns ALL SDC Configuration

@return Map<String, Object>


public static long parseRfc3164Time(String ts) throws OnRecordErrorException {
    LocalDateTime now = LocalDateTime.now();
    int year = now.getYear();
    ts = TWO_SPACES.matcher(ts).replaceFirst(" ");
    LocalDateTime date;
    try {
      MonthDay monthDay = MonthDay.parse(ts, rfc3164Format);
      LocalTime time = LocalTime.parse(ts, rfc3164Format);
      // this is overly complicated because of the way Java 8 Time API works, as compared to Joda
      // essentially, we just want to pull year out of "now" and set all other fields based on
      // what was parsed
      date = now;
      // zero out millis since we aren't actually parsing those
      date = date.with(ChronoField.MILLI_OF_SECOND, 0);
      // set month and day of month from parsed
      date = date.withMonth(monthDay.getMonthValue()).withDayOfMonth(monthDay.getDayOfMonth());
      // set time fields from parsed
      date = date.withHour(time.getHour()).withMinute(time.getMinute()).withSecond(time.getSecond());
    } catch (DateTimeParseException e) {
      throw new OnRecordErrorException(Errors.SYSLOG_10, ts, e);
    }
    // The RFC3164 is a bit weird date format - it contains day and month, but no year. So we have to somehow guess
    // the year. The current logic is to provide a sliding window - going 11 months to the past and 1 month to the
    // future. If the message is outside of this window, it will have incorrectly guessed year. We go 11 months to the
    // past as we're expecting that more messages will be from the past (syslog usually contains historical data).
    LocalDateTime fixed = date;
    if (fixed.isAfter(now) && fixed.minusMonths(1).isAfter(now)) {
      fixed = date.withYear(year - 1);
    } else if (fixed.isBefore(now) && fixed.plusMonths(11).isBefore(now)) {
      fixed = date.withYear(year + 1);
    }
    date = fixed;
    return date.toInstant(ZoneOffset.UTC).toEpochMilli();
  }

Parse the RFC3164 date format. This is trickier than it sounds because this
format does not specify a year so we get weird edge cases at year
boundaries. This implementation tries to "do what I mean".
@param ts RFC3164-compatible timestamp to be parsed
@return Typical (for Java) milliseconds since the UNIX epoch

Parse the RFC3164 date format . This is trickier than it sounds because this format does not specify a year so we get weird edge cases at year boundaries . This implementation tries to do what I mean .


public static String format(String template, Object... args) {
    String[] templateArr = TEMPLATES.get(template);
    if (templateArr == null) {
      // we may have a race condition here but the end result is idempotent
      templateArr = prepareTemplate(template);
      TEMPLATES.put(template, templateArr);
    }
    StringBuilder sb = new StringBuilder(template.length() * 2);
    for (int i = 0; i < templateArr.length; i++) {
      sb.append(templateArr[i]);
      if (args != null && (i < templateArr.length - 1)) {
        sb.append((i < args.length) ? args[i] : TOKEN);
      }
    }
    return sb.toString();
  }

no escaping is supported, no array content printing either.

no escaping is supported no array content printing either .


@Override
  public final BatchContext startBatch() {
    return (BatchContext) AccessController.doPrivileged(new PrivilegedAction() {
      public Object run() {
        try {
          Thread.currentThread().setContextClassLoader(mainClassLoader);
          return pushSourceContextDelegate.startBatch();
        } finally {
          Thread.currentThread().setContextClassLoader(getDefinition().getStageClassLoader());
        }
      }
    });
  }

@ Override public final BatchContext startBatch ( ) { return ( BatchContext ) AccessController . doPrivileged ( new PrivilegedAction ( ) { public Object run ( ) { try { Thread . currentThread ( ) . setContextClassLoader ( mainClassLoader ) ; return pushSourceContextDelegate . startBatch ( ) ; } finally { Thread . currentThread ( ) . setContextClassLoader ( getDefinition ( ) . getStageClassLoader ( ) ) ; } } } ) ; }

For all PushSource callbacks we have to make sure that we get back to a security context
of SDC container module, otherwise we won't be able to update state files with new offsets
and other stuff.

For all PushSource callbacks we have to make sure that we get back to a security context of SDC container module otherwise we won t be able to update state files with new offsets and other stuff .


private boolean checkRecordContainsSolrFields(
      Map<String, Field> recordFieldMap,
      Record record,
      List<String> solrFieldsMap,
      Errors errorToThrow
  ) throws StageException {
    // for (Map.Entry<String, Field> recordFieldMapEntry : recordFieldMap.entrySet())
    List<String> fieldsFound = new ArrayList<>();

    recordFieldMap.keySet().forEach(recordFieldKey -> {
      if (solrFieldsMap.contains(recordFieldKey)) {
        fieldsFound.add(recordFieldKey);
      }
    });

    // if record does not contain solr fields then process error accordingly
    if (solrFieldsMap.size() != fieldsFound.size()) {
      Set<String> missingFields = new HashSet<>();
      solrFieldsMap.forEach(requiredField -> {
        if (!fieldsFound.contains(requiredField)) {
          missingFields.add(requiredField);
        }
      });

      handleError(record, errorToThrow, Joiner.on(",").join(missingFields));

      return false;
    }

    return true;
  }

Checks whether the record contains solr fields in solrFieldsMap or not.

@param recordFieldMap Fields contained in the reocrd
@param record The whole record
@param solrFieldsMap Solr fields that will be searched in record fields
@param errorToThrow error to use if missing Solr fields are detected
@return true if record contains all solr fields in solrFieldsMap otherwise false
@throws StageException Exception thrown by handleError method call

Checks whether the record contains solr fields in solrFieldsMap or not .


private List<String> filterAutogeneratedFieldNames(List<String> fieldNames) {
    List<String> result = new ArrayList<>();
    fieldNames.forEach(name -> {
      if (!autogeneratedFieldNamesMap.contains(name)) {
        result.add(name);
      }
    });
    return result;
  }

private List < String > filterAutogeneratedFieldNames ( List < String > fieldNames ) { List < String > result = new ArrayList <> ( ) ; fieldNames . forEach ( name -> { if ( ! autogeneratedFieldNamesMap . contains ( name ) ) { result . add ( name ) ; } } ) ; return result ; }

Filter auto-generated fields from the list passed as argument.

@param fieldNames List of Solr field names.
@return A copy of {@code fieldNames} with all the auto-generated fields removed.

Filter auto - generated fields from the list passed as argument .


private void sendOnRecordErrorExceptionToHandler(Record record, Errors error, StageException ex)
      throws StageException {
    errorRecordHandler.onError(new OnRecordErrorException(
        record,
        error,
        record.getHeader().getSourceId(),
        ex.toString(),
        ex
    ));
  }

private void sendOnRecordErrorExceptionToHandler ( Record record , Errors error , StageException ex ) throws StageException { errorRecordHandler . onError ( new OnRecordErrorException ( record , error , record . getHeader ( ) . getSourceId ( ) , ex . toString ( ) , ex ) ) ; }

Send exception ex to errorRecordHandler in order to let the handler process it.

@param record The record which was being processed when the exception was thrown
@param error The error template
@param ex The exception that was thrown
@throws StageException Exception thrown by the errorRecordHandler when handling the exception ex

Send exception ex to errorRecordHandler in order to let the handler process it .


public static Optional<Dependency> parseJarName(String sourceName, String jarName) {
    if(SPECIAL_CASES.containsKey(jarName)) {
      Dependency specialCase = SPECIAL_CASES.get(jarName);
      return Optional.of(new Dependency(sourceName, specialCase.getName(), specialCase.getVersion()));
    }

    // Go over all known patterns
    for(Pattern p : PATTERNS) {
      Matcher m = p.matcher(jarName);
      if (m.matches()) {
        LOG.trace("Applied pattern '{}' to {}", p.pattern(), jarName);
        return Optional.of(new Dependency(sourceName, m.group(1), m.group(2)));
      }
    }

    // Otherwise this jar name is unknown to us
    return Optional.empty();
  }

Generate dependency from a jar file name.

Generate dependency from a jar file name .


public static Optional<Dependency> parseURL(URL url) {
    return parseJarName(url.toString(), Paths.get(url.getPath()).getFileName().toString());
  }

public static Optional < Dependency > parseURL ( URL url ) { return parseJarName ( url . toString ( ) , Paths . get ( url . getPath ( ) ) . getFileName ( ) . toString ( ) ) ; }


public static synchronized CouchbaseConnector getInstance(BaseCouchbaseConfig config, List<Stage.ConfigIssue> issues, Stage.Context context) {

    Map<String, Object> runnerSharedMap = context.getStageRunnerSharedMap();

    if(runnerSharedMap.containsKey(INSTANCE)) {
      LOG.debug("Using existing instance of CouchbaseConnector");
    } else {
      LOG.debug("CouchbaseConnector not yet instantiated. Creating new instance");
      validateConfig(config, issues, context);

      if(issues.isEmpty()) {
        runnerSharedMap.put(INSTANCE, new CouchbaseConnector(config, issues, context));
      }
    }

    return (CouchbaseConnector) runnerSharedMap.get(INSTANCE);
  }

Maintains a singleton instance of the CouchbaseConnector object per pipeline

@param config the stage configuration
@param issues the list of configuration issues
@param context the context for the stage
@return a singleton instance for Couchbase connections


public synchronized void close() {

    if(!isClosed) {
      if(bucket != null) {
        LOG.debug("Closing Couchbase bucket");
        bucket.close();
      }

      if(cluster != null) {
        LOG.debug("Disconnecting Couchbase cluster");
        cluster.disconnect();
      }

      if(env != null) {
        LOG.debug("Shutting down Couchbase environment");
        env.shutdown();
      }

      // Explicitly shutdown the RxJava scheduler threads. Not doing so will leak threads when a pipeline stops.
      // Note: this disallows restarting scheduler threads without also explicitly calling Schedulers.start()
      // LOG.debug("Stopping RxJava schedulers");
      // Schedulers.shutdown();
      isClosed = true;
    }
  }

Disconnects from Couchbase and releases all resources


private static void validateConfig(BaseCouchbaseConfig config, List<Stage.ConfigIssue> issues, Stage.Context context){
    if(config.couchbase.nodes == null) {
      issues.add(context.createConfigIssue(Groups.COUCHBASE.name(), "config.couchbase.nodes", Errors.COUCHBASE_29));
    }

    if(config.couchbase.kvTimeout < 0) {
      issues.add(context.createConfigIssue(Groups.COUCHBASE.name(), "config.couchbase.kvTimeout", Errors.COUCHBASE_30));
    }

    if(config.couchbase.connectTimeout < 0) {
      issues.add(context.createConfigIssue(Groups.COUCHBASE.name(), "config.couchbase.connectTimeout", Errors.COUCHBASE_31));
    }

    if(config.couchbase.disconnectTimeout < 0) {
      issues.add(context.createConfigIssue(Groups.COUCHBASE.name(), "config.couchbase.disconnectTimeout", Errors.COUCHBASE_32));
    }

    if(config.couchbase.tls.tlsEnabled) {
      config.couchbase.tls.init(context, Groups.COUCHBASE.name(), "config.couchbase.tls.", issues);
    }

    if(config.credentials.version == null) {
      issues.add(context.createConfigIssue(Groups.CREDENTIALS.name(), "config.credentials.version", Errors.COUCHBASE_33));
    }

    if(config.credentials.version == AuthenticationType.USER) {
      if(config.credentials.userName == null) {
        issues.add(context.createConfigIssue(Groups.CREDENTIALS.name(), "config.credentials.userName", Errors.COUCHBASE_34));
      }

      if(config.credentials.userPassword == null) {
        issues.add(context.createConfigIssue(Groups.CREDENTIALS.name(), "config.credentials.userPassword", Errors.COUCHBASE_35));
      }
    }
  }

private static void validateConfig ( BaseCouchbaseConfig config , List < Stage . ConfigIssue > issues , Stage . Context context ) { if ( config . couchbase . nodes == null ) { issues . add ( context . createConfigIssue ( Groups . COUCHBASE . name ( ) , "config.couchbase.nodes" , Errors . COUCHBASE_29 ) ) ; } if ( config . couchbase . kvTimeout < 0 ) { issues . add ( context . createConfigIssue ( Groups . COUCHBASE . name ( ) , "config.couchbase.kvTimeout" , Errors . COUCHBASE_30 ) ) ; } if ( config . couchbase . connectTimeout < 0 ) { issues . add ( context . createConfigIssue ( Groups . COUCHBASE . name ( ) , "config.couchbase.connectTimeout" , Errors . COUCHBASE_31 ) ) ; } if ( config . couchbase . disconnectTimeout < 0 ) { issues . add ( context . createConfigIssue ( Groups . COUCHBASE . name ( ) , "config.couchbase.disconnectTimeout" , Errors . COUCHBASE_32 ) ) ; } if ( config . couchbase . tls . tlsEnabled ) { config . couchbase . tls . init ( context , Groups . COUCHBASE . name ( ) , "config.couchbase.tls." , issues ) ; } if ( config . credentials . version == null ) { issues . add ( context . createConfigIssue ( Groups . CREDENTIALS . name ( ) , "config.credentials.version" , Errors . COUCHBASE_33 ) ) ; } if ( config . credentials . version == AuthenticationType . USER ) { if ( config . credentials . userName == null ) { issues . add ( context . createConfigIssue ( Groups . CREDENTIALS . name ( ) , "config.credentials.userName" , Errors . COUCHBASE_34 ) ) ; } if ( config . credentials . userPassword == null ) { issues . add ( context . createConfigIssue ( Groups . CREDENTIALS . name ( ) , "config.credentials.userPassword" , Errors . COUCHBASE_35 ) ) ; } } }

Validates connection configurations that don't require runtime exception handling
@param config the stage configuration
@param issues the list of configuration issues
@param context the context for the stage

Validates connection configurations that don t require runtime exception handling


public static String getLabelFromStringCode(String code) throws NumberFormatException {
    try {
      int intCode = Integer.parseInt(code);
      return getLabelFromIntCode(intCode);
    } catch (NumberFormatException ex) {
      throw new NumberFormatException(
          String.format("%s but received '%s'","operation code must be numeric", code)
      );
    }
  }

public
static
String
getLabelFromStringCode
(
String
code
)
throws
NumberFormatException
{
try
{
int
intCode
=
Integer
.
parseInt
(
code
)
;
return
getLabelFromIntCode
(
intCode
)
;
}
catch
(
NumberFormatException
ex
)
{
throw
new
NumberFormatException
(
String
.
format
(
"%s but received '%s'"
,
"operation code must be numeric"
,
code
)
)
;
}
}

Convert from code in String type to label
@param code
@return


private void verifyDependencyExists(
      Map<String, ConfigDefinition> definitionsMap,
      ConfigDefinition def,
      String dependsOnKey,
      Object contextMsg
  ) {
    Preconditions.checkState(definitionsMap.containsKey(dependsOnKey),
        Utils.format("Error while processing {} ConfigDef='{}'. Dependency='{}' does not exist.",
            contextMsg, def.getName(), dependsOnKey));
  }

Verify that the config definition's dependency actually maps to a valid config definition

Verify that the config definition s dependency actually maps to a valid config definition


public String read(String path, String key, long delay) {
    if (!secrets.containsKey(path)) {
      VaultClient vault = new VaultClient(getConfig());
      Secret secret;

      try {
        secret = vault.logical().read(path);
      } catch (VaultException e) {
        LOG.error(e.toString(), e);
        throw new VaultRuntimeException(e.toString());
      }

      // Record the expiration date of this lease
      String leaseId;
      if (secret.isRenewable()) {
        // Only renewable secrets seem to have a leaseId
        leaseId = secret.getLeaseId();
      } else {
        // So for non-renewable secrets we'll store the path with an extra / so that we can purge them correctly.
        leaseId = path + "/";
      }
      leases.put(leaseId, System.currentTimeMillis() + (secret.getLeaseDuration() * 1000));

      secrets.put(path, secret);

      try {
        Thread.sleep(delay);
      } catch (InterruptedException e) {
        Thread.currentThread().interrupt();
      }
    }

    Map<String, Object> data = secrets.get(path).getData();
    String value = getSecretValue(data, key).orElseThrow(() -> new VaultRuntimeException("Value not found for key"));
    LOG.trace("CredentialStore '{}' Vault, retrieved value for key '{}'", csId, key);
    return value;
  }

Reads a secret from the local cache if it hasn't expired and returns the value for the specified key.
If the secret isn't cached or has expired, it requests it from Vault again.

This version of the method will also add the specified delay in milliseconds before returning, but only
if the value wasn't already locally cached. This is because certain backends such as AWS will return
a secret (access keys for example) before they have propagated to all AWS services. For AWS a delay of up to 5
or 10 seconds may be necessary. If you receive a 403 from AWS services you probably need to increase the delay.

@param path path in Vault to read
@param key key of the property of the secret represented by the path to return
@param delay delay in milliseconds to wait before returning the value if it wasn't already cached.
@return value of the specified key for the requested secret.

Reads a secret from the local cache if it hasn t expired and returns the value for the specified key . If the secret isn t cached or has expired it requests it from Vault again .


private WriteOperationType getOperationFromHeader(Record record, String key) {

    String op = record.getHeader().getAttribute(OperationType.SDC_OPERATION_TYPE);
    if (op == null || op.isEmpty()) {
      return config.defaultWriteOperation;
    }

    int opCode;

    try {
      opCode = Integer.parseInt(op);
    } catch(NumberFormatException e) {
        LOG.debug("Unparsable CDC operation. Sending record to error.");
        handleError(record, Errors.COUCHBASE_08, e);
        return null;
      }

    switch (opCode) {
      case OperationType.INSERT_CODE:
        return WriteOperationType.INSERT;
      case OperationType.UPDATE_CODE:
        return WriteOperationType.REPLACE;
      case OperationType.UPSERT_CODE:
        return WriteOperationType.UPSERT;
      case OperationType.DELETE_CODE:
        return WriteOperationType.DELETE;
      default:
        switch (config.unsupportedOperation) {
          case DISCARD:
            LOG.debug("Unsupported CDC operation for key: {}. Discarding record per configuration.", key);
            return null;
          case TOERROR:
            LOG.debug("Unsupported CDC operation for key: {}. Sending record to error configuration.", key);
            handleError(record, Errors.COUCHBASE_09, new RuntimeException());
            return null;
          default:
            LOG.debug("Unsupported CDC operation for key: {}. Using default write operation per configuration.", key);
            return config.defaultWriteOperation;
        }
    }
  }

Evaluates the sdc.operation.type header for a record and returns the equivalent Couchbase write operation type.

@param record the record being evaluated
@param key the document key, used for logging
@return the equivalent WriteOperationType or the default write operation if sdc.operation.type is not configured.

Evaluates the sdc . operation . type header for a record and returns the equivalent Couchbase write operation type .


private Observable<AbstractDocument> writeDoc(String key, int ttl, long cas, ByteArrayOutputStream baos, Record record) {

    WriteOperationType opType = getOperationFromHeader(record, key);

    if(opType == null) {
      return Observable.empty();
    }

    AbstractDocument doc;

    if(config.dataFormat == DataFormat.JSON) {
      try {
         doc = JsonDocument.create(key, ttl, JsonObject.fromJson(baos.toString(config.dataFormatConfig.charset)), cas);
      } catch(Exception e) {
        return handleError(record, Errors.COUCHBASE_10, e);
      }
    } else {
       doc = ByteArrayDocument.create(key, ttl, baos.toByteArray(), cas);
    }

    switch (opType) {
      case DELETE: {
          LOG.debug("DELETE key: {}, TTL: {}, CAS: {}", key, ttl, cas);
          return connector.bucket().remove(doc, config.persistTo, config.replicateTo)
              .timeout(config.couchbase.kvTimeout, TimeUnit.MILLISECONDS);
      }
      case INSERT: {
          LOG.debug("INSERT key: {}, TTL: {}, CAS: {}", key, ttl, cas);
          return connector.bucket().insert(doc, config.persistTo, config.replicateTo)
              .timeout(config.couchbase.kvTimeout, TimeUnit.MILLISECONDS);
      }
      case REPLACE: {
          LOG.debug("REPLACE key: {}, TTL: {}, CAS: {}", key, ttl, cas);
          return connector.bucket().replace(doc, config.persistTo, config.replicateTo)
              .timeout(config.couchbase.kvTimeout, TimeUnit.MILLISECONDS);
      }
      case UPSERT: {
          LOG.debug("UPSERT key: {}, TTL: {}, CAS: {}", key, ttl, cas);
          return connector.bucket().upsert(doc, config.persistTo, config.replicateTo)
              .timeout(config.couchbase.kvTimeout, TimeUnit.MILLISECONDS);
      }
      default:
        return Observable.empty();
    }
  }

private
Observable
<
AbstractDocument
>
writeDoc
(
String
key
,
int
ttl
,
long
cas
,
ByteArrayOutputStream
baos
,
Record
record
)
{
WriteOperationType
opType
=
getOperationFromHeader
(
record
,
key
)
;
if
(
opType
==
)
{
return
Observable
.
empty
(
)
;
}
AbstractDocument
doc
;
if
(
config
.
dataFormat
==
DataFormat
.
JSON
)
{
try
{
doc
=
JsonDocument
.
create
(
key
,
ttl
,
JsonObject
.
fromJson
(
baos
.
toString
(
config
.
dataFormatConfig
.
charset
)
)
,
cas
)
;
}
catch
(
Exception
e
)
{
return
handleError
(
record
,
Errors
.
COUCHBASE_10
,
e
)
;
}
}
else
{
doc
=
ByteArrayDocument
.
create
(
key
,
ttl
,
baos
.
toByteArray
(
)
,
cas
)
;
}
switch
(
opType
)
{
case
DELETE
:
{
LOG
.
debug
(
"DELETE key: {}, TTL: {}, CAS: {}"
,
key
,
ttl
,
cas
)
;
return
connector
.
bucket
(
)
.
remove
(
doc
,
config
.
persistTo
,
config
.
replicateTo
)
.
timeout
(
config
.
couchbase
.
kvTimeout
,
TimeUnit
.
MILLISECONDS
)
;
}
case
INSERT
:
{
LOG
.
debug
(
"INSERT key: {}, TTL: {}, CAS: {}"
,
key
,
ttl
,
cas
)
;
return
connector
.
bucket
(
)
.
insert
(
doc
,
config
.
persistTo
,
config
.
replicateTo
)
.
timeout
(
config
.
couchbase
.
kvTimeout
,
TimeUnit
.
MILLISECONDS
)
;
}
case
REPLACE
:
{
LOG
.
debug
(
"REPLACE key: {}, TTL: {}, CAS: {}"
,
key
,
ttl
,
cas
)
;
return
connector
.
bucket
(
)
.
replace
(
doc
,
config
.
persistTo
,
config
.
replicateTo
)
.
timeout
(
config
.
couchbase
.
kvTimeout
,
TimeUnit
.
MILLISECONDS
)
;
}
case
UPSERT
:
{
LOG
.
debug
(
"UPSERT key: {}, TTL: {}, CAS: {}"
,
key
,
ttl
,
cas
)
;
return
connector
.
bucket
(
)
.
upsert
(
doc
,
config
.
persistTo
,
config
.
replicateTo
)
.
timeout
(
config
.
couchbase
.
kvTimeout
,
TimeUnit
.
MILLISECONDS
)
;
}
default
:
return
Observable
.
empty
(
)
;
}
}

Executes a document write operation.

@param key the document key
@param ttl the document expiry ttl
@param cas the compare-and-swap (CAS) value to apply to the write
@param baos the raw document content
@param record the record being written
@return an observable for the document write or an empty observable on error


private Observable<DocumentFragment<Mutation>> buildSubdocMutation(AsyncMutateInBuilder mutation, int ttl, long cas,
      boolean upsertDoc) {

    return mutation
        .upsertDocument(upsertDoc)
        .withExpiry(ttl)
        .withCas(cas)
        .withDurability(config.persistTo, config.replicateTo)
        .execute()
        .timeout(config.couchbase.kvTimeout, TimeUnit.MILLISECONDS);
  }

private Observable < DocumentFragment < Mutation > > buildSubdocMutation ( AsyncMutateInBuilder mutation , int ttl , long cas , boolean upsertDoc ) { return mutation . upsertDocument ( upsertDoc ) . withExpiry ( ttl ) . withCas ( cas ) . withDurability ( config . persistTo , config . replicateTo ) . execute ( ) . timeout ( config . couchbase . kvTimeout , TimeUnit . MILLISECONDS ) ; }

Applies standard options to sub-document mutations

@param mutation the mutation to be augmented
@param ttl the time to live to be set on the parent document
@param cas the compare-and-swap (CAS) value to apply to the write
@param upsertDoc defines whether the parent document should be created if it does not exist
@return an augmented mutation

Applies standard options to sub - document mutations


@Override
  protected String makeAlterTableSqlString(
      String schema, String tableName, LinkedHashMap<String, JdbcTypeInfo> columnDiff
  ) {
    String tableSchema = (schema == null) ? getDefaultSchema() : schema;
    StringBuilder sqlString = new StringBuilder();
    boolean first = true;
    for (Map.Entry<String, JdbcTypeInfo> entry : columnDiff.entrySet()) {
      if (first) {
        first = false;
      } else {
        sqlString.append("\n");
      }
      sqlString
          .append(ALTER_TABLE)
          .append(" ");
      if (tableSchema != null) {
        sqlString.append(tableSchema);
        sqlString.append(".");
      }
      sqlString.append(tableName)
          .append(" ")
          .append("ADD COLUMN")
          .append(" ")
          .append(entry.getKey())
          .append(" ")
          .append(entry.getValue().toString())
          .append(";");
    }

    return sqlString.toString();
  }

@
Override
protected
String
makeAlterTableSqlString
(
String
schema
,
String
tableName
,
LinkedHashMap
<
String
,
JdbcTypeInfo
>
columnDiff
)
{
String
tableSchema
=
(
schema
==
)
getDefaultSchema
(
)
:
schema
;
StringBuilder
sqlString
=
new
StringBuilder
(
)
;
boolean
first
=
true
;
for
(
Map
.
Entry
<
String
,
JdbcTypeInfo
>
entry
:
columnDiff
.
entrySet
(
)
)
{
if
(
first
)
{
first
=
false
;
}
else
{
sqlString
.
append
(
"\n"
)
;
}
sqlString
.
append
(
ALTER_TABLE
)
.
append
(
" "
)
;
if
(
tableSchema
!=
)
{
sqlString
.
append
(
tableSchema
)
;
sqlString
.
append
(
"."
)
;
}
sqlString
.
append
(
tableName
)
.
append
(
" "
)
.
append
(
"ADD COLUMN"
)
.
append
(
" "
)
.
append
(
entry
.
getKey
(
)
)
.
append
(
" "
)
.
append
(
entry
.
getValue
(
)
.
toString
(
)
)
.
append
(
";"
)
;
}
return
sqlString
.
toString
(
)
;
}

H2 wants an ALTER TABLE command per column


public void configure(Map<String, ?> configs, final String loginContextName) {
    super.configure(configs, loginContextName);
    this.loginContextName = loginContextName;
    this.ticketRenewWindowFactor = (Double) configs.get(SaslConfigs.SASL_KERBEROS_TICKET_RENEW_WINDOW_FACTOR);
    this.ticketRenewJitter = (Double) configs.get(SaslConfigs.SASL_KERBEROS_TICKET_RENEW_JITTER);
    this.minTimeBeforeRelogin = (Long) configs.get(SaslConfigs.SASL_KERBEROS_MIN_TIME_BEFORE_RELOGIN);
    this.kinitCmd = (String) configs.get(SaslConfigs.SASL_KERBEROS_KINIT_CMD);
    this.serviceName = getServiceName(configs, loginContextName);
  }

Login constructor. The constructor starts the thread used
to periodically re-login to the Kerberos Ticket Granting Server.
@param loginContextName
name of section in JAAS file that will be use to login.
Passed as first param to javax.security.auth.login.LoginContext().
@param configs configure Login with the given key-value pairs.
@throws javax.security.auth.login.LoginException
Thrown if authentication fails.

Login constructor . The constructor starts the thread used to periodically re - login to the Kerberos Ticket Granting Server .


public OffsetAndResult<Map.Entry> take() {
    if (producerError != null) {
      throw new RuntimeException(Utils.format("Producer encountered error: {}", producerError), producerError);
    }
    if (consumerError != null) {
      throw new RuntimeException(Utils.format("Consumer encountered error: {}", consumerError), consumerError);
    }
    try {
      Utils.checkState(batchCommitted, "Cannot take messages when last batch is uncommitted");
      while (running) {
        for (ControlChannel.Message controlMessage : controlChannel.getConsumerMessages()) {
          switch (controlMessage.getType()) {
            case PRODUCER_COMPLETE:
              // producer is complete, empty channel and afterwards return null
              running = false;
              break;
            case PRODUCER_ERROR:
              running = false;
              Throwable throwable = (Throwable) controlMessage.getPayload();
              producerError = throwable;
              throw new ProducerRuntimeException(Utils.format("Producer encountered error: {}", throwable), throwable);
            default:
              String msg = Utils.format("Illegal control message type: '{}'", controlMessage.getType());
              throw new IllegalStateException(msg);
          }
        }
        OffsetAndResult<Map.Entry> batch = dataChannel.take(10, TimeUnit.MILLISECONDS);
        LOG.trace("Received batch: {}", batch);
        if (batch != null) {
          batchCommitted = false; // got a new batch
          return batch;
        }
      }
      LOG.trace("Returning null");
      return null;
    } catch (Throwable throwable) {
      if (!(throwable instanceof ProducerRuntimeException)) {
        String msg = "Error caught in consumer: " + throwable;
        LOG.error(msg, throwable);
        error(throwable);
      }
      throw Throwables.propagate(throwable);
    }
  }

Consumes messages off the queue. Returns null when the producer
has indicated it is complete and throws an exception
when the consumer producer has indicated it is in error.

Consumes messages off the queue . Returns null when the producer has indicated it is complete and throws an exception when the consumer producer has indicated it is in error .


public void commit(String offset) {
    batchCommitted = true;
    LOG.trace("Last committed offset '{}', attempting to commit '{}'", lastCommittedOffset, offset);
    Utils.checkState(null != lastCommittedOffset, "Last committed offset cannot be null");
    controlChannel.consumerCommit(offset);
    lastCommittedOffset = offset;
  }

Commit the offset. Required after take has returned a non-null value.

Commit the offset . Required after take has returned a non - null value .


public void error(Throwable throwable) {
    if (consumerError == null) {
      consumerError = throwable;
      controlChannel.consumerError(throwable);
    }
  }

public void error ( Throwable throwable ) { if ( consumerError == null ) { consumerError = throwable ; controlChannel . consumerError ( throwable ) ; } }

Send a control message indicating the consumer has encountered an error.

Send a control message indicating the consumer has encountered an error .


public void injectStage(Object stage, StageDefinition stageDef, StageConfiguration stageConf, Map<String, Object> constants, List<Issue> issues) {
    injectConfigsToObject(stage, new StageInjectorContext(stageDef, stageConf, constants, issues));
  }

public void injectStage ( Object stage , StageDefinition stageDef , StageConfiguration stageConf , Map < String , Object > constants , List < Issue > issues ) { injectConfigsToObject ( stage , new StageInjectorContext ( stageDef , stageConf , constants , issues ) ) ; }

Inject config values to given Stage.

@param stage Stage instance
@param stageDef Definition for given stage
@param stageConf Actual configuration values
@param constants Pipeline constants (parameters)
@param issues List into which issues will be added


public DefinitionsJson getDefinitions (HideStage.Type hideStage) throws ApiException {
    Object postBody = null;
    byte[] postBinaryBody = null;

    // create path and map variables
    String path = "/v1/definitions".replaceAll("\\{format\\}","json");

    // query params
    List<Pair> queryParams = new ArrayList<Pair>();
    if (hideStage != null) {
      queryParams.add(new Pair("hideStage", hideStage.name()));
    }
    Map<String, String> headerParams = new HashMap<String, String>();
    Map<String, Object> formParams = new HashMap<String, Object>();

    final String[] accepts = {
      "application/json"
    };
    final String accept = apiClient.selectHeaderAccept(accepts);

    final String[] contentTypes = {

    };
    final String contentType = apiClient.selectHeaderContentType(contentTypes);

    String[] authNames = new String[] { "basic" };

    TypeRef returnType = new TypeRef<DefinitionsJson>() {};
    return apiClient.invokeAPI(path, "GET", queryParams, postBody, postBinaryBody, headerParams,
        formParams, accept, contentType, authNames, returnType);
  }

public
DefinitionsJson
getDefinitions
(
HideStage
.
Type
hideStage
)
throws
ApiException
{
Object
postBody
=
;
byte
[
]
postBinaryBody
=
;
// create path and map variables
String
path
=
"/v1/definitions"
.
replaceAll
(
"\\{format\\}"
,
"json"
)
;
// query params
List
<
Pair
>
queryParams
=
new
ArrayList
<
Pair
>
(
)
;
if
(
hideStage
!=
)
{
queryParams
.
add
(
new
Pair
(
"hideStage"
,
hideStage
.
name
(
)
)
)
;
}
Map
<
String
,
String
>
headerParams
=
new
HashMap
<
String
,
String
>
(
)
;
Map
<
String
,
Object
>
formParams
=
new
HashMap
<
String
,
Object
>
(
)
;
final
String
[
]
accepts
=
{
"application/json"
}
;
final
String
accept
=
apiClient
.
selectHeaderAccept
(
accepts
)
;
final
String
[
]
contentTypes
=
{
}
;
final
String
contentType
=
apiClient
.
selectHeaderContentType
(
contentTypes
)
;
String
[
]
authNames
=
new
String
[
]
{
"basic"
}
;
TypeRef
returnType
=
new
TypeRef
<
DefinitionsJson
>
(
)
{
}
;
return
apiClient
.
invokeAPI
(
path
,
"GET"
,
queryParams
,
postBody
,
postBinaryBody
,
headerParams
,
formParams
,
accept
,
contentType
,
authNames
,
returnType
)
;
}

Returns pipeline & stage configuration definitions
This will fetch defintions based on the hideStage filter

@return DefinitionsJson

Returns pipeline &amp ; stage configuration definitions This will fetch defintions based on the hideStage filter


private LineagePublisherDefinition getDefinition(String name) {
    String defConfig = LineagePublisherConstants.configDef(name);
    String publisherDefinition = configuration.get(defConfig, null);
    if(StringUtils.isEmpty(publisherDefinition)) {
      throw new IllegalArgumentException(Utils.format("Missing definition '{}'", defConfig));
    }

    String []lineagePluginDefs = publisherDefinition.split("::");
    if(lineagePluginDefs.length != 2) {
      throw new IllegalStateException(Utils.format(
        "Invalid definition '{}', expected $libraryName::$publisherName",
        publisherDefinition
      ));
    }

    LineagePublisherDefinition def = stageLibraryTask.getLineagePublisherDefinition(
      lineagePluginDefs[0], // Library
      lineagePluginDefs[1]  // Plugin name
    );
    if(def == null) {
      throw new IllegalStateException(Utils.format("Can't find publisher '{}'", publisherDefinition));
    }

    return def;
  }

Parse given configuration declaration of lineage plugin and return appropriate definition.

This method will throw exceptions on all error paths.

Parse given configuration declaration of lineage plugin and return appropriate definition .


private static String formatName(String columnName, boolean caseSensitive) {
    String returnValue = format(columnName);
    if (caseSensitive) {
      return returnValue;
    }
    return returnValue.toUpperCase();
  }

private static String formatName ( String columnName , boolean caseSensitive ) { String returnValue = format ( columnName ) ; if ( caseSensitive ) { return returnValue ; } return returnValue . toUpperCase ( ) ; }

Format column names based on whether they are case-sensitive

Format column names based on whether they are case - sensitive


private static String formatValue(String value) {
    // The value can either be null (if the IS keyword is present before it or just a NULL string with no quotes)
    if (value == null || NULL_STRING.equalsIgnoreCase(value)) {
      return null;
    }
    String returnValue = format(value);
    return returnValue.replaceAll("''", "'");
  }


public PipelineConfigurationJson getPipelineInfo (String pipelineId, String rev, String get, Boolean attachment)
      throws ApiException {
    Object postBody = null;
    byte[] postBinaryBody = null;

    // verify the required parameter 'pipelineId' is set
    if (pipelineId == null) {
      throw new ApiException(400, "Missing the required parameter 'pipelineId' when calling getPipelineInfo");
    }

    // create path and map variables
    String path = "/v1/pipeline/{pipelineId}".replaceAll("\\{format\\}","json")
        .replaceAll("\\{" + "pipelineId" + "\\}", apiClient.escapeString(pipelineId.toString()));

    // query params
    List<Pair> queryParams = new ArrayList<Pair>();
    Map<String, String> headerParams = new HashMap<String, String>();
    Map<String, Object> formParams = new HashMap<String, Object>();

    queryParams.addAll(apiClient.parameterToPairs("", "rev", rev));

    queryParams.addAll(apiClient.parameterToPairs("", "get", get));

    queryParams.addAll(apiClient.parameterToPairs("", "attachment", attachment));

    final String[] accepts = {
        "application/json"
    };
    final String accept = apiClient.selectHeaderAccept(accepts);

    final String[] contentTypes = {

    };
    final String contentType = apiClient.selectHeaderContentType(contentTypes);

    String[] authNames = new String[] { "basic" };

    TypeRef returnType = new TypeRef<PipelineConfigurationJson>() {};
    return apiClient.invokeAPI(path, "GET", queryParams, postBody, postBinaryBody, headerParams, formParams, accept,
        contentType, authNames, returnType);
  }

Find Pipeline Configuration by name and revision

@param pipelineId
@param rev
@param get
@param attachment
@return PipelineConfigurationJson


public PipelineFragmentEnvelopeJson createDraftPipelineFragment (
      String fragmentId,
      String description,
      List<StageConfigurationJson> stageInstances
  ) throws ApiException {
    Object postBody = stageInstances;
    byte[] postBinaryBody = null;

    // verify the required parameter 'pipelineId' is set
    if (fragmentId == null) {
      throw new ApiException(400, "Missing the required parameter 'fragmentId' when calling createPipelineFragment");
    }

    // create path and map variables
    String path = "/v1/fragment/{fragmentId}".replaceAll("\\{format\\}","json")
        .replaceAll("\\{" + "fragmentId" + "\\}", apiClient.escapeString(fragmentId.toString()));

    // query params
    List<Pair> queryParams = new ArrayList<Pair>();
    Map<String, String> headerParams = new HashMap<String, String>();
    Map<String, Object> formParams = new HashMap<String, Object>();

    queryParams.addAll(apiClient.parameterToPairs("", "description", description));
    queryParams.addAll(apiClient.parameterToPairs("", "draft", true));

    final String[] accepts = {
        "application/json"
    };
    final String accept = apiClient.selectHeaderAccept(accepts);

    final String[] contentTypes = {

    };
    final String contentType = apiClient.selectHeaderContentType(contentTypes);

    String[] authNames = new String[] { "basic" };

    TypeRef returnType = new TypeRef<PipelineFragmentEnvelopeJson>() {};
    return apiClient.invokeAPI(path, "PUT", queryParams, postBody, postBinaryBody, headerParams, formParams,
        accept, contentType, authNames, returnType);
  }

Add a new Pipeline Fragment Configuration to the store

@param fragmentId Frgament Id
@param description Description
@return PipelineFragmentEnvelopeJson


public List<PipelineInfoJson> getPipelines (
      String filterText,
      String label,
      int offset,
      int len,
      PipelineOrderByFields orderBy,
      Order order,
      boolean includeStatus
  ) throws ApiException {
    Object postBody = null;
    byte[] postBinaryBody = null;

    // create path and map variables
    String path = "/v1/pipelines".replaceAll("\\{format\\}","json");

    // query params
    List<Pair> queryParams = new ArrayList<Pair>();
    Map<String, String> headerParams = new HashMap<String, String>();
    Map<String, Object> formParams = new HashMap<String, Object>();

    queryParams.addAll(apiClient.parameterToPairs("", "filterText", filterText));
    queryParams.addAll(apiClient.parameterToPairs("", "label", label));
    queryParams.addAll(apiClient.parameterToPairs("", "offset", offset));
    queryParams.addAll(apiClient.parameterToPairs("", "len", len));
    queryParams.addAll(apiClient.parameterToPairs("", "orderBy", orderBy));
    queryParams.addAll(apiClient.parameterToPairs("", "order", order));

    final String[] accepts = {
        "application/json"
    };
    final String accept = apiClient.selectHeaderAccept(accepts);

    final String[] contentTypes = {

    };
    final String contentType = apiClient.selectHeaderContentType(contentTypes);

    String[] authNames = new String[] { "basic" };
    TypeRef returnType = new TypeRef<List<PipelineInfoJson>>() {};
    return apiClient.invokeAPI(path, "GET", queryParams, postBody, postBinaryBody, headerParams, formParams, accept,
        contentType, authNames, returnType);
  }

public
List
<
PipelineInfoJson
>
getPipelines
(
String
filterText
,
String
label
,
int
offset
,
int
len
,
PipelineOrderByFields
orderBy
,
Order
order
,
boolean
includeStatus
)
throws
ApiException
{
Object
postBody
=
;
byte
[
]
postBinaryBody
=
;
// create path and map variables
String
path
=
"/v1/pipelines"
.
replaceAll
(
"\\{format\\}"
,
"json"
)
;
// query params
List
<
Pair
>
queryParams
=
new
ArrayList
<
Pair
>
(
)
;
Map
<
String
,
String
>
headerParams
=
new
HashMap
<
String
,
String
>
(
)
;
Map
<
String
,
Object
>
formParams
=
new
HashMap
<
String
,
Object
>
(
)
;
queryParams
.
addAll
(
apiClient
.
parameterToPairs
(
""
,
"filterText"
,
filterText
)
)
;
queryParams
.
addAll
(
apiClient
.
parameterToPairs
(
""
,
"label"
,
label
)
)
;
queryParams
.
addAll
(
apiClient
.
parameterToPairs
(
""
,
"offset"
,
offset
)
)
;
queryParams
.
addAll
(
apiClient
.
parameterToPairs
(
""
,
"len"
,
len
)
)
;
queryParams
.
addAll
(
apiClient
.
parameterToPairs
(
""
,
"orderBy"
,
orderBy
)
)
;
queryParams
.
addAll
(
apiClient
.
parameterToPairs
(
""
,
"order"
,
order
)
)
;
final
String
[
]
accepts
=
{
"application/json"
}
;
final
String
accept
=
apiClient
.
selectHeaderAccept
(
accepts
)
;
final
String
[
]
contentTypes
=
{
}
;
final
String
contentType
=
apiClient
.
selectHeaderContentType
(
contentTypes
)
;
String
[
]
authNames
=
new
String
[
]
{
"basic"
}
;
TypeRef
returnType
=
new
TypeRef
<
List
<
PipelineInfoJson
>
>
(
)
{
}
;
return
apiClient
.
invokeAPI
(
path
,
"GET"
,
queryParams
,
postBody
,
postBinaryBody
,
headerParams
,
formParams
,
accept
,
contentType
,
authNames
,
returnType
)
;
}

Returns all Pipeline Configuration Info

@return List<PipelineInfoJson>


public PipelineFragmentEnvelopeJson importPipelineFragment (
      String fragmentId,
      boolean draft,
      boolean includeLibraryDefinitions,
      PipelineFragmentEnvelopeJson fragmentEnvelope
  ) throws ApiException {
    Object postBody = fragmentEnvelope;
    byte[] postBinaryBody = null;

    // verify the required parameter 'fragmentId' is set
    if (fragmentId == null) {
      throw new ApiException(400, "Missing the required parameter 'fragmentId' when calling importPipelineFragment");
    }

    // verify the required parameter 'fragmentEnvelope' is set
    if (fragmentEnvelope == null) {
      throw new ApiException(
          400,
          "Missing the required parameter 'pipelineEnvelope' when calling importPipelineFragment"
      );
    }

    // create path and map variables
    String path = "/v1/fragment/{fragmentId}/import".replaceAll("\\{format\\}","json")
        .replaceAll("\\{" + "fragmentId" + "\\}", apiClient.escapeString(fragmentId.toString()));

    // query params
    List<Pair> queryParams = new ArrayList<Pair>();
    Map<String, String> headerParams = new HashMap<String, String>();
    Map<String, Object> formParams = new HashMap<String, Object>();

    queryParams.addAll(apiClient.parameterToPairs("", "draft", draft));
    queryParams.addAll(apiClient.parameterToPairs("", "includeLibraryDefinitions", includeLibraryDefinitions));

    final String[] accepts = {
        "application/json"
    };
    final String accept = apiClient.selectHeaderAccept(accepts);

    final String[] contentTypes = {
        "application/json"
    };
    final String contentType = apiClient.selectHeaderContentType(contentTypes);

    String[] authNames = new String[] { "basic" };

    TypeRef returnType = new TypeRef<PipelineFragmentEnvelopeJson>() {};
    return apiClient.invokeAPI(path, "POST", queryParams, postBody, postBinaryBody, headerParams, formParams,
        accept, contentType, authNames, returnType);
  }

Import Pipeline Fragment Configuration & Rules

@param fragmentId Fragment  Id
@return fragmentEnvelope


public String getFilePath(
      String dirPathTemplate,
      Record record,
      Date recordTime
  ) throws StageException {
    String dirPath;
    // get directory path
    if (dirPathTemplateInHeader) {
      dirPath = record.getHeader().getAttribute(DataLakeTarget.TARGET_DIRECTORY_HEADER);
      Utils.checkArgument(!(dirPath == null || dirPath.isEmpty()), "Directory Path cannot be null");
    } else {
      dirPath = resolvePath(dirPathTemplateEval, dirPathTemplateVars, dirPathTemplate, recordTime, record);
    }

    // SDC-5492: replace "//" to "/" in file path
    dirPath = dirPath.replaceAll("/+","/");
    if (dirPath.endsWith("/")) {
      dirPath = dirPath.substring(0, dirPath.length()-1);
    }

    return outputStreamHelper.getTempFilePath(dirPath, record, recordTime);
  }

Returns the temp file path to write records to
@param dirPathTemplate Directory Template
@param record Record for which file path is calculated
@param recordTime Record time
@return Temporary File Path to write records to
@throws StageException


int getOperationFromRecord(
      Record record,
      int defaultOpCode,
      UnsupportedOperationAction unsupportedAction,
      List<OnRecordErrorException> errorRecords
  ) {
    String op = record.getHeader().getAttribute(OperationType.SDC_OPERATION_TYPE);
    int opCode = -1; // unsupported

    if (Strings.isNullOrEmpty(op)) {
      return defaultOpCode;
    }

    // Check if the operation code from header attribute is valid
    try {
      opCode = JDBCOperationType.convertToIntCode(op);
    } catch (NumberFormatException | UnsupportedOperationException ex) {
      LOG.debug(
          "Operation obtained from record is not supported. Handle by UnsupportedOperationAction {}. {}",
          unsupportedAction.getLabel(),
          ex
      );
      switch (unsupportedAction) {
        case SEND_TO_ERROR:
          LOG.debug("Sending record to error due to unsupported operation {}", op);
          errorRecords.add(new OnRecordErrorException(record, JdbcErrors.JDBC_70, op));
          break;
        case USE_DEFAULT:
          opCode = defaultOpCode;
          break;
        case DISCARD:
        default: // unknown action
          LOG.debug("Discarding record with unsupported operation {}", op);
      }
    }
    return opCode;
  }

Get the numeric operation code from record header. The default code is
used if the operation code is not found in the header. This can be
overwritten in inherited classes.

@param record the record to find the operation code
@param defaultOpCode the default operation code
@param unsupportedAction the action to take for unsupported code
@param errorRecords the list to take error records
@return the numeric operation code or -1 for unsupported operation

Get the numeric operation code from record header . The default code is used if the operation code is not found in the header . This can be overwritten in inherited classes .


String getFieldPath(String columnName, Map<String, String> columnsToField, int op){
    return columnsToField.get(columnName);
  }

String getFieldPath ( String columnName , Map < String , String > columnsToField , int op ) { return columnsToField . get ( columnName ) ; }

This function simply returns field path in record for the corresponding column name.
This is needed because records generated by CDC origins store data in different location
for different operation. Subclasses will override this function and implement special handling.
@param columnName column name
@param columnsToField mapping of column name to field path
@param op  Not used here but is used in subclass.
@return

This function simply returns field path in record for the corresponding column name . This is needed because records generated by CDC origins store data in different location for different operation . Subclasses will override this function and implement special handling .


public EnrichedEvent poll(long timeout, TimeUnit unit) throws StageException {
    try {
      return queue.poll(timeout, unit);
    } catch (InterruptedException e) {
      LOG.error(Errors.MYSQL_001.getMessage(), e.toString(), e);
      Thread.currentThread().interrupt();
      throw new StageException(Errors.MYSQL_001, e.toString(), e);
    }
  }

public EnrichedEvent poll ( long timeout , TimeUnit unit ) throws StageException { try { return queue . poll ( timeout , unit ) ; } catch ( InterruptedException e ) { LOG . error ( Errors . MYSQL_001 . getMessage ( ) , e . toString ( ) , e ) ; Thread . currentThread ( ) . interrupt ( ) ; throw new StageException ( Errors . MYSQL_001 , e . toString ( ) , e ) ; } }

Read next event from buffer with respect to maximum timeout.
@param timeout timeout.
@param unit timeout time unit.
@return next event of null
@throws StageException

Read next event from buffer with respect to maximum timeout .


public T getRunner() throws PipelineRuntimeException {
    validateNotDestroyed();

    try {
      return queue.take().runner;
    } catch (InterruptedException e) {
      throw new PipelineRuntimeException(ContainerError.CONTAINER_0801, e);
    } finally {
      runtimeStats.setAvailableRunners(queue.size());
      histogram.update(queue.size());
    }
  }

public T getRunner ( ) throws PipelineRuntimeException { validateNotDestroyed ( ) ; try { return queue . take ( ) . runner ; } catch ( InterruptedException e ) { throw new PipelineRuntimeException ( ContainerError . CONTAINER_0801 , e ) ; } finally { runtimeStats . setAvailableRunners ( queue . size ( ) ) ; histogram . update ( queue . size ( ) ) ; } }

Get exclusive runner for use.

@return Runner that is not being used by anyone else.
@throws PipelineRuntimeException Thrown in case that current thread is unexpectedly interrupted


public T getIdleRunner(long idleTime) {
    // Take the first runner
    QueueItem<T> item = queue.poll();

    // All runners might be currently in use, which is fine in this case.
    if(item == null) {
      return null;
    }

    // If the runner wasn't idle for the expected time, we need to put it back to the queue (it will be added to the
    // begging again).
    if((System.currentTimeMillis() - item.timestamp) < idleTime) {
      queue.add(item);
      return null;
    }

    // Otherwise we do have runner that hasn't been used for at least idleTime, so we can return it now
    return item.runner;
  }

Return a runner that haven't been used at least for the configured number of milliseconds.

@param idleTime Number of milliseconds when a runner is considered an "idle"
@return First runner that fits such criteria or null if there is no such runner

Return a runner that haven t been used at least for the configured number of milliseconds .


public void returnRunner(T runner) throws PipelineRuntimeException {
    validateNotDestroyed();

    queue.add(new QueueItem<>(runner));
    runtimeStats.setAvailableRunners(queue.size());
    histogram.update(queue.size());
  }

public void returnRunner ( T runner ) throws PipelineRuntimeException { validateNotDestroyed ( ) ; queue . add ( new QueueItem <> ( runner ) ) ; runtimeStats . setAvailableRunners ( queue . size ( ) ) ; histogram . update ( queue . size ( ) ) ; }

Return given runner back to the pool.

@param runner Runner to be returned


public void destroy() throws PipelineRuntimeException {
    // Firstly set this runner as destroyed
    destroyed.set(true);

    // Validate that this thread pool have all runners back, otherwise we're missing something and that is sign of
    // a trouble.
    if(queue.size() < runtimeStats.getTotalRunners()) {
      throw new PipelineRuntimeException(ContainerError.CONTAINER_0802, queue.size(), runtimeStats.getTotalRunners());
    }
  }

Destroy only the pool itself - not the individual pipe runners.

This method will also validate that all runners were properly returned to the pool. PipelineRuntimeException will
be thrown if some runners are still running.

@throws PipelineRuntimeException

Destroy only the pool itself - not the individual pipe runners .


private void validateNotDestroyed() throws PipelineRuntimeException {
    if(destroyed.get()) {
      throw new PipelineRuntimeException(ContainerError.CONTAINER_0803, queue.size(), runtimeStats.getTotalRunners());
    }
  }

private void validateNotDestroyed ( ) throws PipelineRuntimeException { if ( destroyed . get ( ) ) { throw new PipelineRuntimeException ( ContainerError . CONTAINER_0803 , queue . size ( ) , runtimeStats . getTotalRunners ( ) ) ; } }

Throw an exception if the runner was already destroyed.

@throws PipelineRuntimeException

Throw an exception if the runner was already destroyed .


public static long checkStreamExists(
      ClientConfiguration awsClientConfig,
      KinesisConfigBean conf,
      String streamName,
      List<Stage.ConfigIssue> issues,
      Stage.Context context
  ) {
    long numShards = 0;

    try {
      numShards = getShardCount(awsClientConfig, conf, streamName);
    } catch (AmazonClientException|StageException e) {
      LOG.error(Errors.KINESIS_01.getMessage(), e.toString(), e);
      issues.add(context.createConfigIssue(
          Groups.KINESIS.name(),
          KINESIS_CONFIG_BEAN + ".streamName", Errors.KINESIS_01, e.toString()
      ));
    }
    return numShards;
  }

public static long checkStreamExists ( ClientConfiguration awsClientConfig , KinesisConfigBean conf , String streamName , List < Stage . ConfigIssue > issues , Stage . Context context ) { long numShards = 0 ; try { numShards = getShardCount ( awsClientConfig , conf , streamName ) ; } catch ( AmazonClientException | StageException e ) { LOG . error ( Errors . KINESIS_01 . getMessage ( ) , e . toString ( ) , e ) ; issues . add ( context . createConfigIssue ( Groups . KINESIS . name ( ) , KINESIS_CONFIG_BEAN + ".streamName" , Errors . KINESIS_01 , e . toString ( ) ) ) ; } return numShards ; }

Checks for existence of the requested stream and adds
any configuration issues to the list.
@param conf
@param streamName
@param issues
@param context

Checks for existence of the requested stream and adds any configuration issues to the list .


public static String getLastShardId(
      ClientConfiguration awsClientConfig,
      KinesisConfigBean conf,
      String streamName
  ) throws StageException {
    AmazonKinesis kinesisClient = getKinesisClient(awsClientConfig, conf);

    String lastShardId = null;
    try {
      StreamDescription description;
      do {
        if (lastShardId == null) {
          description = kinesisClient.describeStream(streamName).getStreamDescription();
        } else {
          description = kinesisClient.describeStream(streamName, lastShardId).getStreamDescription();
        }

        int pageSize = description.getShards().size();
        lastShardId = description.getShards().get(pageSize - 1).getShardId();

      } while (description.getHasMoreShards());

      return lastShardId;

    } finally {
      kinesisClient.shutdown();
    }
  }

public static String getLastShardId ( ClientConfiguration awsClientConfig , KinesisConfigBean conf , String streamName ) throws StageException { AmazonKinesis kinesisClient = getKinesisClient ( awsClientConfig , conf ) ; String lastShardId = null ; try { StreamDescription description ; do { if ( lastShardId == null ) { description = kinesisClient . describeStream ( streamName ) . getStreamDescription ( ) ; } else { description = kinesisClient . describeStream ( streamName , lastShardId ) . getStreamDescription ( ) ; } int pageSize = description . getShards ( ) . size ( ) ; lastShardId = description . getShards ( ) . get ( pageSize - 1 ) . getShardId ( ) ; } while ( description . getHasMoreShards ( ) ) ; return lastShardId ; } finally { kinesisClient . shutdown ( ) ; } }

Get the last shard Id in the given stream
In preview mode, kinesis source uses the last Shard Id to get records from kinesis
@param awsClientConfig generic AWS client configuration
@param conf
@param streamName

Get the last shard Id in the given stream In preview mode kinesis source uses the last Shard Id to get records from kinesis


private void processQueue(
      LinkedList<Record> queue,
      List<OnRecordErrorException> errorRecords,
      Connection connection,
      int maxRowsPerBatch,
      int opCode
  ) throws StageException {
    if (queue.isEmpty()) {
      return;
    }

    int rowCount = 0;
    // Assume that columns are all same for the same operation to the same table
    // If some columns are missing in record, the record goes to error.
    final Record first = queue.getFirst();
    SortedMap<String, String> columnsToParameters = recordReader.getColumnsToParameters(
        first,
        opCode,
        getColumnsToParameters(),
        opCode == OperationType.UPDATE_CODE ? getColumnsToFieldNoPK() : getColumnsToFields()
    );

    if (columnsToParameters.isEmpty()) {
      // no parameters found for configured columns
      if (LOG.isWarnEnabled()) {
        LOG.warn("No parameters found for record with ID {}; skipping", first.getHeader().getSourceId());
      }
      return;
    }

    String query = generateQueryForMultiRow(
        opCode,
        columnsToParameters,
        getPrimaryKeyColumns(),
        // the next batch will have either the max number of records, or however many are left.
        Math.min(maxRowsPerBatch, queue.size())
    );

    // Need to store removed records from queue, because we might need to add newly generated columns
    // to records for Jdbc Tee Processor.
    LinkedList<Record> removed = new LinkedList<>();

    try (PreparedStatement statement = jdbcUtil.getPreparedStatement(getGeneratedColumnMappings(), query, connection)) {
      int paramIdx = 1;
      // Start processing records in queue. All records have the same operation to the same table.
      while (!queue.isEmpty()) {
        Record r = queue.removeFirst();
        if (opCode != DELETE_CODE) {
          paramIdx = setParamsToStatement(paramIdx, statement, columnsToParameters, r, connection, opCode);
        }
        if (opCode != OperationType.INSERT_CODE) {
          paramIdx = setPrimaryKeys(paramIdx, r, statement, opCode);
        }
        removed.add(r);
        ++rowCount;
        if (rowCount == maxRowsPerBatch) {
          // time to execute the current batch
          processBatch(removed, errorRecords, statement, connection);
          // reset our counters
          rowCount = 0;
          paramIdx = 1;
          removed.clear();
        }
      }
    } catch (SQLException e) {
      handleSqlException(e, removed, errorRecords);
    }

    // Process the rest of the records that are removed from queue but haven't processed yet
    // this happens when rowCount is still less than maxRowsPerBatch.
    // This is a bit of an ugly fix as its not very DRY but sufficient until there's a larger
    // refactoring of this code.
    if (!removed.isEmpty()) {
      query = generateQueryForMultiRow(
          opCode,
          columnsToParameters,
          getPrimaryKeyColumns(),
          removed.size() // always the remainder
      );

      try (PreparedStatement statement = jdbcUtil.getPreparedStatement(
          getGeneratedColumnMappings(),
          query,
          connection
      )) {
        int paramIdx = 1;
        for (Record r : removed) {
          if (opCode != DELETE_CODE) {
            paramIdx = setParamsToStatement(paramIdx, statement, columnsToParameters, r, connection, opCode);
          }
          if (opCode != OperationType.INSERT_CODE) {
            paramIdx = setPrimaryKeys(paramIdx, r, statement, opCode);
          }
        }
        processBatch(removed, errorRecords, statement, connection);
      } catch (SQLException e) {
        handleSqlException(e, removed, errorRecords);
      }
    }
  }

Process all records in queue. All records have same operation to same table.
Generate a query and set parameters from each record. INSERT and DELETE can be multi-row operation
but UPDATE is single-row operation.
If maxStatement
@param errorRecords
@param connection
@param maxRowsPerBatch
@param opCode
@param queue
@return
@throws OnRecordErrorException

Process all records in queue . All records have same operation to same table . Generate a query and set parameters from each record . INSERT and DELETE can be multi - row operation but UPDATE is single - row operation . If maxStatement