Module Import alias Import alias Import alias ImportFrom alias ClassDef bases NameLoad body Expr Str Assign NameStore Num Assign NameStore Str Assign NameStore Str FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults Str body Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad Str attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call AttributeLoad Str attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call AttributeLoad Str attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call AttributeLoad Str attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call AttributeLoad Str attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call AttributeLoad Str attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr Call AttributeLoad Str attr AttributeLoad NameLoad attr Assign AttributeStore NameLoad attr IfExp CompareIsNot NameLoad NameLoad AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr While CompareIs Call AttributeLoad NameLoad attr NameLoad body Assign NameStore Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr attr Str If CompareGt Call NameLoad NameLoad Num body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr With Call NameLoad AttributeLoad NameLoad attr Str NameStore body Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str AttributeLoad NameLoad attr Assign NameStore ListLoad Str AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str AttributeLoad NameLoad attr With Call NameLoad AttributeLoad NameLoad attr NameStore body Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword Str Assign AttributeStore NameLoad attr ListComp SubscriptLoad NameLoad Index Str comprehension NameStore NameLoad Assign AttributeStore NameLoad attr IfExp CompareIsNot AttributeLoad NameLoad attr NameLoad ListComp AttributeLoad NameLoad attr comprehension NameStore Call AttributeLoad Call AttributeLoad NameLoad attr attr AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr keyword AttributeLoad AttributeLoad AttributeLoad NameLoad attr attr attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str AttributeLoad NameLoad attr With Call NameLoad AttributeLoad NameLoad attr NameStore body Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword Str Assign AttributeStore NameLoad attr ListLoad For TupleStore NameStore NameStore Call NameLoad NameLoad body For TupleStore NameStore NameStore Call NameLoad NameLoad body If CompareGt NameLoad AttributeLoad NameLoad attr body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Dict Str Str Str Str NameLoad NameLoad NameLoad Num Expr Call AttributeLoad AttributeLoad NameLoad attr attr keyword Lambda arguments args NameParam defaults UnaryOpUSub SubscriptLoad NameLoad Index Str For TupleStore NameStore NameStore Call NameLoad AttributeLoad NameLoad attr body Assign SubscriptStore NameLoad Index Str BinOpAdd NameLoad Num Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str AttributeLoad NameLoad attr With Call NameLoad AttributeLoad NameLoad attr NameStore body Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword Str Assign AttributeStore NameLoad attr ListLoad For TupleStore NameStore NameStore Call NameLoad NameLoad body For TupleStore NameStore NameStore Call NameLoad NameLoad body If CompareGt NameLoad AttributeLoad NameLoad attr body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Dict Str Str Str Str NameLoad NameLoad NameLoad Num Expr Call AttributeLoad AttributeLoad NameLoad attr attr keyword Lambda arguments args NameParam defaults UnaryOpUSub SubscriptLoad NameLoad Index Str For TupleStore NameStore NameStore Call NameLoad AttributeLoad NameLoad attr body Assign SubscriptStore NameLoad Index Str BinOpAdd NameLoad Num decorator_list decorator_list
N/A N/A json N/A os N/A subprocess LDAReader LDAReader STMReader N/A LDAReader N/A N/A 
	lda_db = a SQLite3 database
	modelPath = a model folder containing an R datafile stm.RData
	 N/A PROB_THRESHOLD 0.0001 N/A RDATA_FILENAME stm.RData N/A SCRIPT # Load an STM model and save its content as Termite Data Server files
# Load required libraries
library(stm)
library(lda)

# Define output filenames
app.path = "{DATA_PATH}"
app.path.TermTopicMatrix = "{DATA_PATH}/term-topic-matrix.txt"
app.path.DocTopicMatrix = "{DATA_PATH}/doc-topic-matrix.txt"
app.path.DocIndex = "{DATA_PATH}/doc-index.json"
app.path.TermIndex = "{DATA_PATH}/term-index.json"
app.path.TopicIndex = "{DATA_PATH}/topic-index.json"

# Load input data
load( file = "{MODEL_FILENAME}" )
model = {R_VARIABLE}

library( jsonlite )

data.DocTopicMatrix = model$theta
data.TermTopicMatrix = exp( t( model$beta$logbeta[[1]] ) )

# Document Index
temp.DocCount <- nrow(model$theta)
temp.DocIndex <- 1:temp.DocCount
temp.DocIndexValues <- cbind( temp.DocIndex )
temp.DocIndexHeader <- c( "index" )
colnames( temp.DocIndexValues ) <- temp.DocIndexHeader
data.DocIndexJSON <- toJSON( as.data.frame( temp.DocIndexValues ), pretty = TRUE, digits = 10 )
write( data.DocIndexJSON, file = app.path.DocIndex )

# Term Index
temp.TermCount <- nrow( data.TermTopicMatrix )
temp.TermFreq <- apply( data.TermTopicMatrix, 1, sum )
temp.TermText <- model$vocab
temp.TermIndex <- 1:temp.TermCount
temp.TermIndexValues = cbind( temp.TermIndex, temp.TermFreq, temp.TermText )
temp.TermIndexHeader = c( "index", "freq", "text" )
colnames( temp.TermIndexValues ) <- temp.TermIndexHeader
data.TermIndexJSON <- toJSON( as.data.frame( temp.TermIndexValues ), pretty = TRUE, digits = 10 )
write( data.TermIndexJSON, file = app.path.TermIndex )

# Topic Index
temp.TopicCount <- ncol( data.TermTopicMatrix )
temp.TopicFreq <- apply( data.TermTopicMatrix, 2, sum )
temp.TopicIndex <- 1:temp.TopicCount
temp.TopicIndexValues = cbind( temp.TopicIndex, temp.TopicFreq )
temp.TopicIndexHeader = c( "index", "freq" )
colnames( temp.TopicIndexValues ) <- temp.TopicIndexHeader
data.TopicIndexJSON <- toJSON( as.data.frame( temp.TopicIndexValues ), pretty = TRUE, digits = 10 )
write( data.TopicIndexJSON, file = app.path.TopicIndex )

# Doc-Topic Matrix
# Tab-separated with no headers. Theta (D by K)
rownames( data.DocTopicMatrix ) <- temp.DocIndex
colnames( data.DocTopicMatrix ) <- temp.TopicIndex
data.DocTopicMatrixJSON <- toJSON( data.DocTopicMatrix, pretty = TRUE, digits = 10 )
write( data.DocTopicMatrixJSON, file = app.path.DocTopicMatrix )

# Term-Topic Matrix
# Tab-separated with no headers. Beta (V by K)
rownames( data.TermTopicMatrix ) <- temp.TermText
colnames( data.TermTopicMatrix ) <- temp.TopicIndex
data.TermTopicMatrixJSON <- toJSON( data.TermTopicMatrix, pretty = TRUE, digits = 10 )
write( data.TermTopicMatrixJSON, file = app.path.TermTopicMatrix )

 __init__ N/A N/A self lda_db modelPath corpus_db r_variable N/A mod.out N/A N/A N/A N/A N/A super STMReader self __init__ lda_db N/A N/A self rVariable r_variable N/A N/A self modelPath modelPath N/A N/A self modelRData N/A N/A {}/{} format N/A self modelPath N/A STMReader RDATA_FILENAME N/A N/A self modelScript N/A N/A {}/import.r format N/A self modelPath N/A N/A self ldaTermTopicMatrix N/A N/A {}/term-topic-matrix.txt format N/A self modelPath N/A N/A self ldaDocTopicMatrix N/A N/A {}/doc-topic-matrix.txt format N/A self modelPath N/A N/A self ldaDocIndex N/A N/A {}/doc-index.json format N/A self modelPath N/A N/A self ldaTermIndex N/A N/A {}/term-index.json format N/A self modelPath N/A N/A self ldaTopicIndex N/A N/A {}/topic-index.json format N/A self modelPath N/A N/A self corpus N/A N/A corpus_db None N/A corpus_db db None N/A Execute N/A N/A self N/A N/A N/A N/A N/A N/A self logger info Reading STM topic model... N/A N/A N/A self WriteToDisk N/A N/A N/A self ReadFromDisk N/A N/A N/A N/A self logger info Writing to database... N/A N/A N/A self SaveToDB N/A RunCommand N/A N/A self command N/A N/A N/A p N/A N/A subprocess Popen command stdout N/A subprocess PIPE stderr N/A subprocess STDOUT N/A N/A N/A N/A p poll None N/A N/A line N/A N/A N/A N/A N/A p stdout readline rstrip N/A N/A N/A len line 0 N/A N/A N/A N/A N/A self logger debug line N/A WriteToDisk N/A N/A self N/A N/A N/A N/A N/A N/A self logger debug     Writing R script: %s N/A self modelScript N/A r N/A N/A N/A STMReader SCRIPT format DATA_PATH N/A self modelPath MODEL_FILENAME N/A self modelRData R_VARIABLE N/A self rVariable N/A N/A open N/A self modelScript w f N/A N/A N/A N/A f write N/A N/A r encode utf-8 N/A N/A N/A N/A self logger info Executing R script: %s N/A self modelScript N/A command N/A RScript N/A self modelScript N/A N/A N/A N/A self logger debug     Executing R script: %s N/A self modelScript N/A N/A N/A self RunCommand command N/A ReadFromDisk N/A N/A self N/A N/A N/A N/A N/A N/A self logger debug     Loading json: %s N/A self ldaTermIndex N/A N/A open N/A self ldaTermIndex f N/A N/A data N/A N/A json load f encoding utf-8 N/A N/A self termList N/A N/A d N/A text N/A d data N/A N/A self docList N/A N/A N/A self corpus None N/A N/A d doc_id N/A d N/A N/A N/A N/A self corpus select N/A N/A N/A self corpus corpus doc_id orderby N/A N/A N/A self corpus corpus doc_index None N/A N/A N/A N/A self logger debug     Loading matrix: %s N/A self ldaTermTopicMatrix N/A N/A open N/A self ldaTermTopicMatrix f N/A N/A matrix N/A N/A json load f encoding utf-8 N/A N/A self termTopicMatrix N/A N/A N/A termIndex topicFreqs N/A enumerate matrix N/A N/A N/A topicIndex value N/A enumerate topicFreqs N/A N/A N/A value N/A STMReader PROB_THRESHOLD N/A N/A N/A N/A N/A self termTopicMatrix append N/A term_index topic_index value rank termIndex topicIndex value 0 N/A N/A N/A N/A self termTopicMatrix sort key N/A N/A N/A d N/A N/A N/A d N/A value N/A N/A index d N/A enumerate N/A self termTopicMatrix N/A N/A N/A d N/A rank N/A index 1 N/A N/A N/A N/A self logger debug     Loading matrix: %s N/A self ldaDocTopicMatrix N/A N/A open N/A self ldaDocTopicMatrix f N/A N/A matrix N/A N/A json load f encoding utf-8 N/A N/A self docTopicMatrix N/A N/A N/A docIndex topicFreqs N/A enumerate matrix N/A N/A N/A topicIndex value N/A enumerate topicFreqs N/A N/A N/A value N/A STMReader PROB_THRESHOLD N/A N/A N/A N/A N/A self docTopicMatrix append N/A doc_index topic_index value rank docIndex topicIndex value 0 N/A N/A N/A N/A self docTopicMatrix sort key N/A N/A N/A d N/A N/A N/A d N/A value N/A N/A index d N/A enumerate N/A self docTopicMatrix N/A N/A N/A d N/A rank N/A index 1 N/A N/A