fix: refine chunks

app.py

@@ -45,7 +45,6 @@ def split_text_into_chunks(text, max_chunk_length):
 
     if current_chunk:
         chunks.append(current_chunk.strip())
-
     return chunks
 
 
@@ -88,7 +87,7 @@ def main():
             else:
                 sentence = load_text_file(uploaded_file)
         st.write(f"{len(sentence)} characters and {len(sentence.split())} words")
-        st.write(sentence)
+        #st.write(sentence)
 
     with audioTab:
         st.text("Yet to be implemented...")
@@ -98,8 +97,8 @@ def main():
 
     with st.spinner("Generating Summary..."):
         if button and sentence:
-            chunks = split_text_into_chunks(sentence, 500)
-            print(chunks)
+            chunks = split_text_into_chunks(sentence, 10000)
+            print(f"Split into {len(chunks)} chunks")
 
             text_words = len(sentence.split())
             if summary_balance == "concise":
@@ -112,7 +111,7 @@ def main():
                 min_multiplier = text_words * 0.2
                 max_multiplier = text_words * 0.5
 
-            print(f"min tokens {int(min_multiplier)}, max tokens {int(max_multiplier)}")
+            print(f"Tokenizer min tokens {int(min_multiplier)}, max tokens {int(max_multiplier)}")
             inputs = tokenizer(
                 chunks,
                 max_length=model.config.max_position_embeddings,

data.txt

@@ -1 +1,8 @@
-The Industrial Revolution, often regarded as one of the most transformative periods in human history, commenced in Britain during the latter part of the 18th century and gradually spread across the globe. Its impact was profound and multifaceted, reshaping nearly every aspect of society, economy, and daily life. At its core was the rapid development and adoption of new technologies, such as steam power, mechanization, and the factory system. These innovations revolutionized traditional industries, enabling mass production on an unprecedented scale and significantly increasing efficiency and productivity. As a result, goods once crafted painstakingly by hand could now be manufactured swiftly and in large quantities, leading to an explosion in consumer markets and a surge in global trade. Concurrently, the revolution spurred urbanization as rural populations flocked to burgeoning industrial centers in search of employment opportunities, fundamentally altering the demographic landscape and giving rise to sprawling cities teeming with people and activity. However, the Industrial Revolution was not without its drawbacks and challenges. While it brought about undeniable economic growth and prosperity for many, it also engendered profound social inequalities and labor exploitation. Factory workers, including men, women, and children, toiled for long hours in hazardous conditions for meager wages, often subjected to harsh treatment by factory owners and overseers. Moreover, the relentless pursuit of profit and industrial expansion exacted a heavy toll on the environment, as factories spewed pollutants into the air and waterways, and deforestation accelerated to meet the insatiable demand for timber and resources. Despite these drawbacks, the Industrial Revolution fundamentally transformed the fabric of society and laid the groundwork for the modern world. It fueled technological innovation, spurred scientific inquiry, and catalyzed advancements in transportation, communication, and medicine. Moreover, it set the stage for the rise of capitalism and the development of modern economic systems, shaping the dynamics of global commerce and trade for centuries to come. Today, its legacy endures in the form of industrial infrastructure, urban landscapes, and societal norms, serving as a testament to humanity's capacity for innovation and adaptation in the face of profound change.
+The Industrial Revolution, spanning the 18th and 19th centuries, was a transformative period that reshaped societies, economies, and cultures across the globe. Originating in Great Britain and later spreading worldwide, this pivotal era marked a significant departure from traditional modes of production and ushered in an era of unprecedented progress and development.
+Prior to the Industrial Revolution, much of human labor relied on manual effort or the assistance of animals. However, with the advent of machinery and the utilization of steam power, industries were able to mechanize production processes, leading to a substantial increase in efficiency and productivity. This shift from manual labor to mechanization in factories and mills revolutionized manufacturing, accelerating the pace of industrialization and propelling societies into an era of rapid change.
+One of the key industries transformed by the Industrial Revolution was agriculture. Innovations such as the seed drill, crop rotation techniques, and selective breeding methods revolutionized farming practices. These advancements significantly increased agricultural productivity and yielded surplus crops, resulting in improved living standards for many people. With the newfound abundance of food, populations grew, and individuals began migrating from rural areas to burgeoning urban centers in search of employment opportunities. The rise of urbanization became a defining characteristic of the Industrial Revolution, leading to the formation of sprawling cities and the concentration of populations in these industrial hubs.
+In addition to agriculture, transportation systems underwent revolutionary changes during this period. The development of steam-powered locomotives and the construction of extensive railway networks facilitated the movement of goods and people over long distances. This transformation revolutionized trade, enabling faster and more efficient transportation of raw materials, finished products, and individuals. The establishment of railway and steamship networks fostered the growth of global trade, connecting regions economically and culturally and giving rise to a new era of interconnectedness.
+Technological advancements played a critical role in driving the Industrial Revolution forward. Inventions such as the spinning jenny, power loom, steam engine, telegraph, and later the electric motor transformed industries, making production faster, more reliable, and cost-effective. These technological breakthroughs not only revolutionized manufacturing processes but also laid the foundation for subsequent advancements in communication, transportation, and medicine. The Industrial Revolution was a catalyst for innovation, fostering a culture of scientific and technological progress that has shaped the modern world.
+The social and economic implications of the Industrial Revolution were profound. The rise of factories and urban centers created a new social class, the industrial working class. These workers often toiled under challenging conditions, facing long hours, low wages, and hazardous environments. The period witnessed the emergence of labor unions and the fight for workers' rights, leading to improved conditions and the gradual establishment of labor laws. The Industrial Revolution also played a crucial role in the growth of capitalism and the development of modern economic systems, with factory owners and entrepreneurs accumulating vast wealth and power.
+The significance of the Industrial Revolution cannot be overstated. It propelled humanity into a new era of progress, transforming agrarian societies into industrialized nations. The revolution laid the groundwork for modern economies, technological advancements, and scientific discoveries that continue to shape our world today. By harnessing the power of machinery, the Industrial Revolution propelled humanity forward, revolutionizing the way we live, work, and interact, and setting the stage for the remarkable progress that followed.
+In conclusion, the Industrial Revolution stands as a pivotal period in history, characterized by profound changes in technology, industry, and society. Its impact on agriculture, manufacturing, transportation, and innovation laid the foundation for the modern world. Through the advent of mechanization, the Industrial Revolution spurred remarkable advancements, improving living standards, increasing production capabilities, and expanding opportunities for countless individuals worldwide. This transformative era remains a testament to human ingenuity, resilience, and the ability to adapt and thrive in the face of unprecedented change.