m7mdal7aj commited on
Commit
86d5177
1 Parent(s): aa9fe38

Update my_model/utilities/ui_manager.py

Browse files
Files changed (1) hide show
  1. my_model/utilities/ui_manager.py +6 -35
my_model/utilities/ui_manager.py CHANGED
@@ -3,6 +3,7 @@ import streamlit.components.v1 as components
3
  import pandas as pd
4
  from my_model.tabs.run_inference import InferenceRunner
5
  from my_model.tabs.results import run_demo
 
6
  from my_model.state_manager import StateManager
7
 
8
  class UIManager():
@@ -50,38 +51,7 @@ class UIManager():
50
  def display_home(self):
51
  """Displays the Home page of the application."""
52
 
53
- st.title('Multimodal Learning for Visual Question Answering using World Knowledge')
54
- st.text('')
55
- st.header('(Knowledge-Based Visual Question Answering)')
56
- with open("Files/Model Arch.html", 'r', encoding='utf-8') as f:
57
- model_arch_html = f.read()
58
- col1, col2 = st.columns([1,1])
59
- with col1:
60
-
61
- # st.write("""\n\n\n\n\n\n""")
62
- st.header("Model Architecture")
63
- components.html(model_arch_html, height=1400)
64
-
65
-
66
- with col2:
67
- #st.image("Files/mm.jpeg")
68
- st.header("Abstract")
69
- st.write("""\n\nNavigating the frontier of the Visual Turing Test, this research delves into multimodal learning to bridge the gap between visual perception and linguistic interpretation, a foundational challenge in artificial intelligence. It scrutinizes the integration of visual cognition and external knowledge, emphasizing the pivotal role of the Transformer model in enhancing language processing and supporting complex multimodal tasks.
70
- This research explores the task of Knowledge-Based Visual Question Answering (KB-VQA), it examines the influence of Pre-Trained Large Language Models (PT-LLMs) and Pre-Trained Multimodal Models (PT-LMMs), which have transformed the machine learning landscape by utilizing expansive, pre-trained knowledge repositories to tackle complex tasks, thereby enhancing KB-VQA systems.
71
- \nAn examination of existing Knowledge-Based Visual Question Answering (KB-VQA) methodologies led to a refined approach that converts visual content into the linguistic domain, creating detailed captions and object enumerations. This process leverages the implicit knowledge and inferential capabilities of PT-LLMs. The research refines the fine-tuning of PT-LLMs by integrating specialized tokens, enhancing the models’ ability to interpret visual contexts. The research also reviews current image representation techniques and knowledge sources, advocating for the utilization of implicit knowledge in PT-LLMs, especially for tasks that do not require specialized expertise.
72
- \nRigorous ablation experiments conducted to assess the impact of various visual context elements on model performance, with a particular focus on the importance of image descriptions generated during the captioning phase. The study includes a comprehensive analysis of major KB-VQA datasets, specifically the OK-VQA corpus, and critically evaluates the metrics used, incorporating semantic evaluation with GPT-4 to align the assessment with practical application needs.
73
- \nThe evaluation results underscore the developed model’s competent and competitive performance. It achieves a VQA score of 63.57% under syntactic evaluation and excels with an Exact Match (EM) score of 68.36%. Further, semantic evaluations yield even more impressive outcomes, with VQA and EM scores of 71.09% and 72.55%, respectively. These results demonstrate that the model effectively applies reasoning over the visual context and successfully retrieves the necessary knowledge to answer visual questions.""")
74
- st.write("""\n\n\nThis is an interactive application built to demonstrate the project developed and allow for interaction with the KB-VQA model as part of the dissertation for Masters degree in Artificial Intelligence at the [University of Bath](https://www.bath.ac.uk/).
75
- \n\n\nDeveloped by: [Mohammed H AlHaj](https://www.linkedin.com/in/m7mdal7aj)""")
76
- #st.write("\n\n")
77
- st.header("Acknowledgement")
78
- st.write("""I am profoundly grateful for the support and guidance I have received throughout the course of my dissertation. I would like to extend my deepest appreciation to the following individuals:
79
- \nTo my supervisor, [Dr. Andreas Theophilou](https://researchportal.bath.ac.uk/en/persons/andreas-theophilou), whose expertise, and insightful guidance have been instrumental in the completion of this research. Your mentorship has not only profoundly shaped my work but also my future endeavours in the field of computer science.
80
- Special mention must be made of my mentors at the University of Bath— [Dr. Ben Ralph](https://researchportal.bath.ac.uk/en/persons/ben-ralph), [Dr. Hongping Cai](https://researchportal.bath.ac.uk/en/persons/hongping-cai), and [Dr. Nadejda Roubtsova](https://researchportal.bath.ac.uk/en/persons/nadejda-roubtsova). The wealth of knowledge and insights I have gained from you has been indispensable. Your unwavering dedication to academic excellence and steadfast support have been crucial in navigating my academic journey.
81
- \nMy colleagues deserve equal gratitude, for their camaraderie and collaborative spirit have not only made this journey feasible but also deeply enjoyable. The shared experiences and the challenges we have overcome together have been integral to my personal and professional growth.
82
- \nLastly, my heartfelt thanks are extended to my family, whose unyielding love and encouragement have been my steadfast anchor. Your belief in my abilities has consistently inspired me and bolstered my strength throughout this process.
83
- This dissertation is not merely a reflection of my individual efforts but stands as a testament to the collective support and wisdom of each individual mentioned above. I am honoured and privileged to be part of such a supportive and enriching academic community.
84
- """)
85
 
86
  def display_dataset_analysis(self):
87
  """Displays the Dataset Analysis page."""
@@ -98,15 +68,16 @@ class UIManager():
98
  st.write("\n")
99
 
100
  run_demo()
101
-
102
 
103
 
104
  def display_run_inference(self):
105
  """Displays the Run Inference page."""
106
 
107
  st.title("Run Inference")
108
- st.write("Please note that this is not a general purpose model, it is specifically trained on [OK-VQA Dataset](https://okvqa.allenai.org/) and desgined to give short and direct answers to the given questions about the given image.")
109
- st.write("\n")
 
 
110
  inference_runner = InferenceRunner()
111
  inference_runner.run_inference()
112
 
 
3
  import pandas as pd
4
  from my_model.tabs.run_inference import InferenceRunner
5
  from my_model.tabs.results import run_demo
6
+ from my_model.tabs.home import run_home
7
  from my_model.state_manager import StateManager
8
 
9
  class UIManager():
 
51
  def display_home(self):
52
  """Displays the Home page of the application."""
53
 
54
+ run_home()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
55
 
56
  def display_dataset_analysis(self):
57
  """Displays the Dataset Analysis page."""
 
68
  st.write("\n")
69
 
70
  run_demo()
 
71
 
72
 
73
  def display_run_inference(self):
74
  """Displays the Run Inference page."""
75
 
76
  st.title("Run Inference")
77
+ st.write("""Please note that this is not a general purpose model, it is specifically trained on
78
+ [OK-VQA Dataset](https://okvqa.allenai.org/) and desgined to give short and direct answers to the
79
+ given questions about the given image.\n""")
80
+
81
  inference_runner = InferenceRunner()
82
  inference_runner.run_inference()
83