AND THAT IS ACTUALLY STORED IN ANOTHER X. M. L. DOCUMENT HMM 'KAY UM SO IS IS ANYBODY CREATING AN UH A REAL RAW TEXT THING AT THE MOMENT LIKE WHICH IS JUST THE WORDS YEAH THAT'S WHAT I'M GONNA NEED YEAH THA 'CAUSE THAT'S WHAT I'M GONNA NEED AS WELL YES YEAH IT'S JUST MO CHANGING IT A BIT LIKE IS IT JUST THE FIRST AND THE LAST LINE OR IS IT FOR EVERY SINGLE THING IN WE CAN JUST CHANGE THE CODE YEAH THAT DEPENDS ON WHAT YOU WANT YEAH BUT I DO IT WITH PERL IT'S JUST STRING MANIPULATION SO I WOULD I MEAN I WOULD JUST IF YOU'RE INTO IT CAN YOU MAKE A TEXT FILE WHICH JUST LIKE MAKES JUST THE WORDS 'KAY DO YOU WANT IT STRAIGHT FLOWING 'CAUSE I WOULD NEED SOMETHING THAT MARKS THE END OF UH OF UH IS IS YOURS SEGMENTED BY TOPICS THEN THAT LIKE IS THERE ANY INFORMATION THAT YOU HAVE TO THE TOPIC TO THE AUTOMATED TOPIC TOPIC SEGMENTATION OH THEN I NEED SOMETHING DIFFERENT LATER ANYWAY OKAY BUT FOR NOW IF YOU C IT IT MARKS THE END OF EACH SEGMENT YOU'RE GONNA PUT THAT AS AN OUTPUT OF YOURS THE SEGMENTATION OKAY SO FOR NOW CAN YOU CREATE LIKE SORT OF JUST UH A DUMP WHICH IS PURE TEXT JUST PURE TEXT SO THAT I CAN GET A DICTIONARY AND YOU CAN WORK ON THAT FOR YOUR TOPIC SEGMENTATION AND AND YOU WOULD WANT THAT ALL IN ONE FILE FOR ALL THE CORPUS OR OR FOR FOR THE SERIES BUT I CAN BUT I CAN ALSO DEAL WITH SEPARATE FILES I MEAN I CAN JUST WRITE THE ALGORITHM THAT IT LOADS ALL FILES IN A DIRECTORY OR SOMETHING BUT I MEAN IF YOU BUT IF YOU CAN PUT IT IN ONE SINGLE MEGA FILE THAT WOULD BE QUITE USEFUL FOR ME OH EVEN THOUGH FOR YOU WOULDN'T IT BE EASIER IF YOU HAD DIFFERENT FILES BECAUSE THEN YOU SORT OF KNOW LIKE YEAH FOR ME IT'S BETTER IF THEY'RE BY MEETING UH-HUH LIKE WHATEVER ONE TWO THREE FOUR FIVE OR SOMETHING OR JUST ANYTHING THAT I CAN YEAH THEY WILL JUST I WILL JUST TAKE I WOULD UH TAKE OVER THE NAMES THEY HAVE ANYWAY YEAH IS IS IT SOMETHING THAT'S EASILY ENU LIKE TO ENUMERATE OVER IS IT SOME JUST SOME ORDERED PATTERN OKAY COOL SO YEAH AND YOU BUT UM IN THE RIGHT ORDER YEAH IN THE RIGHT ORDER YES ORDERS UM ORD BASE DOT TIMES YEAH AND DO YOU WANT YEAH SOMETIMES THEY'RE CONTAINED IN ONE ANOTHER SO YEAH THAT DOESN'T MATTER TOO MUCH I THINK 'KAY ORDERED ONLY WORDS WHEN DO YOU THINK YOU'LL HAVE UM LIKE A PRIMITIVE SEGMENTATION BY SOME READY MADE TOPIC SEGMENTATION BY SOME READY MADE TOOL READY THEN THAT'LL BE REALLY EASY TO DO ONCE THEY'VE GOT THE RAW TEXT IT'S JUST A CASE OF RUNNING THE SCRIPT OKAY COOL 'CAUSE I'LL NEED THAT THEN WHEN IT'S DONE YEAH I MEAN HOPEFULLY THIS WEEK OKAY IT'S UM NINE MEGAMI BYTE TO HAVE I MEAN SHOULD BE SHOULD BE SIMILAR TO HAVE THE WORDS SHOULD BE WHAT'S WHAT'S NINE MEGABYTE THE THE ALRIGHT THAT SOUNDS QUITE REASONABLE THAT'S NINE NINE CHARACTERS OVER OKAY OKAY OKAY THAT IS FOR ARE WE ARE WE PICKING ONE PARTICULAR SERIES AT THE MOMENT OR AND WE COULD YEAH I MEAN I'M IT DOING IT FOR ALL OF IT DOESN'T MATTER OKAY YEAH YEAH I GUESS WE CAN PROBABLY PROCESS THE DATA FOR ALL DIFFERENT SERIES AND THEN CHECK WHICH SERIES IS THE BEST FOR THE PRESENTATION IT SOUNDS QUITE REASONABLE NINE MEGABYTE I MEAN IF YOU THINK IF IT'S R ROUGHLY A MILLION WORDS AND NINE CHARACTERS PER WORD SOUNDS REALISTI IT'S JUST WHAT I YEAH YES I'M GONNA BUILD A DICTIONARY THEN FROM THAT LIKE JUST A LIST OF THE WORDS THAT MAYBE A LIST OF THE WORDS WITH THE FREQUENCIES OR A LIST OF THE WORDS SORTED ALPHABETICALLY OR NUMERICALLY WHAT WHAT DOES ANYONE WANT DOES THIS THERE ANY WISHES FOR DICTIONARIES I COULD JUST USE IT WITH THE FREQUENCY I THINK UNTIL THE INFORMATION DENSITY THING'S FINISHED THAT WOULD BE REALLY USEFUL MM-HMM MM SO SO UM I WILL PROBABLY SEND UM JUST ONE FILE OF THE FIRST MEETING UM TO ALL THOSE WHO NEED IT SO THAT YOU CAN HAVE A LOOK WHETHER THAT'S WHAT YOU WANT MM-HMM YEAH I MEAN IF IT'S JUST FOR ONE MEETING IT'S REALLY NOT TOO BIG THEN LIKE THE BIG THING WE PROBABLY SHOULDN'T DO BY EMAIL YEAH HOW LONG WOULD IT TAKE TO MAKE THE FREQUENCY COUNTS WITH A JAVA HASH TABLE OH FROM THE TIME I GET THE FILE I CAN DO THAT IN AN AFTERNOON THE NEXT SORT OF THE NEXT MORNING OH YOU MEAN HOW LONG PROCESSING TIME IT TAKES AH IT'S A IT'S A BOG STANDARD ALGORITHM I'VE I'VE SORT OF I'VE WRITTEN IT FOR FOR DIL JUST IN HALF AN HOUR OR SOMETHING SIMILAR IT'S JUST YOU PUT THEM IN A HASH TABLE AND AND SAY WELL IF IT EXISTS ALREADY IN THE HASH TABLE THEN YOU INCREASE THE COUNT BY ONE AND I'LL PROBABLY IMPLEMENT SOME FILTER FOR FILTERING OUT NUMBERS OR SOMETHING BECAUSE IT'S QUITE EASY IN PERL AS WELL IT'S JUST A LINE OF CODE FOR COUNTING ALL THE WORDS AND YEAH IT'S IT'S BY HASHES OKAY WELL I DON'T KNOW ANY PERL I MEAN IF ANYONE WANTS TO DO A PERL SCRIPT FOR THAT THAT DOES IT DOES IT NICELY I UH I'VE NO PROBLEM WITH THAT I BUT I THINK I HAVE THE JAVA CODE VIRTUALLY READY BECAUSE FOR DIL I WROTE SOMETHING VERY SIMILAR LIKE FOR DIL I WROTE SOMETHING THAT COUNTS THE THE DIFFERENT OCCURRENCES OF ALL THE TAGS UM SORRY IF YOU'RE DOING IT IN JAVA COULD YOU SERIALIZE THE UM DICTIONARY YEAH AS WELL AS WRITING IT TO A FILE UH I'VE NEVER SERIALIZED ANYTHING WOULDN'T THAT BE ABSOLUTELY MASSIVE THOUGH I DON'T SEE WHY IT'D BE ANY MORE MASSIVE THAN THE FILE AND THEN SERIALI AND THEN WRITE THE SERIALIZATION TO A FILE SO YOU WANT LIKE A SE LIKE A FILE WHICH IS THE SERIALIZATION OF A HASH TABLE OKAY FILE REPRESENTATION OF IT AND NOW 'CAUSE I WOULD BE USING IT IN JAVA ANYWAY SO I'D JUST BE BUILDING THE DATA STRUCTURE AGAIN I I'LL I'LL CHECK IF I UNDERSTAND HOW IT WORKS I MEAN OTHERWISE I CAN GIVE YOU THE CODE FOR LOADING A DICTIONARY GIVE YOU MY MY IT'S JUST IT'S IT'S SORT OF IT'S A LINE BREAK SEPARATED FILE YOU KNOW YEAH YEAH I'LL SEE IF I UNDERSTAND HOW TO SERIALIZE THERE'S A THERE'S A SERIALISE COMMAND SO THAT GIVES ME ONE MEGA MOTHER OF A S YEAH BUT DO THEY AUTOMATICALLY WRITE TO THE FILE ANYWAY I'LL I'LL FIGURE THAT OUT WE DON'T HAVE TO YES IS THAT PRETTY MUCH PRETTY MUCH IT SO DAVE AND ME LOOK AT HOW NITE X. M. L. WORKS AND WE'RE HMM I'LL BUILD A DICTIONARY AS SOON AS I GET THE TEXT AND YEAH SO THAT