SO THEY DON'T MATCH BUT THE CLUSTERING IS STILL VERY GOOD IN IN ONE RESPECT UH BUT BUT YOU YOU CAN EXTEND THAT UP EVEN IF YOU HAVE A HUNDRED CATEGORIES P. L. S. A. CAN FIND A HUNDRED CLUSTERS THAT ARE A GOOD CLUSTERING BUT ARE COMPLETELY UNRELATED TO THE DIRECTORY I MEAN YOU CAN'T SAY OF COURSE IT'S ALWAYS GOOD BECAUSE IT'S MAXIMISING THE LIKELIHOOD OF WHATEVER YEAH BUT YEAH YEAH YEAH BUT BUT HERE YOU CAN YOU YOU SAY GIVEN A RANDOM PAGE OF P. U. WHICH OF THOSE TWO MODEL IS MORE LIKELY TO TO TO TO TO SAY FOR EXAMPLE THAT THI THE PAGE OF P. L. WHICH IS THE MOST SIMILAR SO YOU SO SO FROM THIS SO YOU TAKE ANY DISTRIBUTION COMPARISON THINGS SO SO HOFFMANN IN HIS PAPER WAS DOING JUST COSINE SIMILARITY BUT YOU CAN DO ANYTHING SO SO YOU JUST COMPARE UH THOSE TWO DISTRIBUTIONS SO OF D. ONE AND D. TWO OKAY AND SO WITH THAT YOU CAN YOU CAN DETERMINE FOR EXAMPLE FOR A DOCUMENT O OF THE SET TWO TO THE MOST SIMILAR SO YOU YOU YOU YOU SELECT D. ONE SO IN THIS SET WHICH IS THE MOST SIMILAR TO D. TWO ACCORDING TO THE DISTRIBUTION FROM ONE OR THE OTHER MODEL MM-HMM AND THEN AND THEN FOR THIS DOCUMENT YOU AUTOMATICALLY ASSIGN TO IT THE LABEL OF THE DOCUMENT UH ONE AND YOU CHECK WITH THIS WHICH IS YOUR UH YOUR YOUR IF YOU ARE RIGHT OR NOT MM THIS IS WHAT WHAT THIS IS VERY RELATED TO WHAT FRANK DID WITH THE WITH THE KEYWORD UH IMAGE ANNOTATION SO YOU WOULD HAVE A SET OF KEYWORDS THE SET OF IMAGE LABEL WITH KEYWORDS AND HE HAS A SET OF UNLABELLED IMAGE WITH SOME AND IT WOULD CHECK WHETHER THE THE LABEL HE ASSIGNS SO IN THIS CASE ANNOTATION WORD H ARE CORRECT AND I THINK IT'S FAIR UH IT'S FAIR UH SETUP OKAY FUS THERE THERE'S SOME MIGHT BE SOME WORK TO DO BUT THE GENERAL IDEA I THINK IT'S IT'S FAIR EVALUATIONS IT'S A TASK S OKAY I'M I'M JUST N CONCERNED THAT THAT'S UH THAT THAT'S IT'S ANSWERING A QUESTION THAT EVALUATES IN SOME WAY BUT MAYBE NOT IN THE WAY THAT WELL THAT MAYBE NOT IS THE SAME WAY I WAS THINKING BUT MAYBE IT'S A BETTER WAY UM SO WHAT I MEAN BY THAT IS THAT SORRY I'M TRYING TO PROCESS ALL THIS UH YEAH I IT'S THE SAME THING LIKE IT IT DOES TELL YOU WHETHER THIS UH WHICH CLUSTERING IS BETTER FOR THIS TASK BUT IT DOESN'T TELL YOU WHICH CLUSTERING BETTER REPRESENTS THE PAGES NECESSARILY BUT IT IT'S IT YOU YOU COULD SAY THAT IT POSSIBLY DOE IT PROBABLY DOES NO OKAY THE THE THERE IS NOT UH V UH PERFECT UH CLUSTERING CLUS WH WHAT IS THE HOW DO YOU JUDGE UH WHAT IS A GOOD CLUSTERING YEAH WHAT'S GOOD WITH THE DIRECTORY IS THAT YOU HAVE A GOOD COVERAGE OF VARIOUS TOPIC HUMANS HUMANS CAN LABEL BUT THEN IF YOU WANT TO DETECT DIFFERENT LANGUAGES THIS IS ANOTHER TASK I THINK THE TASK OF THIS LATENT MODEL IS TO DISCOVER SOME SOME SEMANTIC LIKE SOME TOPIC PROXIMITY BETWEEN DOCUMENTS AND HERE THIS LA THIS DIRECTORY IS DOING THE SAME THING LIKE ASSOCIATING SOME TOPICS WITH SOME DOCUMENTS MANUALLY SO COMPARING THEM IS IS IS IS GOOD SETUP WITH RESPECT TO THE GOAL YOU'RE COMPARING THEM AH OKAY OKAY YEAH YEAH YEAH I DON'T KNOW IF IF IT IF IT WILL PROVIDE GOOD RESULT RESULT OR NOT BUT IT IS CLEARLY A BENCHMARK FOR THOSE TWO GOALS YOU CAN E EVEN KEEP THE BAG OF WORDS AND DO THE COSINE COMPARISON OR ALSO LIKE A BASELINE YEAH YEAH LIKE BAG O YEAH YEAH YOU CAN HAVE TREE SETUPS UH YEAH YEAH BASIC COSINE OVER THE P. OF T. GIVEN D. SO OVER ALL THIS UH EMPIRICAL DISTRIBUTION AND THE TWO UH ASPECT UH MODELS BUT MM I THINK MM UH NO I TH I'M I'M COMING ROUND TO IT ACTUALLY NO I'M SO I I I THINK I G UH I I GET IT UH I I TH I THINK IT PROBABLY IS FAIR I'M JUST I I'VE GOTTEN UM UNCOMFORTABLE WITH IT BUT I THINK THAT'LL GO IF I THINK ABOUT IT ENOUGH UH BUT BUT YEAH THE IT IT BUT IT JUST SEEMS A LITTLE SUSPECT BUT I THINK THAT'S JUST FROM THE WHOLE POINT THAT YOU CAN'T TELL WHICH CLUSTERING IS B BETTER BUT UH WHAT I'D THOUGHT BEFORE WAS COULD YOU GET PEOPLE TO EVALUATE THIS BY BY HAND BASICALLY COULD YOU GIVE PEOPLE A SET OF PAGES AND LIKE SAY YOU DID IT WITH FIVE TOPICS FOR EXAMPLE UH NOW YOU GIVE PEOPLE FIVE SHEETS UH YOU GIVE UH A T A A USER FIVE SHEETS OF PAPER EACH WITH WORDS IN VARYING SIZES REPRESENTING THE PROBABILITY OF THAT TOPIC FOR EXAMPLE TRY GIVE THE USER THESE PIECES OF PAPER TO GET AN IDEA OF WHAT THESE TOPICS ARE THAT'S THIS P. OF W. GIVEN Z. AND THEN YOU GIVE THEM SOME PAGES AND SAY WHICH ONE WOULD YOU WHICH PIECE OF PAPER WOULD YOU PUT THAT WITH AND THEN YOU GET LIKE AN INTUITIVE IDE UH LIKE HOW WOULD A PERSON CLUSTER PAGES GIVEN THE WHAT THESE TOPICS SUPPOSEDLY ARE 'CAUSE I YOU'VE GOT SOMETHING THAT'S FIXED THE THE P. OF W. GIVEN Z. WITH RESPECT TO BOTH MODELS SO I WAS TR I WAS HOPING YOU COULD USE THAT TO IT'S IT'S MORE LIKE IF YOU WERE GIVING THE PEOPLE YOU YOU SAID TO THEM OKAY UH DIVIDE ME THIS BUNCH OF DOCUMENTS INTO FIVE TOPICS AND YOU DON'T GIVE THEM THE TOPICS A PRIORI YOU KNOW SO THE TWO PERSON COULD COME UP WITH VERY DIFFERENT ONE IS VERY INTERESTING I IN SO I GIVE YOU I GIVE THE REAL USERS MM THE PERSON UM A BUNCH OF MARBLES AND THEY BEGIN TO SAY OKAY AND THEY HAVE TO DIVIDE IT IN FIVE GROUP AND IN THE END IT'S NOT AL IT'S IT'S ALWAYS DIFFERENT TO THE CLUSTERING EACH PEOPLE HAS BECAUSE IT'S COMPLETELY SUBJECTIVE BUT IN BUT THERE IS UH SOME CHARACTERISTIC THAT ARE FOR TRANSPARENCY COLOUR WHICH ARE UM FEATURES THAT MADE THE CLUSTERING THAT CAN BE EXTRACTED EVEN IF THE CLUSTERING IT'S ALWAYS DIFFERENT OKAY SO I I THINK I THINK IT'S YEAH SEE THE THE THING ABOUT WEB PAGES I THOUGHT AS WELL IN THAT GETTING HUMANS TO EVALUATE IT IN SOME WAY IS THAT YOU HAVE MOST OF THE WEB PAGES THAT ARE UH WELL A LOT OF THE WEB PAGES THAT ARE IMPORTANT IN SOME WAY ARE ARE ARE HOME PAGES SO GOOGLE DOT COM DOESN'T ACTUALLY SAY THE WORDS SEARCH ON IT I THINK I MAYBE IT SAYS GO BUT YOU ONLY KNOW IT'S ABOUT SEARCH THROUGH THE THING BUT SOMEONE LOOKING AT IT CAN TELL THAT IT'S A SEARCH ENGINE BECAUSE OF THE WAY IT'S LAID OUT BECAUSE OF THE PICTURES ON IT AND BECAUSE OF ALL OF THAT