ICACT20240357 Slide.23
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
That's all for my presentation.
Anyone have any questions?
|
|
ICACT20240357 Slide.22
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
It can also be extended to various video platforms, such as managing videos with long running times, such as CCTV, by splitting them.
Therefore, it can play a role in saving time and manpower in parts of the social safety surveillance system such as CCTV control centers.
However, there is a limitation in that human conversations can be transcribed by applying STT technology, but background sounds such as car sounds, or other sounds such as animal cries cannot be transcribed.
Therefore, in future research, we will solve the problem of analyzing other sounds based on the proposed model.
|
|
ICACT20240357 Slide.21
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Finally, Here is the Conclusion.
In this paper, we proposed a model to enhance search and recommendation systems by extending metadata with contextual data extracted from actual content through AI-based content analysis.
If this model is applied to various content search and recommendation platforms, the extended metadata can be used to provide results that are close to the search query when the user enters the search query.
It can also be used to help users discover videos that contain similar content based on the video.
This will provide a way to discover more similar videos and clips by allowing metadata to reflect the actual content of content that was previously excluded.
This will compensate for problems with existing search, recommendation, and censorship systems, enabling more in-depth recommendations.
|
|
ICACT20240357 Slide.20
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
For cross-validation, compare the extracted script with the object frequency context data obtained using the YOLO deep learning model.
We then evaluate and test the developed model.
This can be evaluated by putting some videos into the test and checking if the correct metadata is extracted.
You can use it to develop a proof-of-concept video clip search program and integrate it into your environment.
You can continue to evolve your model through user feedback and performance monitoring, and then compare it to traditional search and recommendation methods to improve performance.
|
|
ICACT20240357 Slide.19
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The last is Fine-Tuning.
In this paper, we propose a video-clip indexing model using contextual data based on intelligent content analysis.
This model enables fine-tuning using contextual data extracted from videos.
First, we use scripts extracted using STT technology.
By comparing the extracted script with the subtitle data embedded in the video, the accuracy of the extracted script is checked and calibrated.
Adjust the hyper-parameters used to train the model and improve the model performance through cross-validation.
|
|
ICACT20240357 Slide.18
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
This figure is the Process of Model Fine-Tuning.
|
|
ICACT20240357 Slide.17
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Next is Grouping Video.
In this step, We'll Clips and Indexing.
Since video content has a time-series nature over time, shot boundaries that change with events or scene transitions.
detection segments video based on content. The video is then analyzed and processed on a content basis.
|
|
ICACT20240357 Slide.16
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The next step is Extending Metadata with Contextual Data.
Object frequencies and scripts extracted from videos have the characteristics of time series data.
Object frequencies are expressed as the number and type of objects observed in each frame.
Scripts appear across multiple frames and are generated as contextual data from the frame where the dialog starts.
The object frequencies and scripts are then synchronized.
When dividing the context data into scenes, we do so based on scripts.
We use iterative experiments to divide the scripts into appropriately sized sentences or paragraphs, as too small a division would clutter the data, too large a division would distort the features.
|
|
ICACT20240357 Slide.15
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Text mining libraries that can be used include Python's Natural Language Toolkit (NLTK), spaCy, gensim, scikitlearn, TensorFlow, PyTorch, which utilize deep learning techniques.
The process of quantifying through text mining involves many different steps.
For our proposed method, the text is tokenized by breaking it into small chunks and going through lemmatization to remove unimportant words, root extraction to find the basic form of words, and part-of-speech tagging to understand sentence structure and meaning.
|
|
ICACT20240357 Slide.14
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
First, the Data Preprocessing process.
Select key objects required in a video recommendation system based on object frequency.
We use the YOLO deep learning model to train a deep learning-based object recognition model on the labeled image dataset to detect and extract objects based on frames.
Since the YOLO deep learning model is trained using the COCO Dataset, it can detect and classify a total of 91 objects, from 1 to 91.
In addition, we extract the voice from the video separately and then use STT technology to extract the script according to the frame.
To apply machine learning techniques to the extracted script in natural language form, we apply text mining techniques to quantify it.
|
|
ICACT20240357 Slide.13
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In this study, we use a YOLO deep learning model to extract objects from videos frame by frame to develop a metadata expansion-based search and recommendation system.
At the same time, we extract speech from the video separately and convert it into a script using STT technology.
The extracted script data is then segmented into scenes.
Then, we synchronize the object and text data to create a single contextual data.
We check whether the context data is appropriate for the video and fine-tune it by making adjustments.
Index the generated contextual data by grouping it with the video.
Expand the metadata based on the indexed information.
Include the expanded metadata in search or recommendation to improve the performance of the search and recommendation system.
|
|
ICACT20240357 Slide.12
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Next, This figure is the structure of the search and recommendation systems with metadata extension that we propose.
|
|
ICACT20240357 Slide.11
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Fourth is Text Mining.
Text Mining refers to the process of extracting useful information from large amounts of text data by combining Natural Language Processing and Data Mining techniques.
It is used to discover and analyze patterns, trends, information, and statistical characteristics from text data.
The main purposes for which it is used are information retrieval, sentiment analysis, topic modeling, document classification, and information extraction.
Information retrieval is concerned with finding and ranking relevant documents for a user's search query.
Subject modeling is a technique for identifying and grouping key subjects in large amounts of text data.
This allows you to discover hidden patterns in text data.
Information extraction is the process of extracting important information from text.
Text Mining does this by applying machine learning and statistical analysis techniques to text data.
|
|
ICACT20240357 Slide.10
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Third is Speech Recognition.
Speech Recognition is a technology that recognizes human speech and converts it into text by a computer.
This allows users to interact with computers through voice commands or voice input.
This speech recognition technology is also known as Speech to Text (STT).
STT involves voice input, digital signal processing (DSP), feature extraction, acoustic modeling, and language modeling.
The user enters information by voice through a microphone, and then the voice signal is converted from analog to digital.
This is followed by digital signal processing, which involves preprocessing to remove background noise, cancel echoes, and other things.
Then, important features are extracted from the digitized speech.
|
|
ICACT20240357 Slide.09
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Second, Shot Boundary Detection.
Shot Boundary Detection is a technique for detecting scene transitions in video content.
Video consists of a sequence of successive frames that detect changes in color, lighting, objects in the scene, and automatically segment them into basic units called shots is Shout Boundary Detection.
A shot is a group of consecutive video frames in a video and usually consists of consecutive video frames from a single camera.
Over the years, the design of the Shot Boundary Detection algorithm has evolved from simple feature comparisons to the use of rigorous probabilistic and complex models.
In addition, to detect transitions with higher accuracy, orthogonal polynomials are applied to derive features in the orthogonal transform domain to detect hard transitions in video sequences.
|
|
ICACT20240357 Slide.08
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Next is Related Work. First,
OTT is an acronym for Over The Top, which refers to media content such as video content, audio, and text that is typically delivered over the Internet.
OTT services work by delivering content directly to consumers, bypassing traditional media distribution channels such as traditional broadcast or cable television.
The main characteristics of OTT services include internet-based delivery, diverse content, subscription models, personalized recommendations, content diversity, and creator opportunities.
This allows users to choose and watch content that suits their different interests and needs.
Subscription is a big feature of OTT services. Many OTT services adopt a subscription model, where users pay a set amount of money each month or year to access the service.
Subscribers can use these services to watch the content they want without any ads.
|
|
ICACT20240357 Slide.07
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Additionally, recommendation algorithms based on personal history can suffer from the bubble effect.
The bubble effect is when only content relevant to your history appears in the recommendation algorithm, exposing users to biased content.
Recently, crimes have been committed to exploit this to expose children to harmful videos, raising concerns about recommendation algorithms and video censorship schemes.
To solve these problems, it is necessary to strengthen the censorship work and introduce stricter regulations to block and remove harmful content, which requires a lot of time and manpower.
In this study, we use AI-based content analysis to extract contextual data from real-world content.
By indexing the extracted contextual data into videos and clips, we propose a model to extend metadata and improve search and recommendation systems.
|
|
ICACT20240357 Slide.06
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
While most search and recommendation algorithms work well, they rely heavily on your sensitive information, such as your viewing history, search history, and internet browser cookies.
This can lead to inaccurate recommendations if a user deletes their information or, depending on their level of privacy, lacks the necessary information to make a recommendation, a cold start problem that occurs in most recommendation systems.
In addition, since metadata is added by the content creator or distributor, there is a possibility that information about the actual content is excluded, reducing the accuracy of search or recommendation, or being abused.
Currently, similar content recommendation systems on OTT platforms often exclude key content that could be spoilers, even if the plot is included in the recommendation process.
This can cause users to lose trust in the recommendation system if the actual content is different, even if it is recommended as similar simply because of a similar genre or visual atmosphere.
|
|
ICACT20240357 Slide.05
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Next is Introduction.
The multimedia industry is currently experiencing the proliferation of personal internet broadcasting and OTT platforms, creating a huge amount of new videos.
Currently, general video search and recommendation uses metadata such as filenames, keywords, tags, and genres pre-added by the creator or distributor as an index.
Therefore, access to the video content will take a long time if there is no separate internal index.
OTT platforms and video platforms that have a large amount of content may analyze a user's previous browsing history, viewing history, subscribed channels, and other things to understand their interests in order to provide them with customized videos.
They compare this with metadata to recommend relevant content or analyze what other users with similar interests have watched, what's popular, and other things.
Based on this analysis, it selects recommended videos and serves them to you in order of priority.
|
|
ICACT20240357 Slide.04
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
To ensure that the generated contextual data is appropriate for the video, the proposed model compares the extracted script with the video's subtitle data to check and calibrate its accuracy.
The model can then be fine-tuned by tuning and cross-validating the hyper parameter to improve its performance.
These models can be incorporated into a variety of content discovery and recommendation platforms.
By using expanded metadata to provide results close to a search query and recommend videos with similar content based on the video, it solves problems with traditional search, recommendation, and censorship schemes, allowing users to explore more similar videos and clips.
|
|
ICACT20240357 Slide.03
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Therefore, this study proposes a search and recommendation system by expanding metadata in videos using techniques such as shot boundary detection, speech recognition, and text mining.
The proposed method selects the main objects required by the recommendation system based on the object frequency and extracts the corresponding objects from the video frame by frame.
In addition, we extract the speech from the video separately, convert the speech to text to extract the script and apply text mining techniques to the extracted script to quantify it.
Then, we synchronize the object frequency and the transcript to create a single contextual data.
After that, we group videos and clips based on the contextual data and index them.
Finally, we utilize Shot Boundary Detection to segment videos based on their content.
|
|
ICACT20240357 Slide.02
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
First is Abstract.
We proposes an AI-based Video Metadata Extension Model to overcome the limitations of video search and recommendation systems in the multimedia industry.
Current video searches and recommendations utilize pre-added metadata.
Metadata includes filenames, keywords, tags, genres, and other things.
This makes it impossible to make direct predictions about the content of a video without pre-added metadata.
These platforms also analyze your previous search history, viewing history, and so on, to understand your interests in order to serve you personalized videos.
This may not reflect the actual content and may raise privacy concerns.
In addition, recommendation systems suffer from a cold start problem, which is the lack of an initial target, as well as a bubble effect.
|
|
ICACT20240357 Slide.01
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
This is the table of contents.
The order is abstract, introduction, related work, search and recommendation system with metadata extension, conclusion, Q&A.
|
|
ICACT20240357 Slide.00
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Hi, everybody, my name is WooHyeon Kim. For ICACT 2024, I have cooperated with one partner from Kyonggi University, Dr. JooChang Kim. Our presentation today is about a Search and Recommendation Systems with Metadata Extensions.
|
