Hello, First I would like to thank you for your work, it looks impressive, it says at the end that it is possible to process different languages because as you know there are languages that use a totally different alphabet, so can the same results be obtained?
thanks for your question.
currently, English or most languages used in Europe can be identified since the vocabulary is break by one space. If the languages or vocabularies are not break based on the space, such as Chinese, Japanese, Korean, the corresponding break method is needed. some vocabularies should be recorded and added in the database for sentence segment breaking. however, we believe that after finding the segments from the original content, the training procedure still works.
