Requirement for Multi-Language Supported “Word Segmentation (Tokenizer/Segmenter)” Tool

Functional Requirement: A secondary development based on NLP SDK (i.e. Stanford NLP SDK or similar toolkits) which segments words for open (raw) text, such as a text paragraph or a text file: *Texts in English and Chinese language must be supported; *Texts in other languages shall be supported with an extended module, such as Russian, French and German etc; *It can provide other general functions such as word count and weight calculation; *It can segment text coded in UTF-8, Unicode and GBK etc; *It can extract key words according to specified dictionaries; *It can filter stop words according to specified dictionaries; *The result of segmentation and statistic can be exported or saved; *It has User’s dictionary management, such as adding or deleting a dictionary. Input: *Text files or text-file folders; *Text extracted from database, like MySQL or MongDB. Output: *Saving or exporting as files; *Saving into a database. Technique Requirement: *Accuracy of word segmentation must be above 95%; *Coding language can be determined by both, but Java or Python is plus; *Graphic UI and API are both required; *It’s a system running free with the help of HTTP to call a service. e.g. An application posts a service request to the system (IP: port), then the system will response a result in JSON. Skills Required: *Master NLP and be able to do word segmentation under different languages; *Be very familiar with NLP SDKs such as the Stanford NLP toolkit; *NLP (Tokenizer/Segmenter) developing experience (with the Stanford NLP SDK or similar SDKs) is required. *Providing a Demo or sample work is a plus.