The Registry of Open Data on AWS is now available on AWS Data Exchange
All datasets on the Registry of Open Data are now discoverable on AWS Data Exchange alongside 3,000+ existing data products from category-leading data providers across industries. Explore the catalog to find open, free, and commercial data sets.Learn more about AWS Data Exchange
Sudachi Language Resources
natural language processing
Japanese dictionaries and pre-trained models (word embeddings and language models) for natural language processing.
SudachiDict is the dictionary for a Japanese tokenizer (morphological analyzer) Sudachi.
chiVe is Japanese pretrained word embeddings (word vectors), trained using the ultra-large-scale web corpus NWJC by National Institute for Japanese Language and Linguistics, analyzed by Sudachi.
chiTra is a library for using large-scale pre-trained language models with the Japanese tokenizer SudachiPy.
The dictionaries are updated every few months to include neologism and fixes for the existing words.