The Registry of Open Data on AWS is now available on AWS Data Exchange
All datasets on the Registry of Open Data are now discoverable on AWS Data Exchange alongside 3,000+ existing data products from category-leading data providers across industries. Explore the catalog to find open, free, and commercial data sets. Learn more about AWS Data Exchange

Provision of Web-Scale Parallel Corpora for Official European Languages (ParaCrawl)

machine translation natural language processing

Description

ParaCrawl is a set of large parallel corpora to/from English for all official EU languages by a broad web crawling effort. State-of-the-art methods are applied for the entire processing chain from identifying web sites with translated text all the way to collecting, cleaning and delivering parallel corpora that are ready as training data for CEF.AT and translation memories for DG Translation.

Update Frequency

New data is added according to ParaCrawl release schedule.

License

Creative Commons CC0 license ("no rights reserved").

Documentation

https://paracrawl.eu/releases.html

Managed By

ParaCrawl

See all datasets managed by ParaCrawl.

Contact

For questions regarding the datasets contact Kenneth Heafield, email kheafiel@inf.ed.ac.uk. For reporting any issues about bitextor pipeline visit https://github.com/bitextor/bitextor/issues.

How to Cite

Provision of Web-Scale Parallel Corpora for Official European Languages (ParaCrawl) was accessed on DATE from https://registry.opendata.aws/paracrawl.

Resources on AWS

  • Description
    Parallel Corpora to/from English for all official EU languages
    Resource type
    S3 Bucket
    Amazon Resource Name (ARN)
    arn:aws:s3:::web-language-models
    AWS Region
    us-east-1
    AWS CLI Access (No AWS account required)
    aws s3 ls --no-sign-request s3://web-language-models/

Edit this dataset entry on GitHub

Tell us about your project

Home