Sunday, December 27, 2020

Open Source Machine Translation Systems



System Team Description Link Framework
Tensor2Tensor Google Brain Library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research. Tensorflow
Fairseq Facebook Research Facebook AI Research Sequence-to-Sequence Toolkit written in Python. Pytorch
facebookresearch/fairseq Facebook Research Facebook AI Research Sequence-to-Sequence Toolkit Lua
tensorflow/nmt Google Brain TensorFlow Neural Machine Translation Tutorial Tensorflow
OpenNMT-tf OpenNMT Neural machine translation and sequence learning using TensorFlow Tensorflow
OpenNMT-py OpenNMT Open Source Neural Machine Translation in PyTorch Pytorch
THUMT Tsinghua Natural Language Processing Group Transformer, Multi-GPU training & decoding, Distributed training Tensorflow/Theano
NiuTrans.NMT NiuTrans Transformer and FFN-LM based on NiuTrans.Tensor by NiuTrans Team. C/C++
MARIANNMT Adam Mickiewicz Pure C++ with minimal dependencies, one engine for GPU/CPU training and decoding C++
Seq2Seq Britz Denny and Goldie, Anna and Luong Thang and Le Quoc A general-purpose encoder-decoder framework for Tensorflow Tensorflow
NEMATUS The Natural Language Processing Group at the University of Edinburgh Support for RNN and Transformer architectures, multi-GPU support, server mode Tensorflow
Sockeye Awslabs A sequence-to-sequence framework for Neural Machine Translation MXNet
CytonMT Wang, Xiaolin and Utiyama, Masao and Sumita, Eiichiro An Efficient Neural Machine Translation Open-source Toolkit Implemented in C++ C++
OpenSeq2Seq NVIDIA Modular architecture,support for mixed-precision training,fast Horovod-based distributed training TensorFlow
nmtpytorch The Language and Speech Team of Le Mans University Various end-to-end neural architectures Pytorch
DL4MT Cho Lab at NYU CS and CDS A multi-encoder, multi-decoder or a multi-way NMT model Theano
ModerNMT Marco, Trombetti and Davide, Caroselli and Nicola, Bertoldi A context-aware, incremental and distributed general purpose Neural Machine Translation technology based on Fairseq Transformer model PyTorch
UnsupervisedMT Facebook Research Seq2seq, biLSTM + attention, Transformer. Ability to share an arbitrary number of parameters. Denoising auto-encoder training. PyTorch


System Team Description Link Framework
Moses moses-smt A free software, statistical machine translation engine that can be used to train statistical models of text translation from a source language to a target language C++
GIZA++ moses-smt A SMT toolkit that is used to train IBM Models 1-5 and an HMM word alignment model C++
NiuTrans.SMT NiuTrans NiuTrans.SMT is an open-source statistical machine translation system developed by a joint team from NLP Lab. at Northeastern University and the NiuTrans Team. The NiuTrans system is fully developed in C++ language. C/C++
UCAM-SMT The MT group in Cambridge The Cambridge Statistical Machine Translation system C++
Jane The RWTH Aachen University Supports state-of-the-art techniques for phrase-based and hierarchical phrase-based machine translation C++
Phrasal Stanford NLP Group A state-of-the-art statistical phrase-based machine translation system Java
cdec The Language Technologies Institute in Carnegie Mellon University A decoder, aligner, and learning framework for SMT and similar structured prediction models C++
JOSHUA Juri Ganitkevitch and Matt Post A SMT decoder for phrase-based, hierarchical, and syntax-based machine translation Java