In this paper we introduce a new unified syllabic model for French and English handwriting recognition, based on hidden Markov models (HMM). The recognition system training and recognition components such as optical models, lexicons and language models are designed to be language independent. In this purpose a syllable based model is proposed for French and English. This model is evaluated and compared to n-gram character and words models. A promising performance is achieved by the syllabic model, which meets the words model performance, with the advantage of a reduced system complexity. Furthermore, the unification of likely similar scripts improves the system performance over all models considering the English and French languages. The French RIMES and the English IAM datasets are used for the evaluation.