Russian Webpage parsing support.

[mrgodhani]

• Platform: Mac
• Mercury Parser Version: Web based api (at moment)
• Node Version (if a Node bug):
• Browser Version (if a browser bug):

Expected Behavior

Proper encoding for Russian language.

Current Behavior

When parsing this link https://www.finam.ru/analysis/newsitem/putin-nagradil-grefa-ordenom-20190208-203615/?utm_source=rss&utm_medium=new_compaigns&utm_campaign=news_to_finamb it doesn't give proper encode output and hence format is messed up when rendering in html.

Steps to Reproduce

1. Parse link https://www.finam.ru/analysis/newsitem/putin-nagradil-grefa-ordenom-20190208-203615/?utm_source=rss&utm_medium=new_compaigns&utm_campaign=news_to_finamb
2. Check the content output
3. Try to render that content with Cyrillic font
4. You will see instead of proper format it shows bunch of '�'

Detailed Description

I use this API for parsing articles in my reader app. And there are some Russian news feed try to use and are not able to get proper format output.

Possible Solution

[mrgodhani]

More reference https://github.com/mrgodhani/raven-reader/issues/269

[HenryQW]

The same applies to Chinese and other Asian languages, you get a bunch of unicodes rather than the actual content. See #264

[adampash]

Thanks for reporting @mrgodhani — and @HenryQW. I'll be honest: I don't have a ton of experience with encoding in these scenarios. This is where the encoding currently takes place:

https://github.com/postlight/mercury-parser/blob/e033835c7287904371371f922c487e6d0d7d7db8/src/resource/index.js#L63-L79

Does anything stand out to you as doing it wrong? Or any other suggestions? We're more than happy to accept help.

[vjyanand]

My findings #267

1. Change regex case-insensitive

var ENCODING_RE = /charset=([\w-]+)\b/i;

2. Check truthfulness of metaContentType before comparing

if (metaContentType && properEncoding !== encoding) {

Thanks

[mrgodhani]

@adampash I tested @vjyanand's fix and that's working well.

[toufic-m]

This fix has been merged, and will be included in the next release.

[mindfulme]

Did you get it released? I have the following when parsing Russian language articles