How to write Chinese characters to file by python

Question

I'm walking through a directory and want to write all files names into a file. Here's the piece of code

with open("c:/Users/me/filename.txt", "a") as d:
   for dir, subdirs, files in os.walk("c:/temp"):
      for f in files:
         fname = os.path.join(dir, f)
         print fname
         d.write(fname + "\n")
d.close()

The problem I have is, there are some files that are named in Chinese characters. By using print, I can see the file name correctly in console, but in the target file, it's just a mess... I've tried to open the file like open(u"c:/Users/me/filename.txt", "a"), but it did not work. I also tried to write fname.decode("utf-16"), still does not work...

@TimPietzcker I think the Chinese characters are in the filename. :-) — Sean Francis N. Ballais, Feb 29 '16 at 15:24
As a tip, if you can use/install python3, just do it… the problem might go away magically! — zmo, Feb 29 '16 at 15:25
Python 3x versions supports unicode characters. So use 3x versions. — GLHF, Feb 29 '16 at 15:28
[This](http://stackoverflow.com/questions/491921/unicode-utf8-reading-and-writing-to-files-in-python) question has details on how to prepare your file for UTF. — Laur Ivan, Feb 29 '16 at 15:28

score 3 · Answer 1 · answered Feb 29 '16 at 15:28

In Python 2, it's a good idea to use codecs.open() if you're dealing with encodings other than ASCII. That way, you don't need to manually encode everything you write. Also, os.walk() should be passed a Unicode string if you're expecting non-ASCII characters in the filenames:

import codecs
with codecs.open("c:/Users/me/filename.txt", "a", encoding="utf-8") as d:
   for dir, subdirs, files in os.walk(u"c:/temp"):
      for f in files:
         fname = os.path.join(dir, f)
         print fname
         d.write(fname + "\n")

No need to call d.close(), the with block already takes care of that.

Error raised.. `UnicodeDecodeError: 'ascii' codec can't decode byte 0xd6 in position 8: ordinal not in range(128)` — Bomin, Mar 01 '16 at 01:55

pp_ · Answer 2 · 2016-02-29T15:36:03.590

2

Use str.encode() to encode fname before you write it to the file:

d.write(fname.encode('utf8') + '\n')

edited Feb 29 '16 at 15:36

answered Feb 29 '16 at 15:24

pp_

3,435
4
19
27

2

doesn't not work. `UnicodeDecodeError: 'ascii' codec can't decode byte 0xd6 in position 8: ordinal not in range(128)`. I think the reason is the string is already encoded? – Bomin Mar 01 '16 at 01:53

score 2 · Answer 3 · answered Jul 01 '17 at 08:12

The key is to tell python to prepare the file for being used in "utf-8" format. I wonder why python doesn't assume utf-8 by default. Anyway, try the following:

with open("c:/Users/me/filename.txt", "a", encoding='utf-8') as d:
    for dir, subdirs, files in os.walk("c:/temp"):
        ...

I am using python3.5. So, please be aware that the "encoding" option may be not available in python 2.7. But the idea is to tell python in advance about the encoding, rather than fighting with encoding of each string later.

dsgou · Answer 4 · 2018-12-31T09:31:11.793

To succesfully write chinese characters in python 2 you have to do the following.

Open the file using the codecs library which allows you to provide the encoding parameter and set it to unicode.
Write the string in unicode encoding.

The corrected code would be the following:

import codecs

with codecs.open("c:/Users/me/filename.txt", "a", encoding='utf-8') as d:
    for dir, subdirs, files in os.walk("c:/temp"):
        for f in files:
            fname = os.path.join(dir, f)
            print fname
            d.write(fname.decode('utf-8') + "\n")

Note

The same problem does not exist in python 3 so you should also consider making your script python 3 compatible.

score 1 · Answer 5 · answered Dec 07 '21 at 08:59

1

with open("xyz.xml', "w", encoding='utf-8-sig') as f: worked for me.

answered Dec 07 '21 at 08:59

user3349907

317
3
3

with open("xyz.xml', "a", encoding='utf-8-sig') as f: worked for me – user3349907 Dec 07 '21 at 08:59

How to write Chinese characters to file by python

5 Answers5

Note