python str() function result differs from str() function's result

Question

I am updating a hobby app, written in Python 2.7 on Ubuntu 14.04 that stores railway history data in json. I used it upto now to work on british data.

When starting with french data I encountered a problem which puzzles me. I have a class CompaniesCache which implements __str__(). Inside that implementation everything is using str's. Let's say I instantiate a CompaniesCache and assign into a variable companies. When I, in IPython2, give the command print companies, I get an error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe0' in position 184: ordinal not in range(128)".

Alright, that is not strange. Testing. str(companies) reproduces the error, as expected. But, companies.__str__() succeeds without problems, as does print company.__str__(). What is wrong here ?

Here the code of the __str__ method of the CompaniesCache object:

class CompaniesCache(object):                                                       
    def __init__(self, railrefdatapath):       
        self.cache = restoreCompanies(railrefdatapath)                                             

    def __getitem__(self, compcode):                                                                                     
        return self.cache[compcode.upper()]                                                                              

    def __str__(self):                                                                
        s = ''                                                                            
        for k in sorted(self.cache.keys()):                                                                              
            s += '\n%s: %s' % (k, self[k].title)                                                                
        return s

This is the code for the CompaniesCache object, which contains Company objects in its cache dict. The Company object does not implement the __str__() method.

Your `__str__` method is returning a `unicode` object with non-ascii characters in it. — user2357112, Dec 26 '16 at 19:19
First you have a bug in `s += '\n%s: %s' % (k, self[k].title)` -> `s += '\n%s: %s' % (k, self[k].title())` — Yonatan Kiron, Dec 26 '16 at 19:33
@YonatanKiron: title is a member of the Company object, not a getter. Bad style ? — Michiel Rademakers, Dec 26 '16 at 19:35
@MichielRademakers Yep, especially when you are using it in `__str__`, I was thinking you meant `str.title` function — Yonatan Kiron, Dec 26 '16 at 19:40
@YonatanKiron - what makes you think `title` is a function? I don't see that specified anywhere. — tdelaney, Dec 26 '16 at 19:41
Since it's "passed" to a string, So I assumed it's a string, and string has title function. but NVM, it's not related to your question :) — Yonatan Kiron, Dec 26 '16 at 19:43
Ah, now I get the bad style remark ... I was unaware that title is a standard str function, so the choice of the member name is definitely bad. Thanks. — Michiel Rademakers, Dec 26 '16 at 20:10

score 4 · Answer 1 · answered Dec 26 '16 at 19:56

str doesn't just call __str__. Among other things, it validates the return type, it falls back to __repr__ if __str__ isn't available, and it tries to convert unicode return values to str with the ASCII codec.

Your __str__ method is returning a unicode instance with non-ASCII characters. When str tries to convert that to a bytestring, it fails, producing the error you're seeing.

Don't return a unicode object from __str__. You can implement a __unicode__ method to define how unicode(your_object) behaves, and return an appropriately-encoded bytestring from __str__.

This sounds good ! The problem is that the whole thing started out as british only. It is the french data that is injecting unicode data. It seems I have to refactor the app to work with unicode only, and maybe port it to python3. — Michiel Rademakers, Dec 26 '16 at 20:03

score 0 · Answer 2 · edited May 23 '17 at 10:30

0

Using maxpolk answer I think all you should do is setup your environment variable to

export LC_ALL='en_US.utf8'

All and all I think you can find your answer in this post

edited May 23 '17 at 10:30

Community

1
1

answered Dec 26 '16 at 19:38

Yonatan Kiron

2,718
1
20
25

Tried this, but it does not alter anything. My environment is already US utf8. – Michiel Rademakers Dec 26 '16 at 20:01

python str() function result differs from __str__() function's result

2 Answers2

python str() function result differs from str() function's result