What is the .format() equivalent for python f-string without using eval?

Question

This is NOT a duplicate question, please read fully :

With python f-strings one can write :

s='abcdef'
f'{s[:4]}'
Out[17]: 'abcd'

But the actual str.format() function is not totally compatible with f-string syntax :

'{s[:4]}'.format(s=s)

Traceback (most recent call last):
  ...      
TypeError: string indices must be integers

There are some questions on S.O. speaking about that but all propose to use eval() which is not a good solution for security reasons : Is there a function somewhere I could call on a string so it can be interpreted as a f-string ?

No, I'd like to put '{s[:4]}' somewhere in a text file, so I need a .format() - like function to interpret that. — Eric, Jan 04 '21 at 11:50
I guarantee you from professional experience with i18n/l10n, no, you would not like any such thing. If a programmer is writing the text file, the programmer will wonder why the code is not in a .py file; if a UI designer is writing it, that is a person who is not being paid to understand arcane nonsense like `[:4]`. *Especially* if it's outsourced for translation or something. What you propose also violates the basic programming/design principle of *separation of concerns*. Putting the string together is a separate task from determining the values for interpolation. — Karl Knechtel, Jan 04 '21 at 11:57
While I think the case could be made that this is not a strict duplicate of the other, as far as I can tell the answers there answer this question: the answers on the linked dupe, and the [string format DSL documentation](https://docs.python.org/3.5/library/string.html#format-specification-mini-language) seem to say "no you can't do that". Is there a particular reason I should vote to reopen this? — Jared Smith, Jan 04 '21 at 12:03
@Karl : if str.format() exists, a f-str.format() should exists that's that simple. But I agree with you, if the text file goal is to be outsourced, it is better to have python code somewhere between and re-write a new user-friendly text file format, this will be much cleaner, but much longer. In my case, the configuration file will not be outsourced, only written by developper and then much faster than replace many methods... — Eric, Jan 04 '21 at 12:05
@Jared : if str.format() exists, a f-str.format() should exists. I'd like to use f-string syntax in a template. But it could be useful also at python side if you want to define a pattern without coding a function that do what str.format() cannot do. — Eric, Jan 04 '21 at 12:11
@Eric I still don't understand. Just because you think a thing *should* exist, doesn't mean that it *does* exist. *Should* and consistency are nice, but programming languages violate those all the time. Since f-string is strictly speaking greatly more capable than the string format minilanguage there are going to be lots of f-strings you cannot express in the less capable DSL. So since the thing you want doesn't exist, what are you expecting to accomplish here? — Jared Smith, Jan 04 '21 at 12:14
@Eric: f-strings *are* a special-case of `eval`, for all practical purposes (in the bytecode, they're built piecemeal by evaluating their components and putting them all together). If you want to do `eval`y things with `str.format`, `eval` will be involved. This *specific* case could be handled with `'{!s:.4}'.format` if the argument will be a `str` or you want the first four characters of its stringified form (the `!s` can be omitted if it's always a `str`), but you can't use slicing syntax, it's that simple. — ShadowRanger, Jan 04 '21 at 12:16
@Jared, I was expecting someone saying there is may be an offical or internal function for that, or may be a library somewhere simulating f-string rendering — Eric, Jan 04 '21 at 12:19
@ShadowRanger, the `{s[:4]}` was an example, it could be a `{s.a_method(param=something)}` — Eric, Jan 04 '21 at 12:22
@Eric: "I was expecting someone saying there is may be an offical or internal function for that." Yup, that's called `eval`. f-strings are a *syntactic* feature of the language; they're baked into the Python parser/bytecode compiler itself. In the compiled bytecode, there isn't an f-string at all (see the disassembly [here](https://tio.run/##K6gsycjPM/7/PzO3IL@oRCEls5gLiPWAWEMpTb06US8psyQ@JzUvvSRDQ7O2GsjSSNKsVVfS/P8fAA)). I assumed you were looking for a general solution (thus the emphasis on *specific*); the answer is to use some other templating library with similar features (e.g. Jinja2). — ShadowRanger, Jan 04 '21 at 13:09
@furas: That's just describing the pre-f-string formatting features (which don't satisfy the OP's needs for effectively arbitrary code execution defined by the format string itself). — ShadowRanger, Jan 04 '21 at 13:11
@Eric that doesn't change anything. In an f-string, the contents between the braces is an arbitrary Python expression. In other string-formatting operations it's not, both because that's undesirable (though `str.format` is already completely insecure) and because you can put the expression in the parameter part. The f-string is actually split apart and parsed (into string literals and expressions to evaluate) during parsing. And if you want to write non-trivial templates, use an actual template processor. — Masklinn, Jan 04 '21 at 13:24
@Masklinn: Care to elaborate on "`str.format` is completely insecure"? Obviously if the input objects are from user-provided code they can do anything (but if you ran user-provided code, it can already do anything; no style points for doing it via formatting). But if you accept a format string from a user and use it with securely defined objects, the worst it can do is a denial of service (via excessive memory usage) to my knowledge. It can't go off and execute arbitrary code, unlike f-strings, which are Turing complete. — ShadowRanger, Jan 04 '21 at 13:58
@ShadowRanger it can access arbitrary attributes and indices, that is not completely arbitrary (unlike eval or f-strings) but since it can access "interpreter internal" attributes it's essentially guaranteed that it can access essentially any module in the system, and most of its data. The only probably-safe string-formatting pattern is `%` and even that's not a guarantee. — Masklinn, Jan 04 '21 at 15:13

ShadowRanger · Accepted Answer · 2021-01-04T14:01:31.797

What you want is not possible with any built-in facility in Python shy of eval; f-strings are a syntactic feature of the language itself, there is no built-in function or module that implements them in a way you can use aside from eval. In CPython, when the compiler gets through with an f-string, there is nothing left of the original "string", it's been broken down into bytecode constructing the individual elements, followed by an instruction that puts them all together (omitted for simple cases where there is only one component being built). For example:

import dis
dis.dis("f'{a.bit_length()}{len(b)}'")

produces:

  1           0 LOAD_NAME                0 (a)
              2 LOAD_METHOD              1 (bit_length)
              4 CALL_METHOD              0
              6 FORMAT_VALUE             0
              8 LOAD_NAME                2 (len)
             10 LOAD_NAME                3 (b)
             12 CALL_FUNCTION            1
             14 FORMAT_VALUE             0
             16 BUILD_STRING             2
             18 RETURN_VALUE

Try it online!

(the _NAME bytecodes might be _FAST or _GLOBAL or the like when running in functions, and the RETURN_VALUE is an artifact of how dis works with isolated expressions, but the general structure is the otherwise the same).

If you need essentially arbitrary code execution from provided template text (that's what f-strings are after all), you're stuck going with eval or a third-party library with similar (not identical) functionality, e.g. Jinja2 and the like. In practice, I think you have an XY problem, but on the off chance you don't, you want third party full-featured templating.

Thank you for the explaination : you shown me it is really built-in, that's the reason why f-string is so fast. I was expecting to use it as a better template engine than .format() without using eval or another template engine... But I was also expecting to use f-formating in some python code places where I used to use str.format()... but I am sad it is not possible : it could be usefull, instead, I have to build a small function... — Eric, Jan 04 '21 at 15:33

What is the .format() equivalent for python f-string without using eval?

1 Answers1