0

I am working in a Jupyter Notebook with PySpark v2.3.4 which runs on Java 8, Python 3.6 (with py4j==0.10.7), and Scala 2.11, and I have a Scala case class that takes in a scala.util.matching.Regex (scala doc) as an arg like so: 

case class myClass(myString: String, myRegex: Regex)

I would like to construct an object from myClass but I can't seem to figure out how to construct a scala.util.matching.Regex object in a Python / PySpark environment. Below are a couple of my attempts/docs I've followed to create a Scala regex where sc is my SparkContext.

  • sc._jvm.scala.util.matching.Regex("""(S|s)cala""")
    • Error: Constructor scala.util.matching.Regex([class java.lang.String]) does not exist
    • This error message dumbfounds me because the Scala 2.11 docs clearly state that its constructor takes in a java.lang.String.
  • sc._jvm.scala.util.matching.Regex("(S|s)cala")
    • Same error as above
  • sc._jvm.scala.util.matching.Regex(r"(S|s)cala")
    • Same error as above
  • sc._jvm.scala.util.matching.Regex("(S|s)cala".r) (the way they do it in Scala)
    • Error: Python string does not have attribute "r"
  • sc._jvm.java.util.regex.Pattern.compile("(S|s)cala") successfully creates a Java regex pattern -- and the scala doc clearly states that the Scala regex delegates to the Java regex package...

Any help/advice would be much appreciated! Thanks in advance!

Calvin Wang
  • 81
  • 6
  • So according to [this stackoverflow post](https://stackoverflow.com/questions/9727637/new-keyword-in-scala) it seems that `new Regex` vs `Regex()` should be the roughly the same where the former uses the Regex class's own constructor while the latter refers to a companion object's `apply` method. I might need to overload the singleton Scala function. – Calvin Wang Jun 11 '20 at 16:29

1 Answers1

0

I figured it out lol

Scala Regex takes a second argument called groupNames. It is a variable String argument that should accept args of length 0 to as many String arguments as you want. However, from a Python interpreter, this is seen as a required argument, so you must pass a None into this argument to indicate nothing to populate this second arg. 

sc._jvm.scala.util.matching.Regex("(S|s)cala", None)

NOTE: I haven't figured out how to pass in a vararg yet though... Passing comma separated strings, array, and tuples didn't work... Any help on that would be great thanks :)

Calvin Wang
  • 81
  • 6