You should change the way your data.frame is formated to do this easily with ggplot2 syntax.
Instead of having 5 columns, with x, x1, x2, x3, x4, you may want to have a data.frame with 3 columns : x, y and type with type being a categorical variable indicating from which column your y is from (x1, x2, x3 or x4).
That would be something like this :
df <- data.frame(x = rep(data$x, 4),
y = c(data$x1, data$x2, data$x3, data$x4),
type = rep(c("x1", "x2", "x3", "x4"), each = nrow(data))
Then, with this data.frame, you can set the aes in order to plot x according to y for each category of your variable type thanks to the color argument.
ggplot(df, aes(x = x, y = y, color = type)) + geom_point() + geom_smooth(method = "lm, fill = "NA")
You should check http://www.sthda.com/english/wiki/ggplot2-scatter-plots-quick-start-guide-r-software-and-data-visualization for detailed explanations and customizations.