Merge two files in Python PANDAS?

Question

I have two files from where I need to fetch information for data analysis. I am using Python Pandas for this. Any help on how to do this will be appreciated.

I already know how merge 2 files using Python - I am looking forward to achieve this job in PANDAS particularly.

Once 2 files merged then I need to get some analytical data out of it. Both these file do have same structure of data in CSV format.

Why do you want to merge the files in pandas – cammil Jun 03 '15 at 12:43 — cammil, Jun 03 '15 at 12:43

farhawa · Accepted Answer · 2015-06-04T00:42:16.980

6

I would suggest to read the csv files into dataframes and concatenate them this way

frames = [pd.read_csv('f1.csv'), pd.read_csv('f2.csv')]
result = concat(frames,ignore_index=True)

edited Jun 04 '15 at 00:42

answered Jun 03 '15 at 12:44

farhawa

10,120
16
49
91

Thanks `farhawa`. That's exactly what I was looking for to. – Pralhad Narsinh Sonar Jun 03 '15 at 12:45
my pleasure, for more details: [doc](http://pandas.pydata.org/pandas-docs/stable/merging.html) – farhawa Jun 03 '15 at 12:46
2

it's worth mentioning that you may wish to pass param `ignore_index=True` otherwise you may get duplicate index values – EdChum Jun 03 '15 at 13:14
@PralhadNarsinhSonar please take into consideration Edchum remark – farhawa Jun 03 '15 at 13:20
Thanks @EdChum and farhawa. Solution worked for me :) – Pralhad Narsinh Sonar Jun 03 '15 at 13:22
and if i have two dataframes and i need to intersect them to find the same data in a column that has the same name in that two dataframes, is there any faster way than just iterate through these two columns? – Polly Apr 25 '16 at 12:57
Is that pd.concat or something else? – mavavilj Jan 20 '20 at 09:33

score 4 · Answer 2 · answered Jun 03 '15 at 12:45

4

df1 = pd.read_csv(file1)
df2 = pd.read_csv(file2)

df_combined = pd.concat([df1,df2])

df_combined.to_csv(combined_file_name)

answered Jun 03 '15 at 12:45

cammil

9,499
15
55
89

Isn't the problem of this that df1 and df2 will remain in memory, even if one would utilize only df_combined from that point onwards. Thus one basically stores the same data twice? Unless pd has some functionality that it doesn't copy, but only references df1, df2? In that case farhawa's solution is more efficient? https://stackoverflow.com/q/18295630/4959635 – mavavilj Jan 20 '20 at 09:34

Merge two files in Python PANDAS?

2 Answers2