Reading more than 200M rows in pandas

Question

I want to read a csv file that has more than 200M rows and the data look like this:

id_1	date	id_2
Hf23R	01-01-2005	M9R34

There is no null in the data and there is no special character i tried read it the basic way but it crash my computer every time i run the code i cannot do any analysis on it or anything so is there is a way to read via panda in a sufficient way!!

Here is how i read it!

  import pandas as pd
   path = "data.csv"
  df = pd.read_csv(path)

I guess it's memory problem. Have you tried those solutions? https://stackoverflow.com/questions/25962114/how-do-i-read-a-large-csv-file-with-pandas — Kamil Cieszkowski, Apr 12 '21 at 18:29
Is there an error message? Does the computer just lock up? Please clarify what you mean "crash my computer". — RufusVS, Apr 12 '21 at 18:43
There was no error in the code but i got a blue screen and it says there was a problem and the computer need to restart. — samulKarl, Apr 12 '21 at 18:44

score 0 · Answer 1 · answered Apr 12 '21 at 18:57

0

I used the dask and it works perfectly

 import dask.dataframe as dd

 df = dd.read_csv('data.csv')

answered Apr 12 '21 at 18:57

samulKarl

63
4

Reading more than 200M rows in pandas

1 Answers1