Jovian
Sign In
Learn practical skills, build real-world projects, and advance your career
3 years ago

Notebook to filter a large dataset using Python 3

import csv
def csv_writer(in_path,out_path):
    with open(out_path,'at') as outfile: # a = append, t = text
        writer = csv.writer(outfile)
        with open(in_path, newline='', encoding='utf-8') as f:
            reader = csv.reader(f)
            for row in reader:
                if row[col_number] == "XXXXXXXX":
                    writer.writerow(row)
        outfile.close

where

  • in_path = the path to the big csv file
  • out_path = the path to the filtered csv file
  • col_number is the column, counting from zero, that contains the data to be filtered
  • XXXXXXXX is the string in col_number, which if present, that row will be written to out_path
csv_writer("big.csv","filtered.csv")