Performant Python

Reading and writing files with Numpy

Overview:

• Teaching: 5 min
• Exercises: 15 min

Questions

• How can I read data into a numpy array
• How can I read and write numpy arrays

Objectives

• Know how to use loadtxt to read files with different formatting
• Know how to save numpy arrays as text files with savetxt
• Understand how save and load differ from these and why you might use them

Reshaping Arrays

First as always we must import numpy:

import numpy as np

For a changed we will now import a data set from the data folder. Numpy has a builtin function loadtxt to read the data directly into a numpy array. We have to pass both the filename and give the delimiter in this case a ','':

data_set = np.loadtxt("./data/random_data.csv",delimiter=",")
print(data_set)

[[ 0.00000000e+00  5.00000000e-01  0.00000000e+00  0.00000000e+00
   0.00000000e+00]
 [ 2.04081633e-01  4.79355330e-01 -2.00560658e-01  1.02684298e-02
   9.92266839e-02]
 [ 4.08163265e-01  4.19547901e-01 -3.80483604e-01  3.93778045e-02
   1.82021527e-01]
 [ 6.12244898e-01  3.26712157e-01 -5.21510425e-01  8.24678014e-02
   2.34157403e-01]
 [ 8.16326531e-01  2.10284231e-01 -6.09853878e-01  1.32166983e-01
   2.45536487e-01]
 [ 1.02040816e+00  8.19199477e-02 -6.37751288e-01  1.79589100e-01
   2.11590508e-01]
 [ 1.22448980e+00 -4.58327665e-02 -6.04293415e-01  2.15545729e-01
   1.33977027e-01]
 [ 1.42857143e+00 -1.60947443e-01 -5.15430837e-01  2.31820390e-01
   2.04793046e-02]
 [ 1.63265306e+00 -2.53249583e-01 -3.83163518e-01  2.22340934e-01
  -1.15880217e-01]
 [ 1.83673469e+00 -3.15506351e-01 -2.24014723e-01  1.84098970e-01
  -2.58408426e-01]
 [ 2.04081633e+00 -3.44147416e-01 -5.69808899e-02  1.17695360e-01
  -3.88798720e-01]
 [ 2.24489796e+00 -3.39535081e-01  9.87889495e-02  2.74372341e-02
  -4.89422957e-01]
 [ 2.44897959e+00 -3.05757463e-01  2.26298531e-01 -7.90325673e-02
  -5.45573689e-01]
 [ 2.65306122e+00 -2.49976952e-01  3.12712027e-01 -1.91534570e-01
  -5.47376832e-01]
 [ 2.85714286e+00 -1.81420709e-01  3.50819381e-01 -2.98487387e-01
  -4.91134081e-01]
 [ 3.06122449e+00 -1.10142962e-01  3.39783849e-01 -3.88243141e-01
  -3.79924687e-01]
 [ 3.26530612e+00 -4.57163349e-02  2.85084507e-01 -4.50462282e-01
  -2.23385440e-01]
 [ 3.46938776e+00  3.98333319e-03  1.97668007e-01 -4.77360592e-01
  -3.66898525e-02]
 [ 3.67346939e+00  3.37471033e-02  9.24213677e-02 -4.64675500e-01
   1.61154483e-01]
 [ 3.87755102e+00  4.16381495e-02 -1.38380935e-02 -4.12232019e-01
   3.49486985e-01]
 [ 4.08163265e+00  2.91588879e-02 -1.04600108e-01 -3.24036748e-01
   5.08390948e-01]
 [ 4.28571429e+00  9.62336093e-04 -1.65842916e-01 -2.07885293e-01
   6.21016383e-01]
 [ 4.48979592e+00 -3.58523032e-02 -1.87883195e-01 -7.45274225e-02
   6.75547333e-01]
 [ 4.69387755e+00 -7.27674263e-02 -1.66661110e-01  6.35122381e-02
   6.66575439e-01]
 [ 4.89795918e+00 -1.01064832e-01 -1.04297655e-01  1.93318318e-01
   5.95713253e-01]
 [ 5.10204082e+00 -1.13076132e-01 -8.85328341e-03  3.03009313e-01
   4.71373965e-01]
 [ 5.30612245e+00 -1.03304007e-01  1.06683264e-01  3.83033715e-01
   3.07745598e-01]
 [ 5.51020408e+00 -6.92685087e-02  2.26050250e-01  4.27184569e-01
   1.23085937e-01]
 [ 5.71428571e+00 -1.19668422e-02  3.31805786e-01  4.33213962e-01
  -6.24531545e-02]
 [ 5.91836735e+00  6.41166345e-02  4.07596022e-01  4.02977704e-01
  -2.29206411e-01]
 [ 6.12244898e+00  1.51457225e-01  4.40313134e-01  3.42097600e-01
  -3.60290595e-01]
 [ 6.32653061e+00  2.40341835e-01  4.21871329e-01  2.59188507e-01
  -4.43537193e-01]
 [ 6.53061224e+00  3.20025152e-01  3.50374330e-01  1.64747722e-01
  -4.72829355e-01]
 [ 6.73469388e+00  3.80048038e-01  2.30523049e-01  6.98468902e-02
  -4.48684747e-01]
 [ 6.93877551e+00  4.11555932e-01  7.32064015e-02 -1.52118193e-02
  -3.78019978e-01]
 [ 7.14285714e+00  4.08453306e-01 -1.05682160e-01 -8.21102535e-02
  -2.73132021e-01]
 [ 7.34693878e+00  3.68248616e-01 -2.87062139e-01 -1.25451478e-01
  -1.50029980e-01]
 [ 7.55102041e+00  2.92481733e-01 -4.50899192e-01 -1.43307793e-01
  -2.63293771e-02]
 [ 7.75510204e+00  1.86675936e-01 -5.78583123e-01 -1.37316655e-01
   8.10254552e-02]
 [ 7.95918367e+00  5.98145869e-02 -6.55160439e-01 -1.12318619e-01
   1.57924782e-01]
 [ 8.16326531e+00 -7.65995131e-02 -6.71148935e-01 -7.55886544e-02
   1.94873035e-01]
 [ 8.36734694e+00 -2.09792836e-01 -6.23710702e-01 -3.57629041e-02
   1.88162370e-01]
 [ 8.57142857e+00 -3.27130663e-01 -5.17038755e-01 -1.60086853e-03
   1.40295405e-01]
 [ 8.77551020e+00 -4.17520234e-01 -3.61907996e-01  1.92568993e-02
   5.96063457e-02]
 [ 8.97959184e+00 -4.72640597e-01 -1.74443113e-01  2.13753624e-02
  -4.08698213e-02]
 [ 9.18367347e+00 -4.87852974e-01  2.57483724e-02  2.31558357e-03
  -1.45136036e-01]
 [ 9.38775510e+00 -4.62684002e-01  2.17852319e-01 -3.69733348e-02
  -2.36359531e-01]
 [ 9.59183673e+00 -4.00825837e-01  3.82258145e-01 -9.22131894e-02
  -2.99116765e-01]
 [ 9.79591837e+00 -3.09655542e-01  5.02811737e-01 -1.56301609e-01
  -3.21472546e-01]
 [ 1.00000000e+01 -1.99334203e-01  5.68639629e-01 -2.20201536e-01
  -2.96629104e-01]]

We can save the data using the built-in function savetxt:

np.savetxt("save_data.csv", data_set)

And verify that it has been saved by printing the file to screen:

!cat save_data.csv

0.000000000000000000e+00 5.000000000000000000e-01 0.000000000000000000e+00 0.000000000000000000e+00 0.000000000000000000e+00
2.040816326530000002e-01 4.793553299680000102e-01 -2.005606575140000125e-01 1.026842976289999917e-02 9.922668394049999341e-02
4.081632653060000004e-01 4.195479009029999817e-01 -3.804836043129999901e-01 3.937780449639999986e-02 1.820215267349999944e-01
6.122448979589999452e-01 3.267121566869999882e-01 -5.215104252390000417e-01 8.246780140270000559e-02 2.341574029060000117e-01
8.163265306120000009e-01 2.102842312950000125e-01 -6.098538777749999973e-01 1.321669825339999993e-01 2.455364867350000035e-01
1.020408163269999946e+00 8.191994767499999330e-02 -6.377512880900000347e-01 1.795890996670000128e-01 2.115905081570000068e-01
1.224489795920000068e+00 -4.583276650830000326e-02 -6.042934151850000157e-01 2.155457290460000008e-01 1.339770274350000068e-01
1.428571428569999968e+00 -1.609474432889999862e-01 -5.154308374279999994e-01 2.318203898029999876e-01 2.047930459730000138e-02
1.632653061220000090e+00 -2.532495828479999767e-01 -3.831635179099999733e-01 2.223409344360000039e-01 -1.158802174650000044e-01
1.836734693879999991e+00 -3.155063505449999806e-01 -2.240147227750000081e-01 1.840989698050000123e-01 -2.584084260719999859e-01
2.040816326529999891e+00 -3.441474155720000150e-01 -5.698088987370000108e-02 1.176953602970000068e-01 -3.887987202500000072e-01
2.244897959179999791e+00 -3.395350810119999729e-01 9.878894945379999848e-02 2.743723414980000011e-02 -4.894229572000000084e-01
2.448979591840000136e+00 -3.057574633060000102e-01 2.262985311110000120e-01 -7.903256729869999331e-02 -5.455736887630000531e-01
2.653061224490000036e+00 -2.499769516370000111e-01 3.127120267769999962e-01 -1.915345700809999996e-01 -5.473768316560000402e-01
2.857142857139999936e+00 -1.814207085389999974e-01 3.508193812379999943e-01 -2.984873866669999765e-01 -4.911340806580000162e-01
3.061224489799999837e+00 -1.101429622809999981e-01 3.397838487879999980e-01 -3.882431410709999975e-01 -3.799246870310000190e-01
3.265306122450000181e+00 -4.571633494820000199e-02 2.850845068489999967e-01 -4.504622822530000015e-01 -2.233854400549999952e-01
3.469387755100000081e+00 3.983333188689999749e-03 1.976680069089999936e-01 -4.773605916020000062e-01 -3.668985245109999960e-02
3.673469387759999982e+00 3.374710325219999751e-02 9.242136768330000418e-02 -4.646754998200000042e-01 1.611544828610000091e-01
3.877551020409999882e+00 4.163814947279999984e-02 -1.383809345530000005e-02 -4.122320193480000228e-01 3.494869851819999984e-01
4.081632653059999782e+00 2.915888788850000066e-02 -1.046001077609999969e-01 -3.240367481820000117e-01 5.083909477310000202e-01
4.285714285710000127e+00 9.623360925279999612e-04 -1.658429159289999955e-01 -2.078852933610000064e-01 6.210163827210000154e-01
4.489795918369999583e+00 -3.585230321249999880e-02 -1.878831952789999926e-01 -7.452742248230000144e-02 6.755473331229999712e-01
4.693877551019999927e+00 -7.276742628490000253e-02 -1.666611095530000086e-01 6.351223805569999881e-02 6.665754389439999539e-01
4.897959183670000272e+00 -1.010648322099999968e-01 -1.042976548060000064e-01 1.933183176619999999e-01 5.957132525080000018e-01
5.102040816329999728e+00 -1.130761317709999964e-01 -8.853283410339999812e-03 3.030093134929999765e-01 4.713739648020000006e-01
5.306122448980000073e+00 -1.033040074249999968e-01 1.066832636669999973e-01 3.830337151789999783e-01 3.077455978779999879e-01
5.510204081630000417e+00 -6.926850874590000096e-02 2.260502503519999928e-01 4.271845694190000153e-01 1.230859369839999995e-01
5.714285714289999873e+00 -1.196684216269999941e-02 3.318057860479999843e-01 4.332139619470000147e-01 -6.245315445510000313e-02
5.918367346940000218e+00 6.411663448160000167e-02 4.075960223289999962e-01 4.029777038910000164e-01 -2.292064107429999942e-01
6.122448979589999674e+00 1.514572246429999880e-01 4.403131339690000146e-01 3.420976002119999748e-01 -3.602905954150000256e-01
6.326530612240000018e+00 2.403418349979999924e-01 4.218713291990000003e-01 2.591885065680000211e-01 -4.435371931129999945e-01
6.530612244900000363e+00 3.200251524200000142e-01 3.503743295849999773e-01 1.647477216719999971e-01 -4.728293552780000186e-01
6.734693877549999819e+00 3.800480380210000120e-01 2.305230494999999935e-01 6.984689016589999688e-02 -4.486847467749999763e-01
6.938775510200000163e+00 4.115559323240000067e-01 7.320640147960000288e-02 -1.521181931479999999e-02 -3.780199777689999818e-01
7.142857142859999620e+00 4.084533062799999836e-01 -1.056821600519999987e-01 -8.211025345620000249e-02 -2.731320209259999743e-01
7.346938775509999964e+00 3.682486164820000241e-01 -2.870621394480000199e-01 -1.254514778860000013e-01 -1.500299800849999865e-01
7.551020408160000308e+00 2.924817327240000209e-01 -4.508991916080000162e-01 -1.433077929450000121e-01 -2.632937713929999965e-02
7.755102040819999765e+00 1.866759361820000129e-01 -5.785831228350000188e-01 -1.373166547169999974e-01 8.102545522350000640e-02
7.959183673470000109e+00 5.981458693060000276e-02 -6.551604386930000024e-01 -1.123186187689999976e-01 1.579247822829999981e-01
8.163265306119999565e+00 -7.659951313809999773e-02 -6.711489350899999540e-01 -7.558865436539999316e-02 1.948730348899999909e-01
8.367346938780000798e+00 -2.097928356450000087e-01 -6.237107024279999745e-01 -3.576290406270000305e-02 1.881623696320000039e-01
8.571428571429999366e+00 -3.271306626670000273e-01 -5.170387546960000247e-01 -1.600868533180000016e-03 1.402954045189999943e-01
8.775510204079999710e+00 -4.175202336450000162e-01 -3.619079964810000272e-01 1.925689934449999891e-02 5.960634573959999738e-02
8.979591836730000054e+00 -4.726405969610000257e-01 -1.744431127809999982e-01 2.137536241290000083e-02 -4.086982125739999777e-02
9.183673469389999511e+00 -4.878529738580000275e-01 2.574837241620000008e-02 2.315583574920000026e-03 -1.451360355130000002e-01
9.387755102039999855e+00 -4.626840022959999854e-01 2.178523186040000059e-01 -3.697333482960000334e-02 -2.363595308460000044e-01
9.591836734690000199e+00 -4.008258372949999870e-01 3.822581446409999972e-01 -9.221318938519999786e-02 -2.991167650670000233e-01
9.795918367349999656e+00 -3.096555418260000248e-01 5.028117373889999797e-01 -1.563016087000000109e-01 -3.214725460519999856e-01
1.000000000000000000e+01 -1.993342028450000025e-01 5.686396286789999754e-01 -2.202015360829999990e-01 -2.966291038919999790e-01

Formatting with savedata

Explore the help for savedata to output the file in a more friendly form.

What happens if you instead use the function save. Also load the file produced to verify that the file contains the correct data set.

np.save("save_numpy", data_set)

Solution

Key Points:

• Numpy has built-in functions to read, loadtxt and write savetxt text files.
• Numpy can also write binary forms of arrays to preserve their precision

Previous

Schedule

Next